Hi,
At 1st part I’ve provided usability and speed of coding comparison of search microservices built upon different tech stacks: PHP+Symfony, Python+Flask and Java + Spring Boot. At current article we will speak about performance. To compare performance of above mentioned frameworks I used next load test command line tools: Apache Bench and Siege. All tests were provided at my laptop, which has next parameters: Intel i5 CPU with 8 threads, 32 GB RAM, SSD disk, Ubuntu 20 as operational system – typical configuration for everyday usage.
Let’s start from Apache Bench. In case you don’t know, Apache Bench is a popular open-source command-line tool to perform simple load tests on an HTTP server, be it a website or an API. It can be easy installed at Ubuntu with next command:
apt-get install apache2-utilsThe simplest way to use current utility is to use the command in next format:
ab -n 100 -c 5 <some_url>Parameter -c means the number of concurrent users, while -n – number of requests. So, in example above we are sending 100 requests using 5 concurrent user’s loading simulation. The result’s output have the next format:
Server Software: nginx
Server Hostname: udemy_phpes.com.test
Server Port: 80
Document Path: /api/search/hotels?page=1&size=10&n=star&c=warsaw&lat=52.21&lng=21.01&stars=4&fpn=true&age=5
Document Length: 91 bytes
Concurrency Level: 5
Time taken for tests: 1.034 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 33300 bytes
HTML transferred: 9100 bytes
Requests per second: 96.72 [#/sec] (mean)
Time per request: 51.698 [ms] (mean)
Time per request: 10.340 [ms] (mean, across all concurrent requests)
Transfer rate: 31.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 20 50 12.1 52 72
Waiting: 20 50 12.1 52 72
Total: 21 50 12.1 52 72
Percentage of the requests served within a certain time (ms)
50% 52
66% 57
75% 60
80% 62
90% 64
95% 65
98% 70
99% 72
100% 72 (longest request)Below you can see the results which I’ve got using ab utility for 3 different frameworks. As url I’ve used the request to search API provided by according search microservice. I think that the most interesting would be to show the mean loading time. Lets start with 100 request and 5 concurrency users. At X axis we have mean response time in milliseconds, which I’ve got from ab utility results output. At Y axis you will find the concurrency and number of requests parameters that were used during running Apache bench utility.
So, in case of such a small loading Symfony+PHP showed itself twice slower than Flask+Python or Spring+Java. Let’s go further, and increase number of requests at 1000 while also increasing concurrency to the 10, and then to the 100:
Eventually, the gap between frameworks increased a lot. Flask appeared twice slower than Spring, while Symfony mean response time reached 400 ms. Then I decided increase ab utility parameters even more – up to the 20000 requests with 1000 concurrency. It is really high loading. And here huge surprise was waiting for me. Let’s have a look at chart bar together:
Symfony+PHP almost reached it’s possibilities – 5 sec response, which is non acceptable at web world nowadays. If consider 2 sec response time acceptable (according to non formal web standards), I found that Symfony reached that value while I run Apache bench with next parameters: ab -n 10000 -c 500. It gave me 2.1 sec mean response time.
Spring Boot in case of ab -n 20000 -c 1000 test showed less than 1 sec response time, which is really good result.
But what has happened with Flask+Python? Is is some error? “Yes and no” at the same time. At 1st I could not understand how is it possible. Let me explain how that magic has happened. It appeared that uwsgi+nginx cache worked here very effectively. Now Python fans will say – we are the best 🙂 . Don’t be in a hurry, it is too earlier to open champagne. The deal is Apache bench is limited only at requesting the same one URL. In case search systems, it is far from reality that the same search request will be constantly repeated. At practice – different users will search different things. So, I decided not to stop and continue my investigation.
After short searching at Google I found another load utility – siege. It allows to perform HTTP Load testing on multiple URLs. To install it at Ubuntu I used next command:
apt-get install siege -yIt appeared that siege has another approach for testing than Apache bench, so we can’t compare two current tools. Siege usage example has next view:
siege -t1 -c200 -f urls.txtThe result output has the next format:
** SIEGE 4.0.4
** Preparing 200 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions: 11681 hits
Availability: 100.00 %
Elapsed time: 59.92 secs
Data transferred: 0.44 MB
Response time: 1.02 secs
Transaction rate: 194.94 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 198.08
Successful transactions: 11681
Failed transactions: 0
Longest transaction: 1.54
Shortest transaction: 0.06In that case we cant pass the number of request. We only can define the time, withing which we want to perform loading test, and concurrency. At example above the concurrency is set to 200 users, while test would be provided within 1 min. The most interesting part is file parameter – it allows to load URLs from the file to send requests randomly on those URLs. So, I wrote small generator that created me 1000 urls with different get parameters for every microservice.
Below you may see what results I’ve got while running siege with different parameters. At X axis there is a response time in seconds, at Y – time and concurrency parameters for siege utility:
You may ask – what the “question sign” at chart means? The deal is that while setting concurrency at 200 value – Flask could not processed most of requests – availability was at 25% only. I found that threshold is located at ~110 concurrent users within 1 min. After that – 1st lost requests are appearing.
Let’s have a look at all charts once more and try to make some summary. The winner, without no doubts, is Java+Spring Boot. In case of Flask – despite that it created good competition for Spring during making Apache bench tests, in the end situation appeared not so pink. Looks like that it works 2 time faster in comparison to PHP framework, but availability threshold raises a lot of doubts. In case of Symfony – it appeared to be the most slow among competitors, but stable. Seems we can use it at production having rather high concurrency – 200-500 concurrent users should not be a problem for one node. Thousand concurrent users will require horizontal scaling. I suppose that my tests confirmed some of my observations from commercial life at Web applications branch. In case startups you are not interesting at performance – the main goal to create stable, reliable solution as fast as possible, minimizing costs. In such a cases such frameworks as Symfony or Flask – are good choices. Even in case business would appear to be successful, you will be still able to grown up a long time. It is even not a problem to reach high sky if, of course, you have “head“ :). Such companies as Spotify, BlaBlaCar (which use Symfony as main backend tool) or Pinterest, Reddit (which use Flask accordingly) already proved it. But it is worth to admit that Java+Spring Boot belongs to another, Enterprise world. Yes, it is non convenient to work with it, the speed of coding prolong release of final product a lot, debugging can be a nightmare. But if performance is obviously essential – Spring Boot is a good choice for web development. No wonder that such giants as Alibaba, Uber, Netflix, LinkedIn choose it despite higher cost and time expenditures in comparison to more usable Symfony or Flask alternatives.
Hope you liked current material. If you want to dive deeper at building search microservices with Spring Boot, Symfony or Flask – then welcome to my courses at udemy:
- Elasticsearch as you have never known it before (you may find a discount coupon here)
- Elasticsearch at AWS using terraform and ansible (you may find discount coupon here)
You also may subscribe to my newsletter. Thank you for your attention.
P.S. It appeared that current theme raised so high interest, that I had to continue and created the 3d part: “Symfony vs Flask vs Spring Boot – part 3 – PHP, Symfony tuning“
