symfony framework forum: General discussion => Symfony - Ruby on Rails

symfony forum

Members

FAQ

Home

Home » support » General discussion » Symfony - Ruby on Rails - Django performance test

Show: Today's Messages :: Show Polls :: Message Navigator

Symfony - Ruby on Rails - Django performance test [message #8815]

Fri, 14 July 2006 12:18

matt
Messages: 21
Registered: June 2006

Junior Member

We recently decided to switch our projects to a MVC framework and our choice of the framework came down to three final contenders:

* Symfony on PHP http://www.symfony-project.com/
* Rails on Ruby http://rubyonrails.org/
* Django on Python http://www.djangoproject.com/

To better understand the pros and cons of each, we wrote a very simple demo application in all three. The app we chose was the one built in the Ruby on Rails tutorial Rolling with Ruby on Rails http://www.onlamp.com/pub/a/onlamp/2005/01/20/rails.html.

We really wanted to test the framework, not static file serving or database performance. So we didn't come up with complex queries or serve any static CSS or image files. All requests/hits/transactions are rendered pages. Also, all of the tests are GET operations (database SELECTs) for concurrent testing fairness and simplicity.

Each of the tested pages was configured to have a unique number displayed on it based on microtime (to verify the pages weren't being cached). Also, the same pages across all three apps were edited to have nearly the same number of bytes served.

DISCLAIMER: There are infinitely more test scenarios we could have performed. But we wanted something done in a finite time. We feel these results, though not the whole picture, offer hints about relative performance between the three frameworks/platforms.

Test Hardware:

* Dell 1U server (unknown model)
* Dual Xeon 3.2Ghz (6400 bogomips)
* 2 Gigs of RAM
* 2 240Gig (?) SATA drives

Test OS/Software:

* OS: SuSE 10.1
* kernel: 2.6.16.13-4-smp
* database: Mysql 5.0.18
* stress-test software: Siege 2.64 http://www.joedog.org/JoeDog/Siege
* Used stock RPM's for all software unless noted below

Symfony setup:

* Apache 2.2.0 (mod_php)
* PHP 5.1.2
* APC 3.0.10 (PHP accelerator, compiled from source)
* Symfony 0.7.1587 (not an RPM)

Django setup:

* Apache 2.2.0 (mod_python)
* Python 2.4.2
* Django 0.95-3336 (not an RPM)

Rails setup:

* Apache 2.2.0 (mod_proxy_balancer)
* Ruby 1.8.4
* Rails 1.1.4 (not an RPM)
* Mongrel and lighttpd web server (not an RPM)

Apache Prefork tuning:

StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 150
MaxClients 150
MaxRequestsPerChild 10000

Results

For each of the platforms, we experimented a bit with some options to try to get the best performance by running 3-minute tests. Then we ran an extended 30-minute test on each platform. (Before the actual tests, we pre-ran siege for a minute or two to prime whatever cache's, etc.) Here are some of the results:

For each of the platforms, we experimented a bit with some options to try to get the best performance by running 3-minute tests. Then we ran an extended 30-minute test on each platform. (Before the actual tests, we pre-ran siege for a minute or two to prime whatever cache's, etc.) Here are some of the results:

Symfony - 50 concurrent users

With PHP+APC: (30 min)

Transactions:                  85862 hits
Availability:                 100.00 %
Elapsed time:                1801.15 secs
Data transferred:             114.41 MB
Response time:                  1.04 secs
Transaction rate:              47.67 trans/sec  <---
Throughput:                     0.06 MB/sec
Concurrency:                   49.63
Successful transactions:       85862
Failed transactions:               0
Longest transaction:            4.15  <---
Shortest transaction:           0.02

Mysql: ~7% of CPU
Siege: ~5% of CPU

With APC disabled: (3 min)

Transactions:                   2256 hits
Availability:                  99.34 %
Elapsed time:                 181.76 secs
Data transferred:               3.15 MB
Response time:                  3.64 secs
Transaction rate:              12.41 trans/sec  <--- !
Throughput:                     0.02 MB/sec
Concurrency:                   45.19
Successful transactions:        2256
Failed transactions:              15  <---
Longest transaction:           30.57  <---
Shortest transaction:           0.14

NOTE: NOT having a PHP accelerator yields about 12 trans/sec! Horrible! Rumor has it that APC will be included by default in PHP6. In the above tests, we also had syck (for YAML parsing). However it didn't significantly change the results on a 3 minute test (less than 0.5 trans/sec).

Rails - 50 concurrent users

With 10 mongrels: (30 min)

Transactions:                 158747 hits
Availability:                 100.00 %
Elapsed time:                1800.82 secs
Data transferred:             211.38 MB
Response time:                  0.57 secs
Transaction rate:              88.15 trans/sec  <---
Throughput:                     0.12 MB/sec
Concurrency:                   49.89
Successful transactions:      158747
Failed transactions:               0
Longest transaction:           11.11  <---
Shortest transaction:           0.00

Mysql: ~10% of CPU
Siege: ~10% of CPU

NOTE: In the test above Rails was configured with 10 stand-alone mongrel processes (production mode) and web requests were redirected through Apache using mod_proxy_balancer. In subsequent 3 minute tests, having 20 or 30 mongrels didn't affect the transaction rate more than 0.5 trans/sec.

With lighttpd+fastcgi: (3 min)

Transactions:                  15415 hits
Availability:                 100.00 %
Elapsed time:                 180.66 secs
Data transferred:              20.53 MB
Response time:                  0.58 secs
Transaction rate:              85.33 trans/sec  <---
Throughput:                     0.11 MB/sec
Concurrency:                   49.89
Successful transactions:       15415
Failed transactions:               0
Longest transaction:            1.25  <---
Shortest transaction:           0.04

NOTE: The above test was run with the default lighttpd.conf, but on port 80 and in production mode. The number of trans/sec was about the same, but the longest transaction time was much smaller!

Django, 50 concurrent users

With mod_python: (30 min)

Transactions:                 224588 hits
Availability:                 100.00 %
Elapsed time:                1800.52 secs
Data transferred:             298.94 MB
Response time:                  0.40 secs
Transaction rate:             124.74 trans/sec  <---
Throughput:                     0.17 MB/sec
Concurrency:                   49.92
Successful transactions:      224576
Failed transactions:               0
Longest transaction:            9.31  <---
Shortest transaction:           0.00

Mysql: ~17% of CPU
Siege: ~11% of CPU

NOTE: The above test was run with PythonAutoReload? Off and PythonDebug? Off (production mode). These are by far the best performance numbers of the three frameworks!

High-load Results

I re-ran 3-minute tests on all three platforms with 150 concurrent users to check high-load results.

Symfony - PHP+APC, 150 concurrent users

Transactions:                  14363 hits  <---
Availability:                  98.68 %
Elapsed time:                 183.12 secs
Data transferred:               3.72 MB
Response time:                  1.34 secs
Transaction rate:              78.43 trans/sec
Throughput:                     0.02 MB/sec
Concurrency:                  105.06
Successful transactions:         122  <---
Failed transactions:             192  <---
Longest transaction:           30.67
Shortest transaction:           0.02

NOTE: You can see there were very few successful transactions. Most were "500 - Internal Server Errors" and 192 failed to connect altogether. It looks like the 500 errors were caused by something with Mysql, because the dev interface returned: "connect failed Native Error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock"

Rails - 10 mongrels, 150 concurrent users

Transactions:                  14718 hits
Availability:                 100.00 %
Elapsed time:                 180.61 secs
Data transferred:              20.22 MB
Response time:                  1.79 secs
Transaction rate:              81.49 trans/sec  <---
Throughput:                     0.11 MB/sec
Concurrency:                  146.12
Successful transactions:       14718  <---
Failed transactions:               0  <---
Longest transaction:           18.39
Shortest transaction:           0.00

NOTE: "Transaction rate" down, and "Longest trans" time is up (from 50 concurrent users), but no failed transactions.

NOTE: Attempting this test with lighttpd+fastcgi failed. The web server returned "500 - Internal Server Error"'s and absurd siege result numbers.

Django, 150 concurrent users

Transactions:                  19903 hits
Availability:                 100.00 %
Elapsed time:                 180.78 secs
Data transferred:              26.73 MB
Response time:                  1.35 secs
Transaction rate:             110.10 trans/sec  <---
Throughput:                     0.15 MB/sec
Concurrency:                  148.32
Successful transactions:       19903
Failed transactions:               0  <---
Longest transaction:           20.03  <---
Shortest transaction:           0.01

NOTE: "Transaction rate" down, and "Longest trans" time is up (from 50 concurrent users), but no failed transactions. Still the top performer.

Summary

Rails performed much better than Symfony. And Django performed much better than Rails.