Project Uptime : Progress Report 5 : Getting ready for Reddit and Hacker News

A very timely post on Hacker News by Ewan Leith about configuring a low end server to take ~11million hits/per month gave me some more ideas on optimizing the performance of this website. Ewan used a combination of nginx and varnish to get the server to respond to such traffic.

From my earlier post, you might recall, that I planned on checking out nginx as the web server, but then ended up using Apache. My earlier stack looked like this Based on the recommendations from Ewan’s article, I decided to add Varnish to the picture. So here is how the stack looks currently

And boy, did the performance improve or what. Here are some before and after performance charts based on a test run from blitz.io. The test lasted for 60 seconds and was for 250 simultaneous connections.

BEFORE

• Screenshot of Response times and hit rates. Note that the server essentially stopped responding 25 minutes into the test.
• Screenshot of the analysis summary. 84% error rate!!

AFTER

• Screenshot of response times and hit rates
• Screenshot of summary of Analysis. 99.98% success rate!!

 

What a difference!!.. The server in fact stopped responding after the first test and had to be hard rebooted.  So how did I achieve it? By mostly copying the ideas from Ewan :). The final configuration for serving the web pages looks like this on the server end

Varnish (listens on TCP 80) –> Apache (listens on TCP 8080)

NOTE : All the configuration guides (as with the previous entries of the posts in this series) are specific to Ubuntu.

1. Configure Apache to listen on port 8080
1. Stop Apache [code] sudo service apache2 stop [/code]
2. Edit the following files to change the default port from 80 to 8080
1. /etc/apache2/ports.conf
1. Change [code]NameVirtualHost *:80
   Listen 80
   [/code]
2. to [code]NameVirtualHost *:8080
   Listen 8080
   [/code]
2. /etc/apache2/sites-available/default.conf (NOTE: This is the default sample site that comes with the package. You can create a new one for your site.  If you do so, you need to edit your site specific conf file)
1. Change [code] <VirtualHost *:80> [/code]
2. To [code]<VirtualHost *:8080> [/code]
3. Restart apache and ensure that it is listening on port 8080 by using this trick.
2. Install Varnish and configure it to listen on port 80
1. Add the Varnish repository to the system and install the package[code]sudo curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add –
   sudo echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-3.0" >> /etc/apt/sources.list
   sudo apt-get update
   sudo apt-get install varnish
   [/code]
2. Configure Varnish to listen on port 80 and use 64Mb of RAM for caching. (NOTE: Varnish uses port 8080 to get to the backend, in this case Apache, by default. So there is no need to configure it specifically).
1. Edit the file /etc/default/varnish
1. Change [code]DAEMON_OPTS="-a :6081 \
   -T localhost:6082 \
   -f /etc/varnish/default.vcl \
   -S /etc/varnish/secret \
   -s malloc,256m"
   [/code]
2. To [code] DAEMON_OPTS="-a :80 \
   -T localhost:6082 \
   -f /etc/varnish/default.vcl \
   -S /etc/varnish/secret \
   -s malloc,64m"
   [/code]
3. Restart Varnish [code]sudo service varnish restart[/code]

   and you are ready to rock and roll.

There are some issues with this setup in terms of logging. Unlike your typical web server logs, where every request is logged, I noticed that not all the requests were being logged. I guess, that is because varnish is serving the content from cache. I have to figure out how to get that working. But that is for another post :).