Create a high performance Drupal server for just $30 a month

I was recently in need of a personal server to run some Drupal sites and being a cheap person, set out to get the best bang for my buck in a server. To start out with, forget any shared host servers. They just aren't going to cut it. I looked at several cloud hosting providers and finally decided on amazon based on price and future expansion. After getting everything set up my server is extremely fast and I'm only paying about $30 per month! Here's how I did it.

Calculating the cost

First lets look at the cost.

A small instance of Linux on Amazon EC2 currently costs $0.065 per hour which works out to:

0.065 * 730 = $47.45 per month

This is fine for setting the server up but once the server is configured, purchasing a reserved instance can seriously save some money. I'm going to be running this instance 24x7 since it is a web server and I don't ever want to take it down so I purchased a Heavy Utilization Reserved Instance. Reserved instances are purely an accounting function. You don't actually have to change the instance in any way, just pay the up front fee and get the lower rate. Just be sure to double check the region and get the right region when you purchase the reserved instance.

The heavy utilization reserve instance price works out to:

(0.016 * 8766 + $195)/12 = $28 per month

Add in a couple bucks for ebs storage and transfer costs and you are at $30. Sweet!

Setting up the server on Amazon

I recommend Ubuntu 12.04 (LTS) as it is supposed to be more stable and supported long term. You can find the Amazon AMI instance here: http://cloud-images.ubuntu.com/locator/ec2/
Find the precise 12.04 (LTS) amd64 ebs storage in the region the right region and copy the AMI-ID. That will be needed for launching the EC2 instance.

In the EC2 interface, launch a small instance of the AMI-ID from the previous step. You'll want to set up a security keypair, add an elastic IP address and Add ports 22 and 80 to the default security group.

A small instance has 1.7GB of RAM and 1 EC2 Compute Unit. This should be enough for moderate website traffic but it is easy to ramp the instance up for an additional cost with only a few minutes of downtime.

I'm not going to go into more detail here about starting an instance. Amazon has some excellent documentation at http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html

General Housekeeping

Once logged in to the server lets start with a little house cleaning. This will make sure all the packages are up to date and install some packages we'll need later.

apt-get update
apt-get upgrade
apt-get install emacs vim git-core git-doc rsync unzip patch curl make drush

Install nginx

I've used apache since I started developing for the web a long time ago. I finally decided to try out nginx and will be sticking with it for the near future. So here is what is needed to install it on this server.

apt-get install nginx php5-cli php5-mysql php5-fpm php5-cgi php5-gd php-pear libpcre3-dev
service nginx start
update-rc.d nginx defaults

Install the site settings file. This one is customized for Drupal and is set to listen on port 8080 which is important for Varnish below. If you don't want varnish, either leave off the listen line or set it to 80.

Place this file in /etc/nginx/sites-available/example.com

server {
  listen 8080;
  server_name example.com www.example.com;
  root /var/www/example.com; ## <-- Your only path reference.

  location = /favicon.ico {
    log_not_found off;
    access_log off;
  }

  location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
  }

  # This matters if you use drush
  location = /backup {
    deny all;
  }

  # Very rarely should these ever be accessed outside of your lan
  location ~* \.(txt|log)$ {
    deny all;
  }

  location ~ \..*/.*\.php$ {
    return 403;
  }

  location / {
    # This is cool because no php is touched for static content
    try_files $uri @rewrite;
  }

  location @rewrite {
    # Some modules enforce no slash (/) at the end of the URL
    # Else this rewrite block wouldn't be needed (GlobalRedirect)
    rewrite ^/(.*)$ /index.php?q=$1;
  }

  location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    #NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    fastcgi_pass unix:/tmp/php-fpm.sock;
    #fastcgi_pass 127.0.0.1:9000; # Use this if you didn't change the php-fpm listen.
  }

  # Fighting with ImageCache? This little gem is amazing.
  location ~ ^/sites/.*/files/imagecache/ {
    try_files $uri @rewrite;
  }
  # Catch image styles for D7 too.
  location ~ ^/sites/.*/files/styles/ {
    try_files $uri @rewrite;
  }

  location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
    expires max;
    log_not_found off;
  }
}

Symlink the file to sites-enabled.

cd /etc/nginx/sites-enabled
ln -s ../sites-available/example.com

You may need to bump the memory limit up a bit in php. This depends on how many modules are installed on your site.
Next we need to set up some php settings. edit the file /etc/php5/fpm/php.ini

memory_limit: 256M

With the nginx config file we need to set another setting in php.ini

cgi.fix_pathinfo=0

Next we want to switch where php-fpm listens from the IP address to sockets. This is slightly faster and matches the site configuration below.
In /etc/php5/fpm/pool.d/www.conf change

listen = /tmp/php-fpm.sock

Next install pac caching for php

pecl install apc

Add apc settings by creating the file /etc/php5/conf.d/apc.ini and put the following in it:

extension=apc.so
apc.shm_size = 256M
apc.apc.stat = 0

Install upload progress and drupal isn't happy without it.

pecl install uploadprogress
echo 'extension=uploadprogress.so' > /etc/php5/conf.d/uploadprogress.ini

Install MariaDB

This is another big change from me as I've always used Mysql. MariaDB is a drop in replacement but made by the non-Oracle founders of Mysql and with lots of improvements for speed and freedom.
Create the file /etc/apt/sources.list.d/MariaDB.list and put in it:

# MariaDB repository list - created 2012-07-11 21:37 UTC
# http://downloads.mariadb.org/mariadb/repositories/
deb http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu precise main
deb-src http://ftp.osuosl.org/pub/mariadb/repo/5.5/ubuntu precise main

Then install the key

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xcbcb082a1bb943db

Finally, use Apt to install mariadb

apt-get update
apt-get install libmariadbclient-dev libmariadbclient18 libmariadbd-dev libmysqlclient18 mariadb-client mariadb-client-5.5 mariadb-client-core-5.5 mariadb-common mariadb-server mariadb-server-5.5 mariadb-server-core-5.5 mariadb-test mariadb-test-5.5 mysql-common

Be sure to set and save your root password.

Install Varnish

The next step is we want a reverse proxy cache for anonymous visitors so nginx and php can get a break for repeated tasks.

Create the file /etc/apt/sources.list.d/varnish-3.list and put the following in it:

deb http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0

Then install the key

curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -

or this if you are running everything through sudo

sudo curl http://repo.varnish-cache.org/debian/GPG-key.txt | sudo apt-key add -

Finally, us Apt to install varnish

apt-get update
apt-get install varnish

Now we need to set varnish up to run on port 80 and nginx to run on port 8080.

Edit /etc/default/varnish and change

DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

to

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

Next create a file at /etc/varnish/default.vcl and put in the following. This one is a generic drupal one that I've created from several sources and have had good luck with.

# A drupal varnish config file for varnish 3.x
#
# Will work with Drupal 7 and Pressflow 6.
#
# Default backend definition.  Set this to point to your content
# server. We are assuming you have a web server running on port 8080.
#
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}
#
sub vcl_recv {
  if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For =
	req.http.X-Forwarded-For + ", " + client.ip;
    } else {
    set req.http.X-Forwarded-For = client.ip;
    }
  }

  if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
  }
  if (req.request != "GET" && req.request != "HEAD") {
      /* We only deal with GET and HEAD by default */
      return (pass);
  }

  # Always cache things with these extensions
  if (req.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf)$") {
    return (lookup);
  }

  # Do not cache these paths.
  if (req.url ~ "^/status\.php$" ||
      req.url ~ "^/update\.php$" ||
      req.url ~ "^/cron\.php$" ||
      req.url ~ "^/install\.php$" ||
      req.url ~ "^/ooyala/ping$" ||
      req.url ~ "^/admin" ||
      req.url ~ "^/admin/.*$" ||
      req.url ~ "^/flag/.*$" ||
      req.url ~ "^.*/server-status$" ||
      req.url ~ "^.*/ajax/.*$" ||
      req.url ~ "^.*/ahah/.*$") {
    return (pass);
  }

  ## Remove has_js, toolbar collapsed and Google Analytics cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js|Drupal.toolbar.collapsed|Drupal.tableDrag.showWeight)=[^;]*", "");

  ## Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

  ## Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
  }

  ## fix compression per http://www.varnish-cache.org/trac/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;
    }
  }

  # If they still have any cookies, do not cache.
  if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
  }

  # Don't cache Drupal logged-in user sessions
  # LOGGED_IN is the cookie that earlier version of Pressflow sets
  # VARNISH is the cookie which the varnish.module sets
  if (req.http.Cookie ~ "(VARNISH|DRUPAL_UID|LOGGED_IN)") {
    return (pass);
  }

  return (lookup);
}

sub vcl_hash {
  hash_data(req.url);
  if (req.http.host) {
    hash_data(req.http.host);
  } else {
    hash_data(server.ip);
  }
  return (hash);
}

sub vcl_deliver {
  # From http://varnish-cache.org/wiki/VCLExampleLongerCaching
  if (resp.http.magicmarker) {
    /* Remove the magic marker */
    unset resp.http.magicmarker;

    /* By definition we have a fresh object */
    set resp.http.age = "0";
  }

  #add cache hit data
  if (obj.hits > 0) {
    #if hit add hit count
    set resp.http.X-Varnish-Cache = "HIT";
    set resp.http.X-Varnish-Cache-Hits = obj.hits;
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }

  return (deliver);
}

The next step is to tell nginx to run on port 8080.
Edit all the files in /etc/nginx/sites-enabled and make sure that they are listening on port 8080. It should look something like this:

server {
	listen 8080;

Then restart nginx and then varnish

service nginx restart
service varnish restart

The last step is to make sure that the varnish module is downloaded to sites/all/modules and in your drupal settings.php files you have the following:

$conf['cache_backends'][] = 'sites/all/modules/varnish/varnish.cache.inc';
$conf['cache_class_cache_page'] = 'VarnishCache';
$conf['page_cache_invoke_hooks'] = false;
$conf['reverse_proxy'] = true;
$conf['cache'] = 1;
$conf['cache_lifetime'] = 0;
$conf['page_cache_maximum_age'] = 21600;
$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
$conf['omit_vary_cookie'] = true;

For more information on these options, see http://drupal.org/node/1196916

Install memcache

Memcache will improve the performance of php by moving all the caching from sql tables to memory. If you have a lot of logged in users it will help a lot.

Start with some installs

apt-get install memcached libmemcached-tools memstat php5-memcached

 

Make sure that the memcache module has been downloaded to sites/all/modules and then in your drupal site's settings.php put the following:

Drupal 6

$conf['cache_inc'] ='sites/all/modules/memcache/memcache.inc';
$conf['memcache_key_prefix'] = 'something_unique';

Drupal 7

$conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_key_prefix'] = 'something_unique';

For more information on configuring memcache for drupal see http://drupal.org/node/1131468

Then make sure everything is loaded properly by restarting everything:

service nginx restart
service varnish restart
service mysql restart
service php5-fpm restart
service memcached restart

At this point you should be good to go. Put websites in /var/www/example.com and set up a site configuration in /etc/nginx/sites-available and symlink to /etc/nginx/sites-enabled.

I'd love to hear any other suggestions for improvements to this server. I haven't done any mysql or nginx tuning yet so I'm sure there are some improvements there. Let me know in the comments below.

Photo courtesy of http://www.flickr.com/photos/pikerslanefarm/3226088712/