PHP

A development virtual machine can be really handy. It gives you a sandbox of sorts where you can feel free to test and experiment knowing that in a worst-case scenario you can just delete the VM and start over. It can also be a great way to practice server configurations and sketch out "real-world" server setups. Here's the process I follow to setup my Ubuntu 10.04 LTS Drupal development VM in VirtualBox. Aside from the VM-specific steps, these instructions should work for a regular Ubuntu server (VM or not).

Download and install Ubuntu 10.04

Note: Unless otherwise specified, all commands below should be run on the guest VM (Ubuntu).

If you haven't do so already, download and install VirtualBox. Then, download the Ubuntu 10.04 LTS disk image to your host computer.

Next, Create a new virtual machine in VirtualBox. Be sure to select type of "Linux" and OS Type of "Ubuntu" and 32 or 64 bit, accordingly.

Start the machine and, when prompted to select a disk drive, click the icon to the side and open the Virtual Media Manager. Add the ISO file you downloaded to the list of disk images and select it as the media to mount. Walk through the install wizard. When asked to install software, don't choose any packages. We'll add them later. When asked to install the GRUB boot loader, go ahead and choose "yes." Once installation is complete, click on the CD icon in the VM window and unmount the disk image, then choose "Continue" to reboot into the Ubuntu server VM.

Update, upgrade, and setup SSH

Before we do anything else, let's get comfortable in the environment. Working in the small shell window that VirtualBox provides isn't ideal, and it'd be nice to be able to do the rest of the setup over SSH in the terminal of our choice. Install the OpenSSH Server package, upgrade existing packages and then shutdown the server (because we'll have likely updated "linux-headers-server" and other core OS packages, and also so we can do the next step).

If you'd like to make things easier on yourself and not have to type "sudo ..." for every command, run sudo -u root -s to become root for the rest of the session. Otherwise, be sure to add "sudo" before each of the follwing commands.

apt-get install openssh-server
apt-get update
apt-get upgrade
apt-get dist-upgrade
shutdown -h now

Get in the habit of running apt-get update and apt-get upgrade regularly. It's one way you keep your Ubuntu server patched and secure! apt-get dist-upgrade should be run if you see "The following packages have been kept back..." when you run apt-get upgrade. This means that some packages haven't been upgraded because they involve core, operating system-level updates that require a restart of the machine.

Configure networking on the host machine

On the host machine, run the following commands below. The first line sets up a handy environment variable you can use later. Just replace "Ubuntu 10.04 LTS" with whatever name you gave your VM when setting it up in VirtualBox. The commands below assume you haven't changed your network adapter type in VirtualBox from the default. If you have, you'll likely need to change the device name.

ubuntu_vm="Ubuntu 10.04 LTS"
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guestssh/Protocol" TCP 
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guestssh/GuestPort" 22 
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guestssh/HostPort" 2222
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guesthttp/Protocol" TCP 
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guesthttp/GuestPort" 80
VBoxManage setextradata "$ubuntu_vm" "VBoxInternal/Devices/e1000/0/LUN#0/Config/guesthttp/HostPort" 8888

Now, start the VM back up. You should now be able to SSH into your Ubuntu VM as follows:

ssh localhost -p 2222

Install Apache, PHP and MySQL

Now, install some generally useful packages that will be used for the rest of the process.

apt-get install build-essential git-core subversion cvs python-software-properties curl

Before we get to the web server setup, we need to change Ubuntu's package definitions a bit. Ubuntu 10.04 LTS is configured to pull PHP 5.3.x, but many Drupal 6.x contrib modules aren't compatible with PHP 5.3, so we need to downgrade to PHP 5.2.x. We'll follow some instructions from Khalid at 2bits. We'll take his Approach #3 by adding Ralph Janke's PHP 5.2 repository for Lucid and "pinning" the PHP version to 5.2, to make sure we don't inadvertantly upgrade to 5.3 (I've provided the apt preferences file in my Github repository, for easy access).

add-apt-repository ppa:txwikinger/php5.2
wget http://github.com/jrbeeman/drupal-patches/raw/master/ubuntu-10.04-apt-php-prefs.txt -O /etc/apt/preferences.d/php
apt-get update

Now, we'll install the web server, PHP packages (including APC), and MySQL. Please note that you'll be prompted for a MySQL root user password. The last command below also disables some unneeded Apache modules and enables a few that are good to have for Drupal.

apt-get install apache2 php5 php5-cli php5-gd php-pear php5-dev apache2-dev
apt-get install mysql-server php5-mysql
a2dismod cgi autoindex
a2enmod deflate expires rewrite vhost_alias
pecl install apc

Configure Apache and PHP

Update PHP settings by editing both /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini and setting the variables below as shown:

safe_mode = Off
expose_php = Off
memory_limit = 128M
display_errors = Off
log_errors = On  
error_log = /var/log/php/php.log  ; (or php_cli.log)

Update APC settings by editing /etc/php5/conf.d/apc.ini:

extension=apc.so
apc.enabled=1
apc.enable_cli=1
apc.shm_segments=1
apc.shm_size=32
apc.cache_by_default=1
apc.stat=1

Create the error log files:

mkdir /var/log/php
touch /var/log/php/php.log
touch /var/log/php/php_cli.log
chown -R www-data:www-data /var/log/php
chmod -R 0755 /var/log/php

Due to the way Drupal handles URLs, in particular how it handles creating a URL to itself, we need to tell Apache to listen on port 8888. This gets around the "HTTP Request Fail" issue in the status report, among other things:

Edit /etc/apache2/ports.conf and add Listen 8888, so that the beginning of the file looks like:

NameVirtualHost *:80
Listen 80
Listen 8888

Install and configure Drush

Setup Drush and Drush Make to be used server-wide. The commands below grab the latest verison as of this writing, but you can check the Drush and Drush Make project pages to see if there are newer releases:

cd /usr/local/src
wget http://ftp.drupal.org/files/projects/drush-6.x-3.3.tar.gz
wget http://ftp.drupal.org/files/projects/drush_make-6.x-2.0-beta8.tar.gz
tar xzf drush-6.x-3.3.tar.gz
tar xzf drush_make-6.x-2.0-beta8.tar.gz
mkdir /etc/drush
cp /usr/local/src/drush/examples/example.drushrc.php /etc/drush/drushrc.php
ln -s /usr/local/src/drush/drush /usr/local/bin/drush

Edit /etc/drush/drushrc.php to add Drush Make to include path:

$options['i'] = '/usr/local/src/drush_make';

Install Drupal

Create a database, download, and install Drupal (be sure to put a password in for the MySQL grant statement where appropriate below). I like to keep my sites in sub-directories of /var/www. These commands will prepare Apache for that working setup. I've provided a couple of files that are downloaded with wget to make this simpler, but feel free to modify them along the way if you wish.

mysql -u root -p -e "CREATE DATABASE drupal_dev; GRANT ALL PRIVILEGES ON drupal_dev.* TO drupal@localhost IDENTIFIED BY 'password'; FLUSH PRIVILEGES;"
wget http://github.com/jrbeeman/drush-make/raw/master/drupal-demo.make -O /var/www/drupal-demo.make
wget http://github.com/jrbeeman/drupal-patches/raw/master/ubuntu-drupal-demo_apache2-conf -O /etc/apache2/sites-available/drupal-demo
a2dissite 000-default
a2ensite drupal-demo
chown -R www-data:www-data /var/www
sudo -u www-data drush make /var/www/drupal-demo.make /var/www/drupal-demo
sudo -u www-data cp /var/www/drupal-demo/sites/default/default.settings.php /var/www/drupal-demo/sites/default/settings.php
/etc/init.d/apache2 restart

Walk through Drupal's install wizard by visiting http://localhost:8888. Once you're done, run the following Drush command to quickly enable the modules in the demo site:

cd /var/www/drupal-demo
drush en -y admin \
  content content_copy fieldgroup filefield imagefield nodereference \
  number optionwidgets text userreference bulk_export ctools \
  ctools_custom_content page_manager views_content path search devel \
  features imageapi imageapi_gd imagecache imagecache_ui masquerade \
  pathauto skinr strongarm token panels_mini panels_node panels \
  jquery_update vertical_tabs views views_bulk_operations views_export \
  views_ui fusion_core

Wrap-up

You're done! From here, you've got a fully-functional Ubuntu virtual machine designed for developing Drupal sites.

Todo

There are some issues to work out and other tasks to add to these instructions. Any feedback on these items would be greatly appreciated.

Issues

• Drupal reports that it can't make HTTP requests, but it can. I believe this is because we're making requests over port 8888, but Drupal thinks it's running on port 80. Not sure how to get around this, but also not sure it's really all that important for development environments.

Tasks

• TODO: Install and configure Aegir
• TODO: Install and configure Varnish
• TODO: Install and configure Solr

References

• Running PHP 5.2 on Ubuntu 10.04 (2bits)
• Preparing a server for Aegir (groups.drupal.org)
• Install PHP APC (2bits)
• Optimizing APC for Drupal (Xerosphere)

I've put up on Google Code a small new project that I'm working on called the PHP Gamercard API. It's basically a set of classes for working with Xbox Live Gamercard data. It takes advantage of a web service provided by Duncan Mackenzie for retrieving structured data about individual Gamertags. You can expect to see something in Drupal that takes advantage of this, once I've got some time to write a module that utilizes it.

Update:
Thanks to an anonymous commenter for letting me know that this is an issue of operator precedence.

This is more a personal note than anything, but I've been banging my head against a wall trying to figure out how to reference an array within a variably-named object property in PHP. Having not found anything very useful when search Google, I figure my post may end up being someone's helpful search result. Maybe I just don't know the right terminology for what I'm trying to do...?

Anyways, I've got a module that needs to modify the string in a CCK text field before it's shown to users on the node edit form. It's a "glue" module that helps us handle course enrollment, and the field in question handles course instructor(s) via a comma-delimited list of usernames. The module takes the user's submitted data, parses it into an array and stores each username in a table joined with course ID for other uses (such as passing to our Sakai installation). The CCK field is referenced in several places, so I use an admin settings form to allow us to say, "This CCK field is the field that users fill out to define instructors." This allows us to avoid hard-coding the CCK field name all over our glue module, but it also led to the headache I encountered today.

When a user goes back to the form to edit the course, I want to present the username list cleanly (alphabetical, no accidental whitespace, etc). Here's the code I tried to use:

function ideal_courses_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  // ... (irrelevant stuff here)
 
    case 'prepare':
      $field_instructors = variable_get('ideal_courses_field_instructors', NULL);
      // The line that fails is below:
      $node->$field_instructors[0]['value'] = ideal_courses_instructors_as_string($node);
    break;
 
  // ... (more irrelevant stuff here)
}

The error I kept getting was, "PHP Fatal error: Cannot use string offset as an array." The strange thing is that doing a print_r($node->$field_instructors); works fine. The fix was simple, but difficult to find: wrap the variable property name in curly braces: $node->{$field_instructors}[0]['value']. The full result is below:

function ideal_courses_nodeapi(&$node, $op, $a3 = NULL, $a4 = NULL) {
  // ... (irrelevant stuff here)
 
    case 'prepare':
      $field_instructors = variable_get('ideal_courses_field_instructors', NULL);
      // Fixed line is below:
      $node->{$field_instructors}[0]['value'] = ideal_courses_instructors_as_string($node);
    break;
 
  // ... (more irrelevant stuff here)
}

Now that I had a demonstrably working and functional web server going on my Linode (see Just got rolling with a VPS on Linode (Part 1)), it was time to get the rest of my toolkit on the box, setup users and secure the server a bit.

Installing Subversion and migrating repositories

Well, installing Subversion couldn't be any simpler:

apt-get install subversion

Login to old server and dump current repositories:

svnadmin dump /path/to/repository > repository.dump

Copy dump file to new server, and on new server:

svnadmin create /path/to/repository
svnadmin load /path/to/repository < /path/to/dump/repository.dump

Adding users and groups

I decided I didn't want to be logged in as root all the time, especially since I'll most likely be bringing some other folks in to work on the server in the future. So, I setup the admin group, created myself a new user and put myself in both the admin and staff groups.

addgroup admin
adduser jrbeeman
usermod -G staff,admin jrbeeman

Next, I wanted to make sure admins could sudo to root, so that they could install programs and do other root-y things. The sudoers file, as far as I can tell, can only be edited with the command visudo:

visudo

...and added the line:

%admin  ALL=(ALL) ALL

Setting up the firewall

This was probably the least-traveled territory in the whole VPS setup for me. Thankfully, there is an awesome resource in the website IP Tables Rocks, with a full rundown of how to lock down unneeded ports. It emphasizes locking down everything, and then only opening up those services you want open. I essentially followed the tutorial, but proceeded to lock down every port except those that I knew I would need for web services and working with the server (22, 80, 443, etc.)

Performance

By this point, I've started working on getting the Gamers With Jobs development site migrated over, and I'm working on nailing down any performance issues. As I said in part 1, the main reason for going to a VPS was the sheer size and load on the GWJ site and how shared hosting was really hosing the speed. Most of the tweaks from here on out are related to the GWJ site.

Tweak MySQL settings

Since the Gamers With Jobs site is very database intensive, getting MySQL to perform optimally given the site's load is important. I'm still tweaking these settings here and there, but here's what I'm at so far. I'm attempting to go for large enough buffers and caches to keep things snappy, but without bloating out the caches to the point that things slow down.

#
# * Fine Tuning
#
key_buffer              = 256M
max_allowed_packet      = 16M
thread_stack            = 128K
thread_cache_size       = 8
#max_connections        = 100
table_cache             = 256
thread_concurrency      = 4
sort_buffer_size        = 1M
read_buffer_size        = 1M
read_rnd_buffer_size    = 4M
myisam_sort_buffer_size = 64M
#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M
#
# Turn on slow query logging to help track down performance killers
#
log_slow_queries        = /var/log/mysql/mysql-slow.log
long_query_time = 5
#
# Some further table-type tweaks
#
[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
 
[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

Bringing over the GWJ site required quite a bit of scripting of INSERT and DELETE statements that fudged with table lengths, so I also optimized all the tables with free data space:

-- Get the table names...
SHOW TABLE STATUS WHERE Data_free > 0;
-- ...and run the following for each
OPTIMIZE TABLE table_name;

Tweak Apache settings

The YSlow utility from Yahoo is a great way to track down potential end-user performance issues, so I ran it against the GWJ dev site and tweaked quite a few things to improve the rating and speed reported there.

First, I needed to enable a few Apache modules:

a2enmod deflate
a2enmod expires
a2enmod cache

Then, in /etc/apache2/httpd.conf, I added the following lines to the stanza of the GWJ virtual host definition:

# Gzip html, css, js, etc.
AddOutputFilterByType DEFLATE text/html text/css text/plain text/xml application/x-javascript application/json
# Set expires headers on html, css, js, etc.
<IfModule mod_expires.c>
  ExpiresActive On
  ExpiresByType text/html "access plus 1 seconds"
  ExpiresByType image/gif "access plus 1 month"
  ExpiresByType image/jpeg "access plus 1 month"
  ExpiresByType image/png "access plus 1 month"
  ExpiresByType text/css "access plus 1 week"
  ExpiresByType text/javascript "access plus 1 month"
  ExpiresByType application/x-javascript "access plus 1 month"
</IfModule>
# Set ETags
FileETag MTime Size

Install memcached

In order to squeeze a bit more performance out of the server, I decided to install memcached and the related Drupal module, which allows you to configure Drupal to store certain cache data in memory. I essentially followed the instructions in Robert Douglass's article on Lullabot, but with a couple of modifications.

First, libevent1-1.3b and memcached-1.2.1-1 can be installed via apt-get on Ubuntu gutsy, all with:

apt-get install memcached

Then, I enabled the Apache module:

a2enmod mem_cache

Install eaccelerator

Not much to write here, aside from noting that I followed the great article on 2Bits to get going.

Done... sorta

Seeing how I started writing this article a couple of weeks ago and am just getting around to publishing it, I think I'll call it "finished," for now. I hope that someone out there finds this useful!

Note: A large part of this is taken from Victor Kane's article on Awebfactory about setting up Drupal on a fresh Linode, but I've documented some other things here and did some things a little differently than he did, so I figured it'd be worth writing up a post on the process. I've kept the details thin here in places where Victor's notes are more than satisfactory, but I've made sure to note where that happens.

Update: Be sure to check out part 2 of this article, as well.

I've spent the last several months of my off-work hours plugging away at helping the folks over at Gamers With Jobs get rolling with an upgraded version of Drupal, and in the process we decided to move from a shared hosting environment to a place where we've got a lot more control over performance and site configuration. In the meantime, Victor Kane's article on getting Drupal up and running on a Linode came across my RSS reader and provided the kick in the pants I needed to really investigate it. I looked at several VPS options, but in the end Linode seemed to be the best. They offered a seven day money back guarantee, which honestly isn't much, but it was long enough for me to feel comfortable giving it a shot without being out sixty bucks, so I decided to try it out.