typesafe

Recently, I stumbled upon Joel Spolsky’s Test which measures the quality of a software development team. We might argue whether the results of these tests are important indicators or not, but one cannot deny that these 12 questions actually suggest good development practice.

One really important thing in my (and my organization’s) development practice was missing - a one step build - see #2 in the Joel’s test. Digging a bit deeper, one can realize that it is a dependency for #3 as well.

So what?

Let’s think about the “build” process that consists of multiple manual steps. It basically involves compiling source files (not the case of interpreted languages - PHP, for instance), copying files somewhere (on local disk or remote server), adjusting configuration files. That’s the minimum. In case files need to be writable by the application/web server, you’d have to chmod your files/directories as well. Quite a mess isn’t it? If you have to do this for just one server - say production - you’re lucky but that scenario is very, very unrealistic. You might want to deploy your application to some test environment - like virtual machine or other server etc. And of course, we cannot neglect the fact that people make mistakes - typos, forget things etc. Now THAT is really a mess.

Cleaning up the mess

I found two most likely scenarios of deploying a web application:

1. Deployment or re-deployment to production.
   1. Checking out the source from the SCM system
   2. Compilation if needed
   3. Generation of configuration files (important directories, database access etc.) from some given template
   4. Actual file copying to the server
   5. Changing necessary permissions, invoking other commands on the production server
2. Deployment or re-deployment for testing
   1. Compilation (if needed)
   2. Re-generation of configuration files for the same purpose that in 1.3
   3. File copying to the server

Both scenarios assume that necessary environment (databases, virtual hosts) are already configured. That is a one-time job and should be documented somewhere, configuration file template, for instance.

I started to roll out my own script in Python but then I realized that this is a right job for the Apache Ant. It doesn’t matter whether your project is Java or not, it’s really a swiss army knife for everything. Surely, using Ant you introduce another two dependencies to your build process - Java RE and Ant it self.

I’m not going to post my full build.xml file here, I’ll just mention a few issues I had to solve.

• Ant doesn’t have any stock instruments that check out your source code. For SVN (my case) there’s the corresponding external Ant task available.
• I don’t use FTP for copying files to the server, neither mount any network filesystems. I use SFTP/SSH/SCP to copy files and didn’t want that to change. Ant has optional scp and sshexec tasks but they depend on a JSch library that is a SSH2 implementation in pure Java. By the way, one gotcha here: I found that some versions of JSch library do not work - the execution process just stalls and that’s all. I can confirm though that scp and sshexec tasks work with JSch v. 0.1.29.
• For config file generation, I opted for FMPP which is based on Freemarker template engine. My config file templates are actually Freemarker templates. This library has an Ant task, hence one can fill his/her template with data directly from ant build file properties.
• For file copying, I found a huge speed boost when using rsync together with ssh. Actually without rsync testing, i.e., regular (at most 5 minute interval) re-deployment for testing purposes wouldn’t be possible. Rsync copies just those files that have been actually changed, not all of them. Ant does not have any “rsync” tasks, neither 3rd-party or optional, so the only solution I found was using ant core’s exec task. You will have to overcome a few issues though:
  • You make rsync to use ssh by using the rsync’s -e ssh switch. By default, it makes the process asking for your remote users’ password. Since processes launched by ant are in the non-interactive mode, there’s no way how you can enter this password. Luckily, there is a way how to log in to the remote system via ssh without asking a password - check out some tutorials, this for instance.
  • RSync is not available on Windows systems. In that case, you have to resort to Cygwin. It also involves configuring your file system paths so that cygwin can see them - e.g. adding /cygwin/c/ prefix instead of regular C:/. Otherwise, rsync from exec task will complain that it won’t copy files between two remote filesystems - apparently, colon after C for rsync is confusing. That is achievable from ant - I used condition and replaceregexp tasks
• I don’t keep actual config files in SCM any more. Instead, I keep config file templates and I edit only them. Even more, I added svn:ignore to all the generated files. And forget about editing  config files manually.
• You might consider re-structuring your app so that server-writable directories are outside the directory you deploy your application, i.e., outside the htdocs or public_html directory. It is a matter of environment configuration to make those files reachable by URL from outside directly if needed - see the Alias directive in case of Apache HTTPD web server.

Final notes

I still haven’t made such a build for database, since it does not change so often and its deployment (at least in my case) is very easy.

If you haven’t created your one-step (or as-few-step-as-you-can) build then I urge you to do so. It took me a few days but immediately after that I felt how rewarding it is.