AWS, Chef and scaling a mixed Windows/Linux environment (Part 1)

Note: you'll notice that I rarely talk about what company exactly... it's not them, it's me. I prefer to keep that sort of thing somewhat private.

I've been at $COMPANY for a year and a half now. In that time we've gone through a surprising number of architectural changes. Iterative development indeed.

I'm really happy with where we are today, and would like to take some time to write it all down.

In the Beginning

Whenever I hear someone talk, I always like to hear about the road they took to get to this awesome place. I want to know what they tried and hated; I want to hear about all the failures. I think there's a lot to learn there. So, I'm going to start from my beginning and work my way up to what we are doing now.

When I first came here, the infrastructure was all AWS, managed via RightScale. We weren't really using RS correctly though. We weren't letting instances inherit configurations from deployments, we weren't making smart use of templates, we were achieving a LOT of our functionality as shell code within templates, rather than using their engines.

Deploying software was a nightmare.

We had 2 different core applications (there was a lot more going on than that, but we'll stick to just the main in-house developed stuff).

One was Python, the other a Grails app in the form of a war file.

Python app, running on Windows and Linux.

This was deployed from SVN. The RightScale config included the desired SVN revision. Since we weren't using deployment inheritance, you had to go to each instance and update them one at a time.

The process for deploying code differed wildly from Windows to Linux, even though they were both from the same Python repo.

On the Windows side you would RDC to an existing box, svn up to the revision, manually edit any config files, bundle an AMI, launch new instances and terminate the old ones.

On the Linux side you would update the RightScale template variables and run the deployment script to execute the changes. The scripts were basically shell scripts running a lot of "sed s/this/that/g"

Grails app

The war file was downloaded from S3. Each environment (dev1, dev2, qa, prod, etc...) had a bucket, and the deployment script downloaded the artifact from that. The war files were copied to the buckets by hand, along with the config templates which were sed processed like the python code.


Next: First pass at automation