AWS Autoscaling based on SWF Queue Depth

Amazon has great tools for autoscaling. Combined with our recently completed AMI Bakery this should be awesome right?

• Generate an AMI
• Create an auto scaling group that uses that AMI
• Profit!

But... and there's always a but... 

Amazon's autoscaling can't scale based on the metric that we care about: Simple Workflow queue depth. 

I've got a great idea! It's round, and you can put it under heavy things to help them move!

Start from the basics:

• We have a create_instance script that lets you create an instance/ami of a given type. 
• We have a scale script that compares what you asked for to what is running and adds/removes resources as needed. 

What we lack is the ability to trigger that scaling based on something we care about. 

Enter scalecontroller.py. 

This script takes a bunch of command line arguments to let you tune it to the queue you want to watch, number of jobs that trigger increase/decrease, and how big/small the array should get. 

When a scaling event is initiated, it runs scale.py, which then handles the logic of adding/removing. 

How does scale know which AMI to use?

Chef Data Bags!

Each application has 2 data bag items associated with it: artifact and launch-artifact. Each item contains the branch/build numbers for that product.

To use our "portal" example, you would have something like :

knife data bag show portal

id: portal

branch: master

build: 13

knife data bag show launch-portal

id: portal

branch: master

build: 12

The portal data bag show the most recently deployed version. This is what you would use to pre-bundle an AMI, prior to the release of the code to the world.

The launch-portal databag is the one scale looks at to figure out what AMI to load.

We have a standard naming convention for the AMIs. Using Chef attributes and the branch/build data in the data bags, scale can calculate the AMI to use, test for its existence/availability, and act accordingly.