AWS, Chef and scaling a mixed Windows/Linux environment (Part 4)

The AMI Bakery

Here's the part that I really wanted to write about.

We decided it was time to invest in the ability to generate fully configured AMIs for our applications.

The Roles Cookbook

Back to the roles cookbook from a prior post. Each role had a "role_launch" recipe that did all the things necessary to take a base CentOS/Windows AMI and turn it into that thing. We now added a role_ami recipe that was very similar, but with some important differences:

• nothing would start or be run during creation: db migrations, service starts, etc... would be set to run at boot time, but not now. 
• the ami_prep recipe would install a startup script that would allow the instance to register itself with the chef-server when it came online. 

AMI Prep

Windows. Linux. userdata. So many different ways to do things, and none of them quite as controlled or specific as we wanted. 

Like so many other wheels, I reinvented it. 

• Each platform has an init script. For linux it's a standard SysV init style, and for Windows it's a powershell script that runs at boot via task scheduler. 
• The script only runs on the initial boot. It looks for a file firstboot.txt in the root volume and only runs if it is present. On completion of the run the file is removed. 
• Both use an sns handler recipe to generate an alert if the run fails. This way we only hear about it when something bad happens.  
• In both cases the instances use IAM roles and AWS command line tools to fetch a default runlist from an S3 bucket and execute chef-client. 

Adding the --bake flag

Our main tool for instance creation is the appropriately named "create_instance.py" script.

Previously this script would:

• create/update security groups
• create/update IAM roles
• create instance
• bootstrap chef / run the role_launch recipe

With the new --bake flag it could now:

• create/update security groups
• create/update IAM roles
• create instance
• bootstrap chef / run the role_ami recipe
• stop the instance
• take an ami of the instance
• wait for the ami to be available
• terminate the instance
• clean up the chef node objects

Post deploy task

Immediately post-deploy, a job in Jenkins is triggered to bundle new AMIs for the current release. That way if we need to scale rapidly, we have an image. More on that when I post about scaling.

Previous: Improvements

Automated Chef Cookbook Testing. How? Why? Help! Part 1 of many!

As part of the DevOps Promised Land, we decided that it was time to move Chef out of Operations and into every developer's hands. 

When I started at my current job, we did pretty much everything manually. I spent most of my first year here building up a reasonable infrastructure management system based on Chef, and getting all 200+ hand crafted EC2 instances rebuilt in an automated fashion.

Now that we've gotten pretty happy with where we are at, it's time to put the Dev in DevOps. No more opening tickets to add cookbook attributes! Self-service equals faster response time and less work for me. Let's do this!!

I use the term "fleet" throughout this post... you can replace that with stack, environment or whatever other term you use for a collection of servers that make up an instance of your application.

Before we get into details, lets talk about the high level goals:

• All developers can modify cookbooks.
• All cookbooks must have appropriate tests built in.
• All cookbooks must have an automated "build" process that runs the tests at every commit. 
• No cookbook changes are pushed to the Chef server until they pass all of their tests.
• Cookbook changes are applied to all fleets (including prod!) as soon as they pass their tests.

Here's where we were at the starting point of this project. 

Our testing system was there, but it was manually run.

We were using the minitest-handler cookbook, and had reasonable test coverage. I won't lie to you and say that it was 100%, but it's close.

Every cookbook had a Vagrantfile, and used Berkshelf to maintain dependencies (via the vagrant-berkshelf plugin)

You made a change, and it was on you to vagrant up / make sure the tests pass. Once it's all good, you would commit the code.

Jenkins would get notified of the commit. It would check out the code, and do some REALLY basic tests:

• knife cookbook test
• foodcritic
• knife testcoverage

That last one is a crappy little knife plugin I wrote that would look at each recipe and ensure that there was an _test.rb recipe associated with it. That doesn't mean there were any useful tests, just that you had at least taken the time to create the test file. 

If everything passes it would do a knife cookbook upload. 

When I was the only developer writing code, this was a workable system.  I knew what I changed, when I changed it, and what the impact would be. I could troubleshoot problems very quickly and push fixes within minutes of bug detection.

As the team grows, this falls apart:

• The prod and qa environments are pinned to versions, but dev is running head. broken code breaks all 5 dev fleets!
• Cookbook versions aren't necessarily frozen. who knows what "1.5.4" actually means? it's totally possible for prod to be "pinned" to a cookbook rev, then for someone to change that version!
• What if i just upload directly from my workstation without running tests?

We started off by defining some things that would help us get there

• Every cookbook change bumps the version number.
• Every cookbook upload is frozen.
• Every fleet has cookbook versions pinned (except for the chef test fleet).
• Every cookbook should have a build script that encapsulates the logic for building and testing it. That way our Jenkins jobs are simply check out code / run build.sh

The knife-spork plugin by etsy is a great way to manage all of versioning parts of that list. I'll get into the details of how we use all of that later on.

Building a cookbook

Let's walk through a simple cookbook. We have one that we use internally for basic OS stuff: selinux, iptables, package management, etc...

It's a good one for the example because the tests are for the most part super easy:

package "wget" do  action :installend

Gets a test of:

it "Installs wget." do  package("wget").must_be_installedend

test-kitchen

At first we were using vagrant-berkshelf to tie things together. The 1.5.1 release of Vagrant broke the plugin, and a bit of reading made me feel that test-kitchen was a better choice anyway. 

So first up was adding test-kitchen to the cookbook (and removing Vagrant)

rm Vagrantfile 

kitchen init

And there you go! Now we need to customize the .kitchen.yml

• We use centos-6.4, so I took the ubuntu platforms entry out. 
• Add the runlist
• Attributes.... uh-oh!!!

.kitchen.yml allows you to specify a list of chef attribute to pass. But I don't want to list them in each cookbook! I have an attributes.json file on the Jenkins box that defines all of the credentials that would normally be in an encrypted data bag. Vagrant lets me simply

dna = JSON.parse(File.read("/var/chef/attributes.json"))
chef.json.merge!(dna)

I looked at the test-kitchen docs, but didn't see anything specifically about including a file. I did catch though that the .kitchen.yml file is ERB parsed before being used. Perfect!

I converted attributes.json to yml using one of many tools out there, then added

    attributes:
<%= File.read("/var/chef/attributes.yaml") %>

To my yaml file and it promptly blew up! Since the attributes file is just getting inserted exactly at that point, the indentation has to match up. Shifting the data in the yaml file over by a couple of tabs did the trick.

Kitchen test now works! It boots the VM and processes the runlist, including minitest-handler. We will eventually start using test-kitchen's built in tests but for now we're OK with what we've got.

build script

As I mentioned earlier, we want to have a standardized build script that would be used by all cookbooks. The content would change, but the content would be the same:

• Run whatever tests that cookbook gets
• If the tests pass, upload the cookbook and exit zero
• If the tests fail, exit non-zero.

Our standard set of tests includes:

• knife cookbook test (basic syntax)
• knife spork check --fail (make sure the version number has been bumped)
• kitchen test
• kitchen destroy
• foodcritic
• knife spork upload

To make sure that whoever is running this has a fair chance of success, we include a Gemfile in the cookbook with all of those tools in it. The build script starts off with a bundle install, so as dependencies change, the systems should stay up to date.

The shell script looks at the exit code of each command as it runs, and dies if any of them are non zero.

git-hooks

Every cookbook has a git-hooks directory with recommended scripts to either use or integrate into your existing tools.

The main one that all cookbooks have is a simple pre-commit hook to run knife spork check --fail. This makes sure that you don't try to upload a frozen cookbook version. 

promoting the cookbooks to the fleets

Since we aren't yet running full integration tests, we don't have a really good way to prove that the newly published cookbook plays well with its other friends in the Chef playground. 

For the moment, we are automatically promoting the fleets once a cookbook passes its tests and gets uploaded.

We have a Jenkins job that looks up the latest frozen version of each cookbook and promotes the fleets to that version. Since you don't get a frozen upload without passing your tests, that should be safe enough.

That also allows us to have in flight cookbooks on the chef server without fear -- the fleets will only ever auto promote to the latest frozen revision. 

integration testing

Great! Our cookbook passed its test and has been uploaded. All of the fleets are pinned to the older version though, so no-one is seeing this change.

Now what?

Enter the test fleet. 

We have 1 development environment that does not get anything pinned. It is used only by ops, just for testing new Chef code.

Go forth and deploy / run rubot / do whatever you need to do to feel happy on the test fleet. 

Right now we are doing this by hand, but that will be next on the automation to-do list.

Getting Chef + RVM playing nicely together

I know that all of maybe 3 people read this blog. I typically forget about it myself. But, I wanted to jot this down someplace, and here seemed as good as any.

We've recently started to move into the world of Rails. We use Chef to manage our infrastructure. There are some quirks to getting them all lined up right, and finding a straight up tutorial is surprisingly tricky. So, here goes:

• Install Chef via Omnibus as recommended by them.
• Get the RVM cookbook, maintained by fnichol.
• Pick a non-root user to run your rails app(s) as. 
• Install RVM/Ruby as that user, via the user recipe in the cookbook.

Any place in your other cookbooks that you are going to call ruby scripts (bundle install, rake db migrate, etc...):

• include rvm::default
• wrap the commands with the rvm_shell LWRP from the rvm cookbook. This ensures that the user's rvm environment is inherited properly. 

Here's an example of the recipe I'm using to install RVM:

node.normal['rvm']['user_installs'] = [
    {
        'user'  => node['fleet']['systemuser'],
        'default_ruby' => node['rvm']['default_ruby'],
        'global_gems' => [
            { 'name'    => 'bundler' },
            { 'name'    => 'rake' }
        ]
    }
] 

include_recipe "rvm::user"

I'm using normal for the attributes to ensure that the defaults in the rvm cookbook are overridden. In my case node['fleet']['systemuser'] and node['rvm']['default_ruby'] are defined elsewhere in the run list.


Then some code (bundle install)

include_recipe 'rvm::default'

rvm_shell "Running Bundle install" do
  user node['fleet']['systemuser']
  group node['fleet']['systemgroup']
  ruby_string node['rvm']['default_ruby']
  cwd node['app']['dir']['deploy']
  environment 'RAILS_ENV' => node['app']['rails_env']
  code %{bundle install}
end

 Again, some attributes are defined elsewhere, but you should be able to get the idea.

This may not be perfect, but it works for me!

Other gotchas:

If you try to install RVM system-wide, chef-client gets sad. You end up changing the environment such that the omnibus ruby is no longer what env ruby finds, so you end up needing to install chef gems into your RVM ruby.

Ugly!

Chef-client always runs as root, and your app should never run as root. So, there shouldn't really be any conflicts here.

I'm not good at Vegasing

I'm in Las Vegas for the amazon web services re:invent conference. This is the stuff of corporate dreams: a private suite in the Venetian. Late starts and early endings each day. Sponsored pub crawl featuring free drinks at all of the resort bars.

There's just one problem: I don't drink. Or gamble.

So what to do?

I've been getting up early to talk to my daughter before school. I'm going to bed around 9 and getting up at around 4.

I can only imagine how maddening this is to my friends.

Tonight is the big party hosted by Intel. I'll go for a while, but I imagine that it will largely be about the copious free booze, and that I'll get bored.

Rock of Ages is in my hotel. I've seen it in Boston and on Broadway. Maybe I'll go again ;)

Home in 2 days.

A bit about metrics gathering and Sensu

Sensu has a well documented method for running checks that push metrics to Graphite.

In some cases I opted to do things a bit differently.

One of my checks hits a message queue and gets me the number of items in the queue and the age of the oldest one.

I want to both alert on that (if something has been in the queue for more than X minutes) and graph it (how did we do this week on processing everything in under Y minutes), without making the same query 2 times.

In those cases my checks bypass the Sensu graphite handler and just send the metrics directly to Graphite.

I like that the system is flexible enough that this was easy to do.

Oh Windows...

One thing that makes me sad regularly is that we have Windows instances. Lots of them. And they are important.

Sensu does support Windows, but the documentation is sparse. You pretty much have to either use their Chef cookbook or read it to get the steps required to install the client.

Similarly, go visit the Sensu Community Plugins project and see how many of them cat /proc/somethingorother to get data. No Windows for you!

I ended up writing a couple of Ruby scripts that would get lists of processes, system load and the like on both Linux and Windows platforms. I need to clean them up a bit then submit them back to the project.

It wasn't hard to do, but it is a thing to be aware of if you go down this road.

Collectd, Graphite, Tasseo, Etsy Dashboard, Vacuumetrix

My current visualization suite is made up of 4 tools:

Graphite

The open source graphing darling serves us well. Feed it metrics, make API calls to turn them into graphs. Do one thing and do it well!

Collectd

Collectd runs on every instance, gathering system stats and reporting them to graphite. We went with Collectd because right out of the box it did about 90% of what we wanted, and the remaining 10% was easily addressable via some simple Python based plugins.

This is how disk capacity, free memory, load and other basic health checks get sent to graphite.

Vacuumetrix

Vacuumetrix pulls stats out of AWS and feeds them into graphite. Good for looking at Elastic Load Balancers, RDS instances and other things where we can't install our own tools.


Etsy Dashboard

We used the Etsy Dashboard project to make some generic "Here's how this instance looks" pages. Go to the dashboard for prod-loghost and you'll see disk I/O, free mem, load, network in/out and all the other things that people like to look at when assessing a particular system.

Tasseo

Tasseo is the current NOC monitor / high level overview of the system. On one screen you can see a reasonable view of everything, which is extremely convenient.