Star Wars: The Last Jedi

WARNING: This post is 100% spoiler, all the time. Don't read any further if that is a problem for you!

There is a lot of fan controversy around The Last Jedi. Some love it, some hate it. This isn't an effort to convince anyone of anything, but I wanted to write down my thoughts on it so that I could re-share this post instead of answering the question every time I get it :)

First, my foundational assumptions:

1. The new trilogy is intended to be a reboot of Star Wars for a new generation. Disney was smart enough not to just straight up re-cast Han, Luke and Leia, but they knew the old films don't play as well to younger audiences as they would like. 
2. The Hero Journey is a cyclical thing... there are always new heroes. To paraphrase another franchise: All of this has happened before and will happen again.
3. The Expanded Universe / Legends era stuff was never intended to be "real". I LOVED the Thrawn Series. It gave me more Star Wars when there was none. But I always knew it could be nuked at any time, so I didn't get really attached.

One of the complaints about The Force Awakens was that it was just a rehashing of A New Hope. As I said above, I think that was intentional. Given that, I think The Last Jedi was both ESB and RotJ rolled up into one. My take is that these two movies were setup: Episode IX will be the first one where the new characters are on their own. 

This movie is about saying goodbye to the past, and stepping out from the shadows of those Legends that loom over you.  

This is true both for the characters in the movie and the fans: If you are going to continue to enjoy new Star Wars movies, you have to be prepared to let go of the EU and the characters you grew up with. 

Luke Skywalker

I'm going to start with Luke, because he was the part I was most interested in.  I think this is largely his story, and its a very complex one that truly did not go how we expected it to. 

In this story, Luke plays the parts of his father and his mentors: He is Anakin, Yoda and Obi-wan all in one.

...as Anakin

Like his father, he failed miserably at what he set out to do. Anakin was supposed to be the hero that would save the Jedi order and bring balance to the force. Instead he slaughtered a temple full of children, tried to kill his best friend, and eventually turned into the most feared monster in the galaxy. 

Luke didn't slaughter a bunch of padawans, but he thought about it. He got as far as igniting the saber while looking at his sleeping nephew. "Killing him is the right thing to do", he thought. "I need to kill my sleeping nephew."

Where Anakin swung the blade, Luke snapped out of it at the last moment. Realizing what he nearly did destroyed him. 

At that point he didn't need the roof to come down on him: Luke Skywalker, Jedi Master was already dead. The roof was really just the punctuation at the end of the sentence. 

But what would have driven Luke to that place? I have a theory... 

We saw in the movie that Snoke was able to introduce ideas and connections into the heads of other force sensitive people. He established the connection between Kylo and Rey. He read Kylo's thoughts. What if he was not only influencing Ben, but also Luke? What if he was playing to Luke's anxiety? Perhaps in a very Palpatine like way, Snoke was manipulating things all along? 

As Luke says himself: At the height of the order, the Jedi were collectively manipulated by a single Sith lord. Why would it be hard to believe that the same could happen to Luke? Anakin was twisted by his fear of losing Padme. Luke was twisted by his fear of failing Yoda, Ben and the Jedi.

Snoke feeding the darkness in Ben and the fear in Luke. A perfect Sith Strategy right there! 

...as Yoda

But how could Luke just turn his back on everything? He's a great Jedi master, he can't just run away and hide! He needs to fight the Empire! He needs to train new Jedi! A true Jedi Master could never turn his back on the galaxy! What would Master Yoda have done... in the... oh... uh...

When Luke finds Yoda, he's alone on a distant, deserted planet. The planet itself exudes force energy, masking his presence. Yoda refuses to train Luke. He's gone a bit mad from the isolation. He screws with Luke just for the sake of screwing with him. 

Eventually, grudgingly, he does train Luke. He doesn't complete the training because Luke needs to rush off to save his friends. 

Yoda never leaves Dagobah to join the fight, but he does what he can in a short time to set Luke on his own path. 

Familiar?

...as Obi-Wan

After talking with Yoda again, Luke finds some sort of peace. Where he was closed off from the force before, he is now connected to the world again. But he's still an old hermit in the middle of nowhere.

Then he hears Leia's cry for help. It is her most desperate hour... 

Unlike Ben Kenobi, Luke has no way to leave Ach-To. His X-wing has been in the ocean for so many years now that it couldn't possibly fly again, and he'd never reach her in time even if it could. 

Summoning all of the power within him, he projects himself on Crait. He presents himself not as the broken old man he is, but as the person he was when last they saw him. His final stand as Luke Skywalker: Jedi Master! His presence inspires a feeling of hope in the Resistance. The spark re-ignites! 

His confrontation with Kylo was much like when Obi-Wan fought Vader: He can't win and he knows it. Just like Obi-wan, he knows that he doesn't need to win, he just needs to stall them for long enough that the last hope of the Jedi can get her friends to the Falcon and escape. 

At the end of the fight, just like Obi-Wan, Luke's body disappears; one with the force. And just like Obi-wan he is now more powerful than Kylo can possibly imagine. He can continue to train Rey. He can find other force-powerful people and steer them in the right direction. He's no longer confined in the body of a weak, tired old man. 

Rey

Rey is the desert dwelling orphan who finds her future as the Last Hope of the Jedi Order. She is sent on a seemingly impossible mission: Find the last surviving Jedi Master and learn the ways of the force. 

Like Luke before her, she doesn't find what she expects. Instead of a great warrior she finds a crazy old hermit. At least Luke just hits her with a stick rather than making her carry him around on her back! 

While the experience isn't perfect, she does learn how to connect to the force. Where Luke left Dagobah with more thorough training, Rey leaves Ach-To with the full history and texts of the  Jedi order to interpret as she sees fit. 

Everyone expected a big "no, I am your father" reveal for Rey, but instead we were told that she was a nobody from the desert... A child of drunks that sold her off. 

Of course we believe this whole heartedly because Kylo Ren is a trustworthy source of information!

I think there's a lot more to that story still. 

Snoke

In TFA we see a projected image of the mysterious Supreme Leader Snoke. We know nothing about him other than that everyone is pretty scared of him. He must be something pretty important! This echoes how we meet the Emperor in ESB. He's just a big blue head, but if Vader kneels before him he must be serious business! 

In the Last Jedi we expected to learn who he was, how he came to power, etc... 

In Return of the Jedi, Palpatine did little more than sit in a chair. Eventually he shot lightning out of his hands before being unceremoniously killed by his own apprentice. You would think a Sith Lord as powerful as Palpatine would have seen that coming, right???? 

Decades passed before we learned who Palpatine was or how he came to power. Hopefully we don't wait as long for Snoke, but I'm sure we'll learn. 

Kylo Ren

Kylo is the Vader of the story. He starts off as a Jedi apprentice, becomes corrupted by the dark side, and betrays the ones he used to love. 

In the big Throne Room Battle he has his moment where he turns from the Dark, saving Rey and killing Snoke. But when he's free from the shadow of the Supreme Leader he doesn't renounce his past deeds and join the resistance. He takes over, becoming Supreme Leader himself. 

We now see what Vader could have been: Fully committed to the Dark, running the show himself. 

We also see Kylo getting rid of the mask and being himself. He is no longer pretending to be his grandfather. 

Yoda

Possibly my favorite part of the movie: We see the Yoda we originally knew for the first time since ROTJ. 

He comes to Luke just a Luke is about to burn the last remnant of the Jedi order. He knows what it's like... he's been there himself! 

Sitting in a hut on Dagobah, thinking about how many Jedi died while he was supposed to be the Great Master. 

Wondering how such a powerful Jedi could have failed so completely. 

But after nearly 1000 years, Yoda gets it: A powerful connection to the force doesn't make you perfect. Humans (and whatever Yoda is!) have flaws. They do great things. They fail. They keep trying. 

Yoda agrees with Luke: The Jedi Order is done. He calls down lightning and burns the tree himself. 

Yoda tells Luke that there was nothing in the tree that Rey didn't already have. Now that the history of the Jedi is gone, Luke is all that remains. He feels free now, no longer being the caretaker of this legacy.

I think this is the greatest bit in the movie: Yoda knew that she took the books, and didn't want Luke to know! He set fire to the tree to keep Rey's secret / keep Luke out. 

Poe

I think of Poe as part Han and part Leia in this new series. 

He's the handsome, cocky pilot that does what he wants, when he wants, and how he wants to do it. He's no military leader, he's just a flyboy that likes action. 

But the Rebellion... er... Resistance... needs a leader. Someone needs to inspire and protect people. 

Like Leia's rise from Princess and Senator to General, Poe has to rise up. 

After witnessing the impact of his decisions, he finally understands what it means to be a leader. Rather than rushing out to help Luke he sees the fight for what it is: A distraction to give them a way out. 

In the end we find him positioned to lead the Resistance effectively. 

Finn, Rose, Phasma

Unfortunately I see all of these characters as disposable. I don't see anything in this movie at all that justifies their existence. You could cut their scenes entirely and not change the overall story. 

Leia

I did not like the space scene. I thought the execution was awkward, and I think it gave false hope for the character we've loved for decades. Leia lives! Except we all know that she doesn't... I sort of wish there was closure for the character in the movie. 

That scene should have played a pivotal role in Kylo's development: He had the shot! He could have completed his effort to kill his past by killing his mother. He hesitated, and his wingman took the shot instead. 

Now the choice is gone. His mother is dead and there is nothing of his past left. No longer is there any reason to even consider being Ben Solo again. 

Bringing her back at that point just to drag her along for the ride felt unnecessary to me, knowing that she can't be in Episode IX. 

Star Wars: Fury Road

I thought the slow space chase was dumb. That's really all there is to say there. 

Episode IX

This is where I am most excited. We are now in a place where Episode IX doesn't need to be in the shadow of the original trilogy. For the first time in the third part of this great saga, we can see what the new kids do on their own. 

My thoughts:

• Rey will become the Great Jedi that we wanted to see in Luke.
• Rey will find that the concepts of light and dark side are flawed.
• Poe will rebuild the resistance and lead the defeat of the First Order
• Kylo will eventually fall. I don't think he will be redeemed. 
• We will learn that Rey is something more than the daughter of a space drunk. 
• Finn will continue to just be Finn. 
• Snoke's story will come in book form. I'm leaning toward Palpatine Clone of some sort. 

I've taken up the 3D printing hobby of late

I've got three:

• Formlabs form2. This is the serious business printer, for ultra high quality resin masters to mold from 
• Velleman k8200 aka 3drag. This is the workhorse 
• Folgertech i3 2020. Work in progress. 

The two FDM printers are both kits. I'm going to be comparing the builds and results over the next free days. 

First up some thoughts on the price of the kits. 

The velleman was sort of an impulse buy. The kit was 399 with 2 rolls of filament. The form2 is very very expensive to run, and I figured this would be a good complement. Plus, I like building stuff. 

Well... 399 wasn't really true. 

Early on I found lots of comments about how the extruder wasn't awesome. Velleman had an upgrade ($129) that was much much better. I went from a 3mm to a 1.75mm and thus the free filament wasn't useful. I also needed a z axis upgrade ($69). 

The k8200 needs a computer to drive it, unless you get the standalone controller ($29). 

The heated bed is sloooooooooooow. 20 - 30 minutes to get to 55C, and no way will it get to 100c. $40 shipped for a reprap power expander, $30 for a bigger power supply, and the bed was hot. 

The beds on these are notoriously not level. $25 for a piece of glass for it. 

Finally it was all working well. I've been quite happy with it, and have had it running nearly around the clock for a few weeks. Total cost around $700 and a lot of time invested. I had to solder ever cable, and needed to print a bunch of parts to finish it. Total build around 2 weeks of nights. 

In the end not a bad value. A comparable machine (8x8x8 build area) pre built is way more. 

Next is the i3 from Folgertech. The kit is much cheaper. $280 shipped, including all of the same parts other than the glass. The wires are all pre soldered, chips are on, etc... I just started the build so more on this one later. 

I am the law

After lots of tech talk, it's time to switch gears and get into costuming a bit again.

I've been working on one big project since January 2013: Judge Dredd, from the 2012 movie. Here is the work in progress:

As I close in on completion, I wanted to record all of the parts of the suit and their origins. This is both so that I don't forget, and so that when people ask I can point them to this page :)

Starting from the top:

• Helmet: My own sculpt. I started with a freely available Pepakura template, molded that, did a casting, cleaned it up / refined it, made another mold and another casting. See Pepakura helmet build and dredd helmet casting threads on the RPF. 
• Collar / Neck roll: My own creation. See RI Mod thread on the RPF
• Jacket and vest: Reel Images Leatherwear 
• Armor: Studio Creations
• Badge: Cold cast brass with swappable name plates (SC)
• Elbow pads: Cast rubber, my own sculpt.
• Gloves: Leathernext Version 4.
• Belt: Buckle is cast from screen used, leather provided with it. (JB)
• Belt pouches: Leather, made by me.
• Belt accessories:
• Knife: Boker Armed Forces Tanto II as used in the movie
• Flash bangs: 1.75" wooden balls painted green
• Smoke grenate: My own sculpt.
• Mag holders: Styrene from my own molds
• Magazines: rubber castings from Glock mags (dsheerin)
• Zip cuffs: rubber, my own sculpts
• Holster: Styrene base (my sculpt) wrapped in leather, with accurate AustriAlpin Cobra buckles
• Lawgiver: Fan made replica (joatrash/morganthirteen
• Pants: Reel Images Leatherwear
• Knee pads: Cast rubber, my own sculpt
• Boots: TBD. Replicas have been ordered but are 6+ months overdue.
• Boot Armor: TBD. They have to fit the boots, so I can't do much. I'll be sculpting/molding/casting them myself. 

One of the first things everyone always asks is cost. I haven't kept careful tabs, but it's in the neighborhood of $1500 - $2000.

The hopeful debut of for our sector house is Free Comic Book Day. Want to see us in person? Visit Double Midnight Comics in Manchester, NH on May 3rd 2014!

AWS Autoscaling based on SWF Queue Depth

Amazon has great tools for autoscaling. Combined with our recently completed AMI Bakery this should be awesome right?

• Generate an AMI
• Create an auto scaling group that uses that AMI
• Profit!

But... and there's always a but... 

Amazon's autoscaling can't scale based on the metric that we care about: Simple Workflow queue depth. 

I've got a great idea! It's round, and you can put it under heavy things to help them move!

Start from the basics:

• We have a create_instance script that lets you create an instance/ami of a given type. 
• We have a scale script that compares what you asked for to what is running and adds/removes resources as needed. 

What we lack is the ability to trigger that scaling based on something we care about. 

Enter scalecontroller.py. 

This script takes a bunch of command line arguments to let you tune it to the queue you want to watch, number of jobs that trigger increase/decrease, and how big/small the array should get. 

When a scaling event is initiated, it runs scale.py, which then handles the logic of adding/removing. 

How does scale know which AMI to use?

Chef Data Bags!

Each application has 2 data bag items associated with it: artifact and launch-artifact. Each item contains the branch/build numbers for that product.

To use our "portal" example, you would have something like :

knife data bag show portal

id: portal

branch: master

build: 13

knife data bag show launch-portal

id: portal

branch: master

build: 12

The portal data bag show the most recently deployed version. This is what you would use to pre-bundle an AMI, prior to the release of the code to the world.

The launch-portal databag is the one scale looks at to figure out what AMI to load.

We have a standard naming convention for the AMIs. Using Chef attributes and the branch/build data in the data bags, scale can calculate the AMI to use, test for its existence/availability, and act accordingly. 

AWS, Chef and scaling a mixed Windows/Linux environment (Part 1)

Note: you'll notice that I rarely talk about what company exactly... it's not them, it's me. I prefer to keep that sort of thing somewhat private.

I've been at $COMPANY for a year and a half now. In that time we've gone through a surprising number of architectural changes. Iterative development indeed.

I'm really happy with where we are today, and would like to take some time to write it all down.

In the Beginning

Whenever I hear someone talk, I always like to hear about the road they took to get to this awesome place. I want to know what they tried and hated; I want to hear about all the failures. I think there's a lot to learn there. So, I'm going to start from my beginning and work my way up to what we are doing now.

When I first came here, the infrastructure was all AWS, managed via RightScale. We weren't really using RS correctly though. We weren't letting instances inherit configurations from deployments, we weren't making smart use of templates, we were achieving a LOT of our functionality as shell code within templates, rather than using their engines.

Deploying software was a nightmare.

We had 2 different core applications (there was a lot more going on than that, but we'll stick to just the main in-house developed stuff).

One was Python, the other a Grails app in the form of a war file.

Python app, running on Windows and Linux.

This was deployed from SVN. The RightScale config included the desired SVN revision. Since we weren't using deployment inheritance, you had to go to each instance and update them one at a time.

The process for deploying code differed wildly from Windows to Linux, even though they were both from the same Python repo.

On the Windows side you would RDC to an existing box, svn up to the revision, manually edit any config files, bundle an AMI, launch new instances and terminate the old ones.

On the Linux side you would update the RightScale template variables and run the deployment script to execute the changes. The scripts were basically shell scripts running a lot of "sed s/this/that/g"

Grails app

The war file was downloaded from S3. Each environment (dev1, dev2, qa, prod, etc...) had a bucket, and the deployment script downloaded the artifact from that. The war files were copied to the buckets by hand, along with the config templates which were sed processed like the python code.


Next: First pass at automation

AWS, Chef and scaling a mixed Windows/Linux environment (Part 2)

The first pass

Our first pass was to get everything moved off of RightScale and into a system we managed. We had the technical skills to do this ourselves, so no need to give RS so much money!

After a month+ of evaluation and testing, we settled on Chef 10. We found the cookbook system to fit better into our overall development workflow, and the open source community was strong. Puppet was a serious contender too, and to this day I think either would have worked as well as the other.

But, there can be only one, and the one was Chef.

Starting at the foundation

We began with a simple goal: We need to be able to launch and minimally configure a base system of either Windows or Linux using the same set of tools.

By minimally configure, I mean:

System is online and accessible from our office or over our VPN
Users have their accounts with appropriate access
Tools that are universally used on all systems are present (sysstat on Linux, Powershell on Windows, etc...)

Sounds easy, right?

Turns out there are a LOT of awesome open source tools for managing your AWS infrastructure, and not many of them support Windows AND Linux equally. Now start looking for things that understand Amazon Simple Workflow or some of the other offerings outside of EC2 and S3.  Even Chef required a different plugin and bootstrapping syntax to do Windows.

This led to creating a basic AWS management platform in-house. We used Python/Boto, and did our own tooling around security groups, s3 buckets.

At the end of this phase we had a config file for each "fleet" (our term for stack or environment). This config file contained the core information needed to manage it: AWS credentials, root SSH keys, things like that. From there you could initialize an entire fleet in a brand new AWS account. All buckets, groups, instances, roles, etc... would be generated for you. Cool!

But that just gets us instances. How did we make them do the work?

We set up a couple of initial cookbooks. Over time this has grown, but in the beginning we had company_system and company_mainproduct.

The system cookbook contained all the OS level stuff: How do I install Java? Python? What users should have access here?

The mainproduct cookbook contained everything about our app: How do I find the Grails war file? How should tomcat be configured for this app? That level of things.

Deploying code was now as simple as a knife command

knife ssh "role:portal" "sudo chef-client -o recipe[product::portal_deploy]"

Woo!

Oh... wait... Windows.... Strange hostnames (long story... has to do with the lack of unique hostnames on the windows boxes messing with Chef's node discovery)

knife winrm -m hostname -x user -P password "chef-client -o recipe[product::worker_deploy]"

Hmm... remembering that will be tough.

Add "deploy.py" to the mix, which wraps up the logic for each app into something simple and easy.

./deploy --fleet test --artifact portal --branch master --build 12

And it sorts the rest.

Cool! Thus closes our first pass at Chef and Automation.

Previous: Starting point
Next: Improvements

AWS, Chef and scaling a mixed Windows/Linux environment (Part 3)

Next phase: Improvements

After running for a few weeks, we started to really feel some pain points.

Chef roles take effect immediately on save. 

This makes phased dev/prod/qa rollouts tough. Say we have a new recipe we want to add to the "portal" runlist. It would be good to test that in dev first. 

We started putting feature flags into recipes, so that they would only run if an environment attribute was set. But really, what we wanted was versioning of roles.

A speaker at ChefConf had the answer for us: Roles cookbook. Rather than having your runlists in the role, have a Roles cookbook. Each role has a recipe that includes anything that role should run. 

That lets you pin your environments to a particular version of the Roles cookbook, effectively giving you 

Windows is slow. Always. 

Bootstrapping a new Windows instance from scratch takes about 35 minutes... so long that you really just can't adapt to dynamic processing needs.

We knew that ultimately we needed to start building fully configured AMIs to scale off of, but we didn't have the devops resources to do that. So, for now, we settled on launching a ton of instances after a release, then stopping most of them.

When we needed more power we could start the instances, which would have them online in 5 - 10 minutes. 

Each release we'd start up all the stopped instances, deploy code, then stop a bunch again. 

Previous: Improvements
Next: Bakery

AWS, Chef and scaling a mixed Windows/Linux environment (Part 4)

The AMI Bakery

Here's the part that I really wanted to write about.

We decided it was time to invest in the ability to generate fully configured AMIs for our applications.

The Roles Cookbook

Back to the roles cookbook from a prior post. Each role had a "role_launch" recipe that did all the things necessary to take a base CentOS/Windows AMI and turn it into that thing. We now added a role_ami recipe that was very similar, but with some important differences:

• nothing would start or be run during creation: db migrations, service starts, etc... would be set to run at boot time, but not now. 
• the ami_prep recipe would install a startup script that would allow the instance to register itself with the chef-server when it came online. 

AMI Prep

Windows. Linux. userdata. So many different ways to do things, and none of them quite as controlled or specific as we wanted. 

Like so many other wheels, I reinvented it. 

• Each platform has an init script. For linux it's a standard SysV init style, and for Windows it's a powershell script that runs at boot via task scheduler. 
• The script only runs on the initial boot. It looks for a file firstboot.txt in the root volume and only runs if it is present. On completion of the run the file is removed. 
• Both use an sns handler recipe to generate an alert if the run fails. This way we only hear about it when something bad happens.  
• In both cases the instances use IAM roles and AWS command line tools to fetch a default runlist from an S3 bucket and execute chef-client. 

Adding the --bake flag

Our main tool for instance creation is the appropriately named "create_instance.py" script.

Previously this script would:

• create/update security groups
• create/update IAM roles
• create instance
• bootstrap chef / run the role_launch recipe

With the new --bake flag it could now:

• create/update security groups
• create/update IAM roles
• create instance
• bootstrap chef / run the role_ami recipe
• stop the instance
• take an ami of the instance
• wait for the ami to be available
• terminate the instance
• clean up the chef node objects

Post deploy task

Immediately post-deploy, a job in Jenkins is triggered to bundle new AMIs for the current release. That way if we need to scale rapidly, we have an image. More on that when I post about scaling.

Previous: Improvements

Automated Chef Cookbook Testing. How? Why? Help! Part 1 of many!

As part of the DevOps Promised Land, we decided that it was time to move Chef out of Operations and into every developer's hands. 

When I started at my current job, we did pretty much everything manually. I spent most of my first year here building up a reasonable infrastructure management system based on Chef, and getting all 200+ hand crafted EC2 instances rebuilt in an automated fashion.

Now that we've gotten pretty happy with where we are at, it's time to put the Dev in DevOps. No more opening tickets to add cookbook attributes! Self-service equals faster response time and less work for me. Let's do this!!

I use the term "fleet" throughout this post... you can replace that with stack, environment or whatever other term you use for a collection of servers that make up an instance of your application.

Before we get into details, lets talk about the high level goals:

• All developers can modify cookbooks.
• All cookbooks must have appropriate tests built in.
• All cookbooks must have an automated "build" process that runs the tests at every commit. 
• No cookbook changes are pushed to the Chef server until they pass all of their tests.
• Cookbook changes are applied to all fleets (including prod!) as soon as they pass their tests.

Here's where we were at the starting point of this project. 

Our testing system was there, but it was manually run.

We were using the minitest-handler cookbook, and had reasonable test coverage. I won't lie to you and say that it was 100%, but it's close.

Every cookbook had a Vagrantfile, and used Berkshelf to maintain dependencies (via the vagrant-berkshelf plugin)

You made a change, and it was on you to vagrant up / make sure the tests pass. Once it's all good, you would commit the code.

Jenkins would get notified of the commit. It would check out the code, and do some REALLY basic tests:

• knife cookbook test
• foodcritic
• knife testcoverage

That last one is a crappy little knife plugin I wrote that would look at each recipe and ensure that there was an _test.rb recipe associated with it. That doesn't mean there were any useful tests, just that you had at least taken the time to create the test file. 

If everything passes it would do a knife cookbook upload. 

When I was the only developer writing code, this was a workable system.  I knew what I changed, when I changed it, and what the impact would be. I could troubleshoot problems very quickly and push fixes within minutes of bug detection.

As the team grows, this falls apart:

• The prod and qa environments are pinned to versions, but dev is running head. broken code breaks all 5 dev fleets!
• Cookbook versions aren't necessarily frozen. who knows what "1.5.4" actually means? it's totally possible for prod to be "pinned" to a cookbook rev, then for someone to change that version!
• What if i just upload directly from my workstation without running tests?

We started off by defining some things that would help us get there

• Every cookbook change bumps the version number.
• Every cookbook upload is frozen.
• Every fleet has cookbook versions pinned (except for the chef test fleet).
• Every cookbook should have a build script that encapsulates the logic for building and testing it. That way our Jenkins jobs are simply check out code / run build.sh

The knife-spork plugin by etsy is a great way to manage all of versioning parts of that list. I'll get into the details of how we use all of that later on.

Building a cookbook

Let's walk through a simple cookbook. We have one that we use internally for basic OS stuff: selinux, iptables, package management, etc...

It's a good one for the example because the tests are for the most part super easy:

package "wget" do  action :installend

Gets a test of:

it "Installs wget." do  package("wget").must_be_installedend

test-kitchen

At first we were using vagrant-berkshelf to tie things together. The 1.5.1 release of Vagrant broke the plugin, and a bit of reading made me feel that test-kitchen was a better choice anyway. 

So first up was adding test-kitchen to the cookbook (and removing Vagrant)

rm Vagrantfile 

kitchen init

And there you go! Now we need to customize the .kitchen.yml

• We use centos-6.4, so I took the ubuntu platforms entry out. 
• Add the runlist
• Attributes.... uh-oh!!!

.kitchen.yml allows you to specify a list of chef attribute to pass. But I don't want to list them in each cookbook! I have an attributes.json file on the Jenkins box that defines all of the credentials that would normally be in an encrypted data bag. Vagrant lets me simply

dna = JSON.parse(File.read("/var/chef/attributes.json"))
chef.json.merge!(dna)

I looked at the test-kitchen docs, but didn't see anything specifically about including a file. I did catch though that the .kitchen.yml file is ERB parsed before being used. Perfect!

I converted attributes.json to yml using one of many tools out there, then added

    attributes:
<%= File.read("/var/chef/attributes.yaml") %>

To my yaml file and it promptly blew up! Since the attributes file is just getting inserted exactly at that point, the indentation has to match up. Shifting the data in the yaml file over by a couple of tabs did the trick.

Kitchen test now works! It boots the VM and processes the runlist, including minitest-handler. We will eventually start using test-kitchen's built in tests but for now we're OK with what we've got.

build script

As I mentioned earlier, we want to have a standardized build script that would be used by all cookbooks. The content would change, but the content would be the same:

• Run whatever tests that cookbook gets
• If the tests pass, upload the cookbook and exit zero
• If the tests fail, exit non-zero.

Our standard set of tests includes:

• knife cookbook test (basic syntax)
• knife spork check --fail (make sure the version number has been bumped)
• kitchen test
• kitchen destroy
• foodcritic
• knife spork upload

To make sure that whoever is running this has a fair chance of success, we include a Gemfile in the cookbook with all of those tools in it. The build script starts off with a bundle install, so as dependencies change, the systems should stay up to date.

The shell script looks at the exit code of each command as it runs, and dies if any of them are non zero.

git-hooks

Every cookbook has a git-hooks directory with recommended scripts to either use or integrate into your existing tools.

The main one that all cookbooks have is a simple pre-commit hook to run knife spork check --fail. This makes sure that you don't try to upload a frozen cookbook version. 

promoting the cookbooks to the fleets

Since we aren't yet running full integration tests, we don't have a really good way to prove that the newly published cookbook plays well with its other friends in the Chef playground. 

For the moment, we are automatically promoting the fleets once a cookbook passes its tests and gets uploaded.

We have a Jenkins job that looks up the latest frozen version of each cookbook and promotes the fleets to that version. Since you don't get a frozen upload without passing your tests, that should be safe enough.

That also allows us to have in flight cookbooks on the chef server without fear -- the fleets will only ever auto promote to the latest frozen revision. 

integration testing

Great! Our cookbook passed its test and has been uploaded. All of the fleets are pinned to the older version though, so no-one is seeing this change.

Now what?

Enter the test fleet. 

We have 1 development environment that does not get anything pinned. It is used only by ops, just for testing new Chef code.

Go forth and deploy / run rubot / do whatever you need to do to feel happy on the test fleet. 

Right now we are doing this by hand, but that will be next on the automation to-do list.

Getting Chef + RVM playing nicely together

I know that all of maybe 3 people read this blog. I typically forget about it myself. But, I wanted to jot this down someplace, and here seemed as good as any.

We've recently started to move into the world of Rails. We use Chef to manage our infrastructure. There are some quirks to getting them all lined up right, and finding a straight up tutorial is surprisingly tricky. So, here goes:

• Install Chef via Omnibus as recommended by them.
• Get the RVM cookbook, maintained by fnichol.
• Pick a non-root user to run your rails app(s) as. 
• Install RVM/Ruby as that user, via the user recipe in the cookbook.

Any place in your other cookbooks that you are going to call ruby scripts (bundle install, rake db migrate, etc...):

• include rvm::default
• wrap the commands with the rvm_shell LWRP from the rvm cookbook. This ensures that the user's rvm environment is inherited properly. 

Here's an example of the recipe I'm using to install RVM:

node.normal['rvm']['user_installs'] = [
    {
        'user'  => node['fleet']['systemuser'],
        'default_ruby' => node['rvm']['default_ruby'],
        'global_gems' => [
            { 'name'    => 'bundler' },
            { 'name'    => 'rake' }
        ]
    }
] 

include_recipe "rvm::user"

I'm using normal for the attributes to ensure that the defaults in the rvm cookbook are overridden. In my case node['fleet']['systemuser'] and node['rvm']['default_ruby'] are defined elsewhere in the run list.


Then some code (bundle install)

include_recipe 'rvm::default'

rvm_shell "Running Bundle install" do
  user node['fleet']['systemuser']
  group node['fleet']['systemgroup']
  ruby_string node['rvm']['default_ruby']
  cwd node['app']['dir']['deploy']
  environment 'RAILS_ENV' => node['app']['rails_env']
  code %{bundle install}
end

 Again, some attributes are defined elsewhere, but you should be able to get the idea.

This may not be perfect, but it works for me!

Other gotchas:

If you try to install RVM system-wide, chef-client gets sad. You end up changing the environment such that the omnibus ruby is no longer what env ruby finds, so you end up needing to install chef gems into your RVM ruby.

Ugly!

Chef-client always runs as root, and your app should never run as root. So, there shouldn't really be any conflicts here.

I'm not good at Vegasing

I'm in Las Vegas for the amazon web services re:invent conference. This is the stuff of corporate dreams: a private suite in the Venetian. Late starts and early endings each day. Sponsored pub crawl featuring free drinks at all of the resort bars.

There's just one problem: I don't drink. Or gamble.

So what to do?

I've been getting up early to talk to my daughter before school. I'm going to bed around 9 and getting up at around 4.

I can only imagine how maddening this is to my friends.

Tonight is the big party hosted by Intel. I'll go for a while, but I imagine that it will largely be about the copious free booze, and that I'll get bored.

Rock of Ages is in my hotel. I've seen it in Boston and on Broadway. Maybe I'll go again ;)

Home in 2 days.

A bit about metrics gathering and Sensu

Sensu has a well documented method for running checks that push metrics to Graphite.

In some cases I opted to do things a bit differently.

One of my checks hits a message queue and gets me the number of items in the queue and the age of the oldest one.

I want to both alert on that (if something has been in the queue for more than X minutes) and graph it (how did we do this week on processing everything in under Y minutes), without making the same query 2 times.

In those cases my checks bypass the Sensu graphite handler and just send the metrics directly to Graphite.

I like that the system is flexible enough that this was easy to do.

Oh Windows...

One thing that makes me sad regularly is that we have Windows instances. Lots of them. And they are important.

Sensu does support Windows, but the documentation is sparse. You pretty much have to either use their Chef cookbook or read it to get the steps required to install the client.

Similarly, go visit the Sensu Community Plugins project and see how many of them cat /proc/somethingorother to get data. No Windows for you!

I ended up writing a couple of Ruby scripts that would get lists of processes, system load and the like on both Linux and Windows platforms. I need to clean them up a bit then submit them back to the project.

It wasn't hard to do, but it is a thing to be aware of if you go down this road.

Collectd, Graphite, Tasseo, Etsy Dashboard, Vacuumetrix

My current visualization suite is made up of 4 tools:

Graphite

The open source graphing darling serves us well. Feed it metrics, make API calls to turn them into graphs. Do one thing and do it well!

Collectd

Collectd runs on every instance, gathering system stats and reporting them to graphite. We went with Collectd because right out of the box it did about 90% of what we wanted, and the remaining 10% was easily addressable via some simple Python based plugins.

This is how disk capacity, free memory, load and other basic health checks get sent to graphite.

Vacuumetrix

Vacuumetrix pulls stats out of AWS and feeds them into graphite. Good for looking at Elastic Load Balancers, RDS instances and other things where we can't install our own tools.


Etsy Dashboard

We used the Etsy Dashboard project to make some generic "Here's how this instance looks" pages. Go to the dashboard for prod-loghost and you'll see disk I/O, free mem, load, network in/out and all the other things that people like to look at when assessing a particular system.

Tasseo

Tasseo is the current NOC monitor / high level overview of the system. On one screen you can see a reasonable view of everything, which is extremely convenient.

Enter Sensu and friends

We knew what we didn't like about Nagios, but what did we actually want?

After some discussion it came down to this:

• We want to know when a critical system/service is unavailable.
• We want graphs to easily analyze trends.
• We don't want to have to restart anything when instances are added or removed.

Sensu is a popular modern monitoring router (their term!). It doesn't try to be a Swiss Army knife... it takes input from scripts, and performs actions based on that input. 

For a long time Nagios user, Sensu is a bit disconcerting. There's no enormous "all clear" dashboard with hundreds of green dots. You know when something is wrong, but have no real way to assess that everything is right. 

We chewed on that for a while and eventually decided we were OK with it. 

One thing I really liked about Sensu was the way clients were added and removed. I won't re-write their documentation, but it comes down to this:

• When sensu-client starts, the node self-registers.
• A simple API call takes a node out of the clients list. 

Want to stop a node? Just make sure that your stop script makes the right API call. When it starts back up it will put itself back in. 

The other thing I liked is that it uses Nagios style checks (based on exit code of the plugin), so we were able to drop it in place pretty easily and just call the plugins that were already installed. 

So that gets monitoring out of the way for now... on to graphing!

Nagios + Chef = tolerable

We use Chef for system configuration management, which makes nagios sort of manageable...

Our in-house nagios cookbook has a recipe called monitor-clients. This recipe does a bunch of things:

• Deletes all of the existing config files in the /etc/nagios/dynamic directory. Otherwise how do we know when a node has gone away? It's not like Chef tells you that it used to be there but is now gone. This is the first bad part... Chef run goes badly, Nagios is hosed!
• Looks up the nodes on the chef server and generates configuration files for each of them based on role and attributes defined in each server's environment (more frequent checks for prod, no evening alerting for dev, stuff like that)
• At the end of the Chef run, the nagios server restarts, loading all of these new configs.

So why does this suck?

Any time a node is created or destroyed, this recipe has to be re-run to pick up the changes. To ensure that, we've added hooks into the tools we use to provision instances so that this is run automatically. Just hope no-one uses the AWS console to create/destroy something... 

One of the cool things with AWS is that you can stop instances and not get charged for them until you turn them back on (other than the EBS storage fees). You can save thousands a month on your AWS bill by doing this. Unfortunately that means the nodes still exist in Chef, still get detected by this recipe, and end up getting flagged as down by Nagios. 

While you are waiting for this script to run your new nodes aren't monitored and your terminated ones are trying hard to make it to critical and page you one last time before they are removed.  The bigger your environment the slower this gets.  We have anywhere from 100 to 200 nodes at any given time. On the high end of that the config generation phase of the Chef run can take 3-4 minutes. Imagine 5 times the hosts. 10 times. Somewhere this would fall over. 

I'm sorry Nagios, but it's time to say goodbye.

Often when one starts the awkward breakup proceedings, the conversation goes something like this:

It's not you, it's me. I've changed, and we're just not compatible anymore.

Truer words were never spoken.

I've been using Nagios for a long time... all the way back to when it was called Netsaint. In those days, setting up a new server was a Big Deal. You would have to do all sorts of things, not the least of which being the process of physically unboxing/setting up the machine.

In those days, Nagios worked well. You probably have a bunch of hostgroups defined and some nice templates... Create a host definition for web17c, add it to the webservers hostgroup, restart nagios and you are good to go! At athenahealth I was monitoring 200 odd boxes this way and have very little to complain about.

It's a different world today. At my current job we are all cloud based (AWS, more on that in another post!). We scale our resources up and down throughout the day to the order of +/- 50 nodes over the course of an 8 hour workday.

This puts us in a position where we need to monitor hosts we don't know about, running services we may not be aware of.

I've tried a lot of things to get this working well. My next couple of posts are going to be about the ways we tried to make it work, and what we ended up doing instead.

Here we are... Yet Another New Blog.

Like most tech-folks, I have an account on just about any social networking site out there. Twitter for quick brain dumps, Facebook to see what my friends are having for dinner. Tumblr for pictures of cats, etc...

For many many years I've used Livejournal for more substantial posts. Lately though LJ has felt like a somewhat dead community. It doesn't integrate well with other systems, and not many people I know are there anymore.

Enter blogger.

What will I be writing here? Most posts will fall into one of these buckets:

• Technology of the Linux/DevOps/Sysadmin sort.
• Sci-Fi props, costumes and collectibles.
• The 501st.
• My kid.

If any of those things appeal to you, this might not be a terrible experience. If I lost you there just move along (see, Star Wars reference!)