A bit about metrics gathering and Sensu

Sensu has a well documented method for running checks that push metrics to Graphite.

In some cases I opted to do things a bit differently.

One of my checks hits a message queue and gets me the number of items in the queue and the age of the oldest one.

I want to both alert on that (if something has been in the queue for more than X minutes) and graph it (how did we do this week on processing everything in under Y minutes), without making the same query 2 times.

In those cases my checks bypass the Sensu graphite handler and just send the metrics directly to Graphite.

I like that the system is flexible enough that this was easy to do.