We first blogged about Graphiti, our open source alternate front end to Graphite back in December of 2011, and have been happily using it in production at Paperless Post for about 6 months now. We get a lot of great feedback about Graphiti, mostly positive, but one question comes up a lot about a design choice we made when writing the library: “Why the dependency on Amazon S3?”

At first we was a bit confused that people didn’t understand what was so cool about it, until we realized that we never took the time to explain it. In this quick post, we’ll reveal three awesome ways we use Graphiti that wouldn’t be possible without Amazon’s Simple Storage Service (S3) or an equivalent place to serve and store image snapshots.

Several Points in Time

It becomes clear very quickly when you start using Graphite to record data that although it’s straightforward to get information relative to NOW, point in time graphs are harder to come by. Instead of building an extra Javascript or Ruby layer that could interpolate this information for us, we decided to take a somewhat low-tech approach, and added this route to the Sinatra application that powers Graphiti’s backend:

1
2
3
4

post '/snapshot' do
  filename = Graph.snapshot(params[:uuid])
  json :filename => filename
end

The implementation of Graph.snapshot shows what’s going on here: we’re grabbing the Graphite URL, saving the body of the response (the image) as a file, and then uploading it to S3. This allows us to provide each graph with the following dropdown menu:

which lets you recall, view, and share “frozen” point in time graphs that represent what you’d expect them to represent. This way whenever a graph is interesting for some reason, and we want to share it, it is also automatically persisted: a record of it is saved in Redis, and the image is uploaded to an external service. We have a historical record, as above, of interesting graphs. It’s been extremely helpful to see what the differences are between a graph saved a few weeks or months ago and one saved today.

Another obvious benefit to graph snapshotting became clear as soon as we implemented them: we’d be able to view them in places where we didn’t have access to the VPN that our Graphiti instance sits behind. This gave us the opportunity to implement one of our favorite Graphiti related features - daily emailed graph summaries.

Email Reports

Every morning any Paperless Post employee who is subscribed to the Graphiti email list we’ve set up gets an inbox full of graphy goodness:

Each of these emails represents a Graphiti Dashboard, and contains snapshots from that morning of each of the graphs in the dashboard:

This has been very valuable for us across a wide swath of our company. People following product numbers check out product-related graphs, Operations people follow systems-related graphs, and developers who have created dashboards to track data for features can follow graphs they’ve implemented. We all have a common discusion point for the large variety of metrics we collect, and email inbox delivery is appropriately more passive than actively having to visit the site to see the graphs. It’s become a part of many of our morning rituals to check in on our graphs and bring up any issues that might pop up. It greatly aids in correlating seemingly disparate events, and gives us more information, which is always what we’re striving for.

Sensible Chat Logs

The final point here is sort of a humorous one. Immediately after getting data flowing into Graphite, of course we wanted to share our graphs in Campfire. The first time we ever went back into our history to look at a graph we had pasted, we noticed that the graph was strange. It was now relative to the time we were checking Campfire logs, not relative to the time it was first created. So now instead of pasting Graphite links, we post snapshot links:

and when we review Campfire history, the graph history is preserved. This saves us a lot of confusion and keeps Campfire’s narrative history straight.

…but we’re flexible

There is an open issue on Github to remove the S3 dependency from Graphiti. The reason we’ve been hesitant to do this is simply that while having S3 credentials in a config file is a requirement to run the application, they don’t need to be “real.” You can insert some dummy credentials, and then just not use the snapshotting features as outlined above. Our hope is that we’ve been able to explain a little bit better why the decision to include a connection to this service was made, and that you’ll think they’re cool enough to try out. Regardless, we hope you enjoy using Graphiti, and if anyone wants to finish up the patch open on Github to provide alternate storage backends for the images, we’re flexible.