Over the summer, I’ve been traveling around a bit speaking at meetups and conferences sharing how Paperless Post thinks about metrics. Its been a fun and interesting experience and not only have I gotten some great feedback, it’s helped me formulate my thoughts to bring back to the team.

Most recently, I gave a (more technical) version of the talk at the SF Metrics Meetup which you can watch here:

Aaron Quint (@aq) talks about metrics at Paperless Post and Graphiti. from Librato on Vimeo.

and heres just the slides:

Moving the needle

At Paperless Post our business is very driven by holidays and seasons. For the past 3 years the months between Sept-Dec are our biggest traffic months by far. Last year we set some very large goals for ourselves and our team. The biggest goal, that we talked about loudly throughout the company was expressed as a big number - N Million Cards sent in December 2011. We got close, really close, but we didn’t meet the goal. Though we still had a very successful holiday season, the fact that we promised something and failed was really painful. In January, we looked back and tried to analyze what the real failure was and came to some interesting conclusions about the way we were setting goals.

The clearest failure was that everyone felt very little connection to the goal. Though everyone worked their asses off, it was really hard to see how their work was or wasn’t affecting this big number. No one felt truly empowered to move the needle. The second problem was that the big number itself was actually a conglomeration of a lot of different smaller changes in our numbers and no one change could have produced that goal. Moreover, there were two very different very separate processes at work that would drive total success. We realized if we started to separate them out, they could give so much more clarity to our process.

Quantity vs Quality

The two main categories of metrics as we saw them were separated by how and where they changed our totals. If the end goal was growth of the business in general, we realized that we needed to improve in two distinct areas.

Quality metrics measure the quality of the product. This is often what most startups focus on: Adding and refining features and supporting new and existing users. For us, over the past year, it has also been removing and simplifying. Quality metrics are measured in averages of X per Y. For example average Events created by an Account, or average Coins used per Event. Changes to our product, like simplifying the process of creating events can have a great affect on these numbers. We like to think of this as the velocity of our growth.

However, to grow exponentially (as many companies would like to do) you need to add another factor we call Quantity. Quantity is driven by the sheer volume of people going through the product. Quantity metrics are really growth metrics, they are measured in deltas % of X. For example: % of guests registering for accounts, % of visitors to the homepage creating a card. These aren’t numbers that directly effect our bottom line, instead they drive people into the Quality metrics and the multiplication of the two are what produce the big numbers that let us meet our goals.

Metrics Driven

We’ve tried to shift our process to match these ideas. We want not only the management but everyone to really think about these metrics and push them forward. After breaking down the goals in terms of these smaller consumable deltas in ‘rates’ (as opposed to big numbers), we can attach the metrics we want to change to feature specs and issues. This allows us to have clear, measurable goals attached to features.

Measuring

Once the goals are set up and the development process begins, we measure throughout the process. We have a ton of different tools for measuring and we like to think about it in two separate categories: Accounting vs Trending. The differences between these two are really stark.

Accounting are raw numbers that are exact and not-real time. They come from our production data and they are used for making decisions and measuring our progress in very specific ways. We have internal scripts to pull our data out of PostgreSQL into what we call The KPIs (Key Performance Indicators). We also use a data-warehousing tool called RJ Metrics to get cohort and other deeper analysis.

Trending is relative data, it is completely devoid of real numbers and is all about deltas and changes over time. It is real time and lets us get feedback directly after deploying new code. We have some custom built dashboards for this and then also use Graphite and Graphiti (which we have blogged about here).

The difference is we watch the trends, and we study the KPIs.

Constantly improving

For this upcoming holiday season, we are going to attempt to still drive our team to put out their truly best effort without setting everyone up to fail. We’re going to do this by using these different categories of metrics and tools to set smaller achievable goals that each team can actually change. When combined the goal is all the small changes in our growth rates will make this a very successful year.

We’ve been changing our process continuously over the past couple years and some things have worked and some haven’t. The point is that we are converging on something that truly works and its exciting. We hope to continue to share our learnings with everyone as we improve.