Lets be clear that as a team we have many theme songs. They range from MOP to Boyz || Men to Culture club to …. I know of few that have struck such a cord (especially with the ops team) as Da Dip.

I gave a talk a couple of weeks ago where I tried to explain the methodology of Paperless Post’s culture of shipping. It is sung to the tune of Da Dip: when I ship you ship we ship. You can check out the slides and video below:

Automate or Die - Aaron Quint from devopsdays on Vimeo.

Also be sure to check out Bethany’s Talk about Sensu

I wanted to explain a little further about our process and why I think it’s been successful. You don’t have to look far to see or hear people talk about continuous delivery and deployment these days. While I fully believe in the tenants and real benefits of all of these practices and systems, I also feel like everyone is missing the point a bit. When I’ve talked to people at conferences, meetups, etc, a lot (definitely the majority in my very small and unscientific test group) are excited about the ideas and potential results of implementing any kind of automated deployment system but are frankly lost as to where to begin. Worse, the effort that has to go into building or implementing such a huge system seems insurmountable. I want to say that this shouldn’t be the case and I think we’ve proved that at Paperless Post.

Iterative Devops

The first assumption that is incorrect about these kind of projects is that it has to be all or nothing. We, and a lot of other teams, when working on our product are most comfortable and successful when taking an iterative approach to shipping features. We split big projects into small deployable steps, look for the 80% solution, and ship one thing at a time. Why then, on the Ops side are we obsessed with silver bullets? Why are we doing large rewrites and replacements of existing infrastructure? For our case we’ve been able to take the same iterative approach with our developer infrastructure. What I was trying to show in the presentation above was that we didn’t get to where we are today overnight. Rather, it’s taken years, and most importantly, each step was worthwhile and an improvement. We’ve been benefiting since the beginning with even the most simple layer of automation that we added years ago. Don’t wait for the ultimate solution.

Pick the right problems

Of course we all want the ‘push a button’ ideal of an infrastructure. One click deploys, scaling, beer delivery. However, as DevOps Engineers and Leaders that might not be what is actually needed now. While the end result of all that kind of automation can not be argued with, that doesn’t mean it’s the most important problem for your team to solve. At PP, we started with the whining.

It turned out that our developer setup process and our git workflow were actually the most difficult and slow parts of our daily process, and hence what was taking the majority of our dev and ops time. So instead of doing what we wanted first (one click deploys), we focused on the pain points one at a time and made them better. It’s important to remember what I see a the real goal of DevOps - to support the developers and the product they build. In the talk, I presented a ‘DevOps Oath’:

I, [YOU], Do solemnly swear to help developers solve their problems, quickly and without [too many] yak-shaves and make their lives easier, so they can ship code, so that the company can improve, so that we all can be successful. I will do what it takes to avoid repeated tasks, help unblock the blocked, and generally make things work better.

Every team is different

A big question I get whenever I talk about our devtools or other internal tools is ‘is this open source?’ To me that brings up a fundamental point of how I hope people can think about deploying automation into their workflow - every team is different. Even though a lot of people see this as a given, they have no problem grabbing and forcing tools and workflows from other companies and team into their own. This isn’t to say that OSS isn’t valuable and doesn’t have a place in Ops. I really believe, however, that in order to really have a process and tools that work, you need to start with how your team works organically, and then gently apply the tools to that process. Teams are living and breathing organisms, and applying solutions that are too abstract or worse too specific (but in the wrong direction) can be disastrous.

Keep moving

Our process and tools continue to evolve as we run into new blockers or new problems. We’re proud of how far we come, but we know that this is a moving target. Our key to success in this realm has been to just acknowledge that fact, and work with it.