Introducing Shipit
3 minute read
After a year of internal use, we’re excited to open-source our deployment tool, Shipit.
With dozens of teams pushing code multiple times a day to a variety of different targets, fast and easy deploys are key to developer productivity (and happiness) at Shopify. Along with key improvements to our infrastructure, Shipit plays a central role in making this happen.
Motivation
Coordinating frequent deployments in a large development team poses a number of challenges. You need to ensure that no one else is currently deploying, that the revision you are about to deploy has been successfully tested on CI, and even that there is no ongoing maintenance operation. With smaller teams, it’s easy to give developers deployment access via something like Capistrano, but giving production credentials to many people can become a problem. Additionally, keeping an audit trail of when specific commits were deployed is indispensable when debugging production issues.
The first version of Shipit was built in early 2013 using an open source project by Rackspace called Dreadnot. It allowed our developers to deploy their code without needing to request credentials, understand the production hardware, or involve the Operations team. While this worked well initially, we hit some limitations. Configuring new projects was difficult, which meant few projects were using it. We also hit performance limitations.
During Hack Days, we undertook a complete rewrite to address these problems. Here are some of the improvements we've made:
Synchronization
To ensure code isn’t deployed during incidents like system maintenance, there needs to be safety mechanisms in place. Shipit allows developers to set a lock to prevent other developers from deploying when it’s unsafe to do so.
Easy setup
To make configuration easier, we adopted a model similar to Travis CI and introduced the notion of a shipit.yml file. This allows deployment recipes to be maintained within each project and kept under version control. Here's the one used to ship Shipit:
We don't need to specify a deployment command - Shipit will infer the necessary steps to deploy to Pypi, Rubygems, and anything using Capistrano.
Audit trail
Shipit keeps logs and metadata of all the deploys and rollbacks performed.
Better performance
We were able to make gains in performance and simplicity by adopting GitHub as the source of truth. Instead of constantly updating a local copy of the repository and polling for CI status, Shipit relies on GitHub's push and status webhook events. This has the added benefit of playing well with the plethora of third-party services that use the Statuses API (including our Docker image builder).
Deployment
Before deployments, Shipit allows you to display key metrics, or add a checklist that should be followed. We’ve also made it possible to write visualizations shown during deployments, that make it easy to monitor progress and abort the deployment if needs be.
Shipit and You
We've put together a detailed README to get you started. Please report any issues you might run into, and feel free to submit bug fixes and improvements via pull request.
At Shopify, Shipit deploys over 200 projects (including itself) at the press of a button - something we do several hundred times daily. It handles a broad set of general purpose tasks ranging from updating DNS configurations to publishing new versions of Python eggs, and deploys our applications to Heroku, EC2 and our datacenters. Over time, we've learned from it and improved it, and we’re excited to share it with the community.