Interested in Mobile Tooling? Shopify's Production Engineering team is hiring and we’d love to hear from you. Please take a look at the open positions on the Engineering career page.
Mobile Release Engineering at Scale with Shipit Mobile
One of the most important phases of software development is releasing it out to the final users. At Shopify, we've invested heavily in tooling for continuous deployment for our web apps. When a developer working on a web project wants to deploy their changes to production, the process is as simple as:
Merge → Build container → Run CI → Ship to production
In contrast, uploading a new version of one of our mobile apps to Google Play or the App Store involved several manual steps and a lot of human interaction that caused various problems. We wanted to provide the same level of convenience when releasing mobile apps as we do for web apps, and also take the opportunity to define a framework for all the mobile teams to adopt. For this reason, we developed a new tool: Shipit Mobile, a platform to create, view, and manage app releases.
The Issues With Mobile Releases
Automatic deploys and continuous delivery aren’t possible in mobile for several reasons including approval wait time; coordination between developers, designers, and product managers; and because our users need to update the app. If they don’t have automatic updates enabled, finding several updates of the same app multiple times a week, or even a day, is annoying.
Moreover, a new release isn’t just deploy the latest version of the code from our repository to our infrastructure. Third-party services (the app stores) are involved, and software approval and distribution is owned by them. This means that we can’t update our apps several times a day even if we wanted to.
Uploading a new version of our mobile apps to Google Play or the App Store was fraught with problems. Releasing new apps was error prone due to the high number of manual steps involved, like selecting the commit to release from or executing the script to upload the binary to the store. Different teams had different processes to release mobile apps. The release process wasn’t transferable—knowledge couldn’t be shared within the same organization and the process was inconvenient and complex. Each team had variants of their release scripts, and those scripts were complex and untested. Release version and build numbers had to be managed manually. Finally, there was a lot of responsibility and burden on the release manager. They had to make decisions, fix bugs found along the way, communicate with stakeholders, and coordinate other side tasks.
Our Solution: Shipit Mobile
Mobile Release flow
Releasing a new mobile app requires performing multiple steps:
Pick a commit to release from
- Increment build and version numbers
- Run CI
- Manually test the app (QA)
- Iterate on testing until all the bugs and regressions are fixed
- Update screenshots and release notes
- Upload it to the store
- Wait for app submission to be approved
Our existing tools for releasing web apps weren't suitable for the mobile release process, so we decided to build something new, Shipit Mobile.
Shipit Mobile Home
Creating a New Release in Shipit Mobile
Creating a new release in Shipit Mobile
A release starts with a new branch in the repository. We follow a trunk-based development approach in which the release branch is branched off of master. We opted for trunk-based development instead of doing the release directly from the master branch because it reduces the risk of including code that doesn’t belong to the current release by mistake if both new features and bug fixes are pushed to the branch where the release is taking place. Release branches are never merged back to the master branch, and bug fixes are pushed to master and cherry-picked to the release branch.
The branching model
This branching model allows us to automatically manage the release branches and avoid merge conflicts when a release is completed.
Testing and Building the Release Candidate
Release page in Shipit Mobile
When a new release is created, a candidate is built. A candidate in Shipit Mobile corresponds with a releasable unit of work. Every new commit in the release branch creates a new candidate in Shipit Mobile. For every candidate, the build number is incremented. A new candidate triggers two continuous integration (CI) pipelines. One is a test pipeline that ensures that the app works as expected, and the other pipeline builds the app for release. We decided to decouple these two pipelines because, if there is an emergency, we want to allow developers to release the app even if the test pipeline has not finished or has failed.
Distributing the App to Different Channels
Distributing the App to Different Channels
Once the app has been built, tested, and CI is passing, it’s time to upload it to the store. In Shipit Mobile we have the concept of distribution channels. A distribution channel is a platform from which the app can be downloaded. The same release binary can be uploaded to different channels. This is useful if we want to publish the app to Google Play and send the same version of the app internally to our support team so they can quickly access the right version of the app when a support request comes in, and we will know that they have the same app our user downloaded from the store.
The two main channels we support are Google Play and the App Store. For Google Play, we have used the Google Play’s official API. For the App Store, Fastlane makes it easy to upload iOS apps to AppStore Connect. We use them alongside with Apple’s app store connect API to communicate with the App Store, upload the apps and metadata, and check if the apps have been approved and are ready to release.
Configuring Shipit Mobile
Every project that releases using Shipit Mobile needs a configuration file. This file tells Shipit Mobile some basic information about the project, such as the platform, the channels to where we want to upload the app, and optionally the Slack channels that need to be notified with the state of the release. We went with a simple configuration favouring convention over configuration. This can be seen in the way we manage the metadata (release notes, screenshots, app description…). If the configuration file tells Shipit Mobile to upload the metadata to the app store, it knows where to find it and how to upload it as long as it is located in the expected folder.
The Release Captain
A mobile release is a discrete process. Every release contains new features and bugfixes, and we need a person responsible for making decisions along the way. This person is the release captain.
Our previous release process was complex, and the role of the release captain wasn’t easy to transfer to others. Moreover, the release captain had to communicate the state of the release to all the people involved in it.
With Shipit Mobile we wanted to make the release captain role transferable. We made this a reality by building the platform to guide the release captain through the release process and notify the right people when the status of the release changes. It’s now easier for others to see who’s in charge of the release.
Useful Notifications
Providing useful notifications was a request we got from our users since we started to work on Shipit Mobile. Before Shipit Mobile, only the person in charge of the release knew its exact state and was responsible for communicating it to others. With Shipit Mobile we offload this responsibility from the release captain by sending notifications to Slack channels, so every developer in the team can know the release status, removing the burden of communicating the status of the release to stakeholders.
Shipit Mobile Notifications
Emergency Release and Rollout
At Shopify we have trust in our developers and their decisions. Although we strongly recommend a passing CI for both creating a release and uploading the app to the store, we have some mechanisms in place to bypass this restriction. A developer can decide to start a release or upload the app to the store without waiting for CI. This is useful if one of the services we rely on is misbehaving and the status of the builds aren’t being received. At the end of the day, we build tools to make our developers’ lives easier, not to get in their way.
At Shopify, we work on our infrastructure to build better products quickly. Last year, the Mobile Tooling team spent time working on a scalable CI system for Android and iOS, pioneering the use of new technologies like Anka.
Also, we worked on providing a reliable CI system resilient to test and infrastructure flakiness and built tools on top of this system to improve how our mobile developers test apps.
Defining standards through tools across mobile teams and bringing good practices and conventions makes it easier for developers to share knowledge and jump between projects. This is one goal of the Mobile Tooling team.
Shipit Mobile has been used in production for over six months now. During this time we have changed and evolved the platform to accommodate our users’ needs, add new release channels and improve the user interface. It’s shown to be a useful and stable product our developers can trust to release their apps. It’s reduced the complexity that is incurred during a mobile app release, enabling us to speed up our release cadence from three weeks to one week, and we’ve seen more people take on the role of Release Captain for the first time.