Our engineering blog is home to our stories sharing technical knowledge and lessons learned. But that's only part of the story: we hire passionate people who love what they do and are invested in mastering their craft. Today we launch "The Side Hustle," an occasional series highlighting some side projects from our devs while off the Shopify clock.
When Gabriel O'Flaherty-Chan noticed quadcopter controllers on mobile mostly translated analog controls to digital, he took it upon himself to find a better design.
For under $50, you can get ahold of a loud little flying piece of plastic from Amazon, and they’re a lot of fun. Some of them even come with cameras and Wi-Fi for control via a mobile app.
Unfortunately, these apps are pretty low quality — they’re unreliable and frustrating to use, and look out of place in 2017. The more I used these apps, the more frustrated I got, so I started thinking about ways I could provide a better solution, and two months later I emerged with two things:
1. An iOS app for flying quadcopters called SCARAB, and
2. An open-source project for building RC apps called QuadKit
Shopify production engineering will be at SREcon Americas 2017, held on March 13 and 14. We'll be sharing our best practices and the lessons we’ve learned along the way.
Tracking Service Infrastructure at Scale - John Arthorne
With the Shopify SRE team focused mainly on the most critical systems, we faced a looming crisis with hundreds of other important applications suffering from lack of clear ownership, inconsistent infrastructure, and poor automation. Here are some of the tools Shopify built for scaling out strong SRE infrastructure and practices across a large fleet of applications.
Monday, March 13, 2017 at 2:50 pm to 3:40 pm, Track 2
Lightning Talks (times to be confirmed)
Data Center Automation at Shopify - David Radcliffe
The flexibility and speed offered by cloud computing solutions have raised the bar for bare metal deployments. Automation is essential to speedy and reliable provisioning and capacity management. We’ll share some of the tools we’ve utilized to automate our data center and empower our developers to move quickly and keep up with the times.
Monday, March 13, 2017 at 2:50 pm to 3:40 pm, Track 3
Four-Minute Deploys: No Engineers Necessary - Lei Lopez
Previously, our devs had to ping an SRE to deploy. This process was prone to error and wasteful. Today, a chatbot automatically deploys Shopify over 40 times a day in four minutes, without losing any requests. The journey wasn’t easy, but undeniably worth it. I’ll share how we made this possible by leveraging Nginx, Docker, and our open-source deployment tool Shipit.
Tuesday, March 14, 2017 at 3:50 pm to 4:45 pm, Track 2
How Three Changes Led to Big Increases in On-call Health - Dale Neufeld
Burnout, unfortunately, is commonplace in operations, and its negative effects are well-documented. However, it doesn’t have to be inevitable. Recently, we realized that action had to be taken to establish a better on-call experience, including moving to a production engineering model. I’ll share specific actions that not only helped to keep our team healthy but also grew people’s expertise.
Tuesday, March 14, 2017 at 3:50 pm to 4:45 pm, Track 2
You can also catch us at our booth (#3, in the Grand Foyer), where we'll be hosting office hours, chatting about topics related to our talks.
Monday, March 13:
9:55 am to 12:50 pm - Handling Massive Flashes of High-Write Traffic
12:50 pm to 3:45 pm - Road to an SRE Model
3:45 pm to 6:30 pm - Tools for Tracking Service Infrastructure at Scale
Tuesday, March 14:
9:55 am to 12:50 pm - Auto-deploying Anywhere and At Any Time
12:50 pm to 3:45 pm - Road to an SRE Model
3:45 pm to 7:00 pm - Automating Data Center Deployments
Bug bounties have become commonplace as companies realize the advantages to distributing the hunt for flaws and vulnerabilities among talented people around the world. We're no different, launching a security response program in 2012 before evolving it into a bug bounty with HackerOne in 2015. Since then, we've seen meaningful results including nearly 400 fixes from 250 researchers, to the tune of bounties totalling over half a million dollars.
Security is vital for us. With the number of shops and volume of info on our platform, it's about maintaining trust with our merchants. Entrepreneurs are running their businesses and they don't want to worry about security, so anything we can do to protect them is how we measure our success. As Tobi recently mentioned on Hacker News, “We host the livelihoods of hundreds of thousands of other businesses. If we are down or compromised all of them can't make money.” So, we have to ensure any issue gets addressed.
In the first post of this series, I outlined Shopify’s history with flash sales, our move to Nginx and Lua to help manage traffic, and the initial attempt we made to throttle traffic that didn’t account sufficiently for customer experience. We had underestimated the impact of not giving preference to customers who’d entered the queue at the beginning of the sale, and now we needed to find another way to protect the platform without ruining the customer experience.
This Sunday, over 100 million viewers will watch the Super Bowl. Whether they’re catching the match-up between the Falcons and the Patriots, or there for the commercials between the action, that’s a lot of eyeballs—and that’s only counting America. But all that attention doesn’t just stay on the screen, it gets directed to the web, and if you’re not prepared curious visitors could be rewarded with a sad error page.
The Super Bowl makes us misty-eyed because our first big flash sale happened in 2007, after the Colts beat the Bears. Fans rushed online for T-shirts celebrating the win, giving us a taste of what can happen when a flood of people convene on one site in a very short duration of time. Since then, we’ve been continually levelling up our ability to handle flash sales, and our merchants have put us to the test: on any given day, they’ll hurl Super Bowl-sized traffic, often without notice.
My name is Emil Stolarsky and I work on the Performance and Capacity Planning team at Shopify. This series (with part one today, and part two next week) shares the problems we faced due to overwhelming traffic from flash sales and the thrifty (and nifty!) solution we created that allowed merchants to continue running sales without requiring a major overhaul of our platform.
While not every company faces flash sales, many need to handle high-traffic events that can overload their system, and we hope this post provides inspiration for solutions that can be implemented with a small team and some elbow grease.
Hi, I'm Graeme Johnson, and I work on Shopify's Developer Acceleration team. Our mission is to provide tools that let developers ship fast and safely. Recently we began shipping Shopify automatically as developers hit the merge button in GitHub. This removes the final manual step in our deploy pipeline, which now looks like this:
Merge → Build container → Run CI → Hit deploy button → Ship to production
We have invested a lot of engineering effort to make this pipeline fast enough to run end-to-end in about 15 minutes—still too slow for our taste—and robust enough to allow cancellation at any stage in the process. Automating the actual deploy trigger was the next logical step.
Hey everyone! I’m Daniel and I lead our VR efforts at Shopify.
When I talk to people about VR and commerce, the first idea that usually pops into their heads is about all the possibilities of walking around a virtual shopping mall. While that could be an enjoyable experience for some, I find it’s a very limiting view of how virtual reality can actually improve retail.
If VR gave you the superpowers to do anything, create anything, and go anywhere you want, would you really want to go shopping in a regular mall?
More than a virtual mall
It’s easy to take a new medium and try to shoehorn in what already exists and is familiar. What’s hard is figuring out what content makes the medium truly shine and worthwhile to use. VR offers an amazing storytelling platform for brands. For the first time, brands can put people in the stories that their products tell.
If you’re selling scuba gear, why not show what it’d look like underwater with jellyfish passing by? Or a tent on a windy, chilly cliff, reflecting the light of a scrappy fire? It sure would beat being in a fluorescent-lit camping store. In VR, you could explore inside a tent before you buy it, or change the environment around you at a press of a button.
Today we're excited to share our involvement with the AMP Project.
Life happens on mobile. (In fact, there are over seven billion small screens now!) We're not only comfortable with shopping online, but increasingly we're buying things using our mobile devices. Delays can mean the difference between a sale or no sale, so it's important to make things run as quickly as possible.
AMP, or Accelerated Mobile Pages, is an open source, Google-led initiative aimed at improving the mobile web experience and solving the issue of slow loading content. (You can learn more about the tech here.) Starting today, Google is pointing to AMP’d content beyond their top stories carousel to include general web search results.
The traditional model of running large-scale computer systems divides work into Development and Operations as distinct and separate teams. This split works reasonably well for computer systems that are changed or updated very rarely, and organizations sometimes require this if they’re deploying and operating software built by a different company or organization. However, this rigid divide fails for large-scale web applications that are undergoing frequent or even continuous change. DevOps is the term for a movement that’s gathered steam in the past decade to bring together these disciplines.
Until about a year ago, Shopify followed the traditional model and felt the pain of having ownership separated across teams. Developers were responsible for deploying changes, while three separate teams owned scaling, monitoring, and maintaining the runtime infrastructure respectively. Having many distinct teams with sometimes divergent goals trying to run the same production system created short-term chaos and made it hard to align on long-term goals.
We thought carefully about how to solve this problem in the right way. Running a large-scale web platform requires very deep operational skills in key areas such as networking, data storage, server management, scaling infrastructure, and transaction processing, so Shopify still required people dedicated to expertise in these areas. On the other hand, the company was building out products and features at blistering speed, so we couldn't accept any kind of organizational or technical barriers that would slow the rate of innovation.
- Tags: production engineering
Last updated: September 9, 2016
At Shopify, our mission is to make commerce better for everyone. When we say better, we’re talking about caring deeply about making quality products. To us, a quality web product means a few things: certainly beautiful design, engaging copy, and a fantastic user experience, but just as important are inclusivity and the principles of universal design.
“Everyone” is a pretty big group. It includes our merchants, their customers, our developer partners, our employees, and the greater tech community at large, where we love to lead by example. “Everyone” also includes:
- 9.1% of adults with vision trouble, and 8% of men and 0.5% of women who are colorblind
- 16.8% of adults with hearing trouble
- 15.1% of adults with physical functioning difficulty
- 4.4% of adults with cognitive disabilities [PDF]
We take our mission to heart, so it’s important that Shopify products are useable and useful to all our users. This is something we’ve been thinking about and working on for a few years, but it’s an ongoing, difficult challenge. Luckily, we love tackling challenging problems and we’re constantly chipping away at this one. We’ve learned a lot from the community and think it’s important to contribute back, so — in celebration of Global Accessibility Awareness Day — we’re thrilled to announce a series of posts on accessibility.