Shopify Heads to Dublin for SREcon Europe

Shopify Heads to Dublin for SREcon Europe

3 minute read

Production engineers from Shopify will be crossing the pond to speak at SREcon Europe from July 11 to 13, 2016 in Dublin, Ireland. From flash sale engineering to fuzz testing to multi-tenant architecture across multiple data centers, we got you covered!

From stores with ads in the Super Bowl to selling Kanye’s latest album, Shopify has built a name for itself handling some of the world’s largest flash sales. These high profile events generate write-heavy traffic that can be four times our platform’s baseline throughput and don’t lend themselves to off-the-shelf solutions. 

This talk is the story of how we engineered our platform to survive large bursts of traffic. Since it’s not financially sound for Shopify to have the required capacity always running, we built queueing and page caching layers into our Nginx load balancers with Lua. To guarantee these solutions worked, we tested them with a purpose-built load testing service.

Although flash sales are unique to commerce platforms, the lessons we learn from them are applicable to any services that experience bursts of traffic.

Monday, July 11, 2:20 pm to 2:40 pm — Track 1: Capacity Planning

Fuzz testing has been used to evaluate the robustness of operating system distributions for over twenty years. Eventually, a fuzz test suite will suffer from reduced effectiveness. The first obstacle is the pesticide paradox: as you fix the easy defects, it gets difficult to find the remaining obscure defects. Also, the test execution time and the debug/fix cycle tends to be manual work that can take hours or even days of effort. During the presentation, a structured framework for creating new fuzz tests will be introduced, along with a competitive analysis approach used to minimize defect reproduction complexity.

Wednesday, July 13, 11:00 am to 11:40 pm — Track 1: Wildcard

Multi-tenant architectures are a very convenient and economical way to share resources like web servers, job workers, and datastores among several customers on your platform. Even the smallest Shopify store on a $9/month plan can easily survive getting hammered with a 1M RPM flash sale by leveraging the resources of the entire platform. However, architectures like this can also have several drawbacks. They are potentially harder to scale and things like resource starvation or back-end outages are harder to isolate.

In this talk, I’m going to walk you through the history of how Shopify grew from being a small standard single-database single-datacenter Rails application to the multi-database multi-datacenter setup that we run today. We will talk about the advantages in terms of resiliency, scalability, and disaster recovery that this architecture gives us, how we got there, and where we want to go in the future.

Wednesday, July 13, 11:00 am to 11:40 pm — Track 2: Network 

If you're an engineer interested in life at Shopify, Shannon will be at SREcon: reach out to her through Twitter. To check out open roles or to learn more, head over to our Careers page.

Image credit: Giuseppe Milo

Back to blog