For three years we, Shopify’s Application Security team, have set aside time to reflect on our bug bounty program and share recent insights. This past year has been quite a ride as our program has been busier than ever! We’re excited to share what we have learned and some of the great things we have planned for the rest of 2021.

Recent Program Highlights

H1-2102 Virtual Live Hacking Event

One of our main program goals is to attract the best talent on the HackerOne platform and show them why Shopify is the program they should be hacking on. Live hacking events are a great way to build these collaborative relationships with hackers. We recently ran H1-2102 in collaboration with HackerOne, our first Virtual Live Hacking Event (and third live hacking event overall). We invited 38 skilled hackers from 7 different countries to hack on areas of Shopify that have never been in scope on the main program like Plus organizations and Shopify Support Services. We received 83 valid reports and paid out over $220,000 in bounties and $54,000 in bonuses. Some extremely creative bugs came out of this event, so keep an eye on our hacktivity to see those disclosures! Congratulations to @ngalog, @rhynorater, and @francisbeaudoin for taking top 3 on the event leaderboard.

Bounty Milestones

At Shopify, Trust and Security are our top priorities and our bounties demonstrate our commitment to both. For this reason, we increased our max bounty to $50,000 at the beginning of 2020. This had an immediate impact for our program as @ngalog reported multiple critical vulnerabilities in Shopify ID (our single-sign on service) just days after that increase. Disclosing his amazingly creative research in this area is incredibly valuable for our program, and many hackers are building on that foundation to report other issues. This is a shining example of the value of attracting top hackers with competitive maximum bounties and disclosing their reports to help level up the community as a whole. We also saw a large increase in report volumes after those disclosures, receiving approximately triple our usual report volume in the month that followed. We also hit a huge milestone of awarding over $2 million in total bounties across our programs and paying out more than $460,000 in bounties over the course of 2020 alone. With our increased max bounty, we see many hackers dig deep on the platform and submit very impactful reports.

Paying Bounties via Common Vulnerability Scoring System

As part of our partnership with hackers, we want our bounties to be fair, reliable and consistent over time. We realized there was room for improvement, so in October 2020 we began a month-long experiment. We retired our bounty table and began calculating our bounty payments via the Common Vulnerability Scoring System (CVSS) with the goal of implementing any permanent changes in January 2021. However, the experiment was such a resounding success that we immediately made it our main method of determining bounties. Using CVSS increases our commitment to transparency with hackers because we share a breakdown of the metrics involved in each score. Additionally, we released the public calculator we use for determining bounties, so hackers can assess the impact of their own findings. Ultimately, using CVSS led to more objective discussions about impact, both internally and with hackers reporting to our program. This severity-based approach to bounties also ensures we’re paying based on the real-world impact for our merchants and their buyers.

GraphQL Hacking Guide

We know that return on investment is critical for hackers, and we want to remove as many barriers to testing as we can. In September 2020, we released a guide on Hacking GraphQL at Shopify to chip away at one of those barriers. GraphQL is the cornerstone of our applications, and the investment our engineering team is making in this area only continues to grow. This guide walks ethical hackers through

• getting the full GraphQL schema including functionality only available in the unstable version
• performing queries against our Admin API
• automatically detecting changes in the schema.

After its release, we saw a noticeable increase in the amount of reports regarding undocumented areas in our Admin API. These types of resources ultimately help hackers find bugs faster, and help us squash them as quickly as possible.

2021 Program Improvements

Dedicated Bug Bounty Team

Our bug bounty program was busier than ever in 2020. To keep up with the higher volume of incoming reports and ensure duplicates were low, the Application Security team ran monthly bug squash sprints throughout the year. The results were overwhelmingly positive, so we decided to create a permanent team dedicated to our HackerOne program. This shift allows us to improve our response times and further our goal of being the most responsive program on the platform.

We still track our duplicates and days to resolve as key metrics of success. We’d like to see less than 10 duplicate reports each month as this was as high as 38 in our busier months in 2020 (before starting our bug squash initiative). As an added bonus, the new team also increases the Application Security team’s bandwidth to do a variety of other awesome things this year, some of which we’ve described in the rest of this post.

More Resources for Hacking on Shopify

We always work to remove barriers for hackers and help them track down the most interesting areas of Shopify to focus their testing. In 2019, we added Shopify’s Changelog (the main source for recent updates on the platform) to our program page and encouraged hackers to monitor it for new functionality. Last year, we started the Bug Bounty Resources repository to provide more HackerOne-focused references to hackers and make it easier to get started on the Shopify program.

We have big things planned for additional reference material, starting with tips and tricks for testing apps that authenticate using session tokens. We ultimately want to provide all of the tools and foundation that we can and chip away at any barriers to entry when hacking on our program. We recommend watching the Bug Bounty Resources repo if you’re interested in hacking on Shopify! If you have any requests for particular resources that you’d like to see, feel free to send those ideas to us at bugbounty@shopify.com.

Followup Responses

With our report volume reaching all-time highs this year, it became more of a challenge to follow up on hacker questions, particularly on closed reports. We weren't meeting our commitment to answering all inquiries about our report decisions. To fix this, we’re developing tooling to surface reports that require follow-up from our team. Our internal dashboard allows us to filter based on reports that a hacker last commented on so we can easily see which reports are awaiting responses.

Shopify Experiments Private Program, Version 2.0

In 2019, we launched a private program called Shopify Experiments where we have run a series of experiments with the goal of improving our public program. You can read more about these experiments in our post from last year, Bug Bounty Year in Review 2019. We’re currently working on a revised version of this program that focuses on special apps and features that aren’t yet publicly available or feasible to test on our main program. We plan to run “mini events” throughout the year to highlight these new targets with other promotions and bonuses attached. The relaunch date for this hasn’t yet been set, but invites will be going out within the next few months to hackers based on their reports from the last 12 months. These stats will be reviewed monthly with invites sent at the start of each month. The criteria for an invite is

• At least 4 bounty-eligible reports Triaged or Resolved
• No more than 25% of submissions closed as Informative or Not Applicable on the main program

Hackers must also maintain the same signal to noise ratio on the Experiments program itself. We’ll be reviewing the list of hackers on this program quarterly and ensuring that all those invited are meeting this requirement and actively engaging on the program.

2020 Bug Bounty Statistics

Hackers kept us very busy throughout 2020! We saw a huge overall increase in volume with 3,093 reports compared to 1,379 in the previous year. When looking at the monthly breakdown, we see that April and September were particularly busy months, each with well over 400 reports. These coincided with some very interesting disclosures that brought extra attention to our program. 

In April, we disclosed @ngalog’s reports on the Shopify ID merge flow and in September @francisbeaudoin’s email confirmation bypass. We continue to see tremendous value in public disclosure. It brings additional attention to our program and helps hackers learn from one another resulting in an overall increase in the quality of reports that we, and other programs, receive.

Taking a look at the breakdown of report states throughout 2020, we see some interesting new trends as well. Overall, there was a significantly higher proportion of Informative and Not Applicable reports. Specifically, we saw more than a 300% increase in Informative and an 80% increase in Not Applicable when compared to 2019, though we didn’t make any changes to our policy that would have prompted these increases. Looking at this month by month, we again see huge spikes around key disclosures and a general upward trend throughout the year. This could be due in part to changes in how Signal is calculated on HackerOne, and hackers taking more chances with Informative reports. It’s also possible that restrictions due to COVID-19 contributed to this increase (as well as the overall increase in volume over the course of 2020), as more hackers were at home and may have been looking to supplement income with bounties.

 

Clear, consistent and timely communication is the cornerstone of our program. Shopify strives to be among the most responsive programs on the HackerOne platform and we worked hard to keep our communication times low under the increased volume. We did lose a little ground during these spikes, but overall we were able to keep our response times similar to those in 2019. Our average time to first response was 25 hours compared to 16 in the 2019. Overall, we triaged a significantly higher amount of reports, 215 in 2020, compared to 131 in 2019. The average time to triage for these increased slightly from 2 days and 13 hours in 2019 to 3 days 11 hours.

Bounty decisions also took a bit longer toward the end of 2020, largely due to the transition to paying bounties via CVSS. For this reason, time to bounty increased from 7 days and 1 hour in 2019 to 12 days and 15 hours in 2020. We continue to meet as a team twice a week to discuss and align on our scores for each report, and have significantly streamlined this process to bring that metric back down.

We also awarded significantly more bounties in 2020, paying out over $460,000 compared to approximately $130,000 in 2019. This was due in part to the overall spike in report volume, but also by increasing our maximum bounty to 50k at the start of 2020. This led to a higher average bounty of $2,070 in 2020 compared to $1,139 the previous year.

The last year on our program was an exciting one, and we can’t wait to see what the rest of 2021 brings. If you’re interested in helping to make commerce more secure, visit our program page to start hacking or our careers page to check out our open Trust and Security positions.

Happy Hacking!

- Shopify Trust and Security

Learn More About Shopify's Bug Bounty Program

• Sharing the Philosophy Behind Shopify's Bug Bounty
• 2017 Bug Bounty Year in Review
• Bug Bounty Year in Review 2018
• Bug Bounty Year in Review 2019
• One Million Dollars in Bug Bounties
• Building Shopify’s Application Security Program

---

Wherever you are, your next journey starts here! If building systems from the ground up to solve real-world problems interests you, our Engineering blog has stories about other challenges we have encountered. Intrigued? Visit our Engineering career page to find out about our open positions and learn about Digital by Default.