Webhook Best Practices

Webhooks are brilliant when you’re running an app that needs up-to-date information from a third party. They’re simple to set up and really easy to consume.

Through working with our third-party developer community here at Shopify, we’ve identified some common problems and caveats that need to be considered when using webhooks. Best practices, if you will.

When Should I Be Using Webhooks?

Let’s start with the basics. The obvious case for webhooks is when you need to act on specific events. In Shopify, this includes actions like an order being placed, a product price changing, etc. If you would otherwise have to poll for data, you should be using webhooks.

Another common use-case we’ve seen is when you’re dealing with data that isn’t easily searchable though the API you’re dealing with. Shopify offers several filters on our index requests, but there’s a fair amount of secondary or implied data that isn’t directly covered by these. Re-requesting the entire product catalog of a store whenever you want to search by SKU or grabbing the entire order history when you need to find all shipping addresses in a particular city is highly inefficient. Fortunately some forward planning and webhooks can help.

Let’s use searching for product SKUs on Shopify as an example:

The first thing you should do is grab a copy of the store’s product catalog using the standard REST interface. This may take several successive requests if there’s a large number of products. You then persist this using your favourite local storage solution.

Then you can register a webhook on the product/updated event that captures changes and updates your local copy accordingly. Bam, now you have a fully searchable up-to-date product catalog that you can transform or filter any way you please.

How Should I Handle Webhook Requests?

There’s no official spec for webhooks, so the way they’re served and managed is up to the originating service. At Shopify we’ve identified two key issues:

• Ensuring delivery/Detecting failure
• Protecting our system

To this end, we’ve implemented a 10-second timeout period and a retry period for subscriptions. We wait 10 seconds for a response to each request, and if there isn’t one or we get an error, we retry the connection several times over the next 48 hours.

If you’re receiving a Shopify webhook, the most important thing to do is respond quickly. There have been several historical occurrences of apps that do some lengthy processing when they receive a webhook that triggers the timeout. This has led to situations where webhooks were removed from functioning apps. Oops!

To make sure that apps don’t accidentally run over the timeout limit, we now recommend that apps defer processing until after the response has been sent. In Rails, Delayed Jobs are perfect for this.

What Do I Do if Everything Blows Up?

This one is a key component of good software design in general, but I think it’s worth mentioning here as the scope is beyond the usual recommendations about data validation and handling failures gracefully.

Imagine the worst case scenario: Your hosting centre exploded and your app has been offline for more than 48 hours. Ouch. It’s back on its feet now, but you’ve missed a pile of data that was sent to you in the meantime. Not only that, but Shopify has cancelled your webhooks because you weren’t responding for an extended period of time.

How do you catch up? Let’s tackle the problems in order of importance.

Getting your webhook subscriptions back should be straightforward as your app already the code that registered them in the first place. If you know for sure that they’re gone you can just re-run that and you’ll be good to go. One thing I’d suggest is adding a quick check that fetches all the existing webhooks and only registers the ones that you need.

Importing the missing data is trickier. The best way to get it back is to build a harness that fetches data from the time period you were down for and feeds it into the webhook processing code one object at a time. The only caveat is that you’ll need the processing code to be sufficiently decoupled from the request handlers that you can call it separately.

Webhooks Sound Magic, Where Can I Learn More?

We have a comprehensive wiki page on webhooks as well as technical documentation on how to manage webhooks in your app.

There’s also a good chunk of helpful threads on our Developer Mailing List.