How Shopify Governs Containers at Scale with Grafeas and Kritis

Today, Google and its contributors launched Grafeas, an open source initiative to define a uniform way for auditing and governing the modern software supply chain. At Shopify, we’re excited to be part of this announcement.

Grafeas, or “scribe” in Greek, enables us to store critical software component metadata during our build and integration pipelines. With over 6,000 container builds per day and 330,000 images in our primary container registry, the security team was eager to implement an appropriate auditing strategy to be able to answer questions such as:

• Is this container deployed to production?
• When was the time this container was pulled (downloaded) from our registry?
• What packages are installed in this container?
• Does this container contain any security vulnerabilities?
• Does this container meet our security controls?

Using Grafeas as the central source of truth for container metadata has allowed the security team to answer these questions and flesh out appropriate auditing and lifecycling strategies for the software we deliver to users at Shopify.

Here’s a sample of some of the container introspection we gain from Grafeas. In this example we have details surrounding the origin of this container including its build details, base image and the operations that resulted in the container layers.

Build Details:

Image Basis:

As part of Grafeas, Google also introduced Kritis, or “judge” in Greek, which allows us to use the metadata stored in Grafeas to build and enforce real-time deployment policies with Kubernetes. During CI, a number of audits are performed against the containers and attestations are generated. These attestations make up the policies we can enforce with Kritis on Kubernetes.

At Shopify we use PGP to digitally sign our attestations, ensuring the identity of our builder and other attestation authorities.

Here’s an example of a signed attestation:

The two key concepts of Kritis are attestation authorities and policies. Attestation authorities are described as a named entity which has the capability to create attestations. A policy would then name one or more attestation authorities whose attestations are required to deploy a container to a particular cluster. Here’s an example of what that might look like:

Given the above attestation authorities (built-by-us and tested) we can deploy a policy similar to this example:

This policy would preclude the deployment of any container that does not have signed attestations from both authorities.

Given this model, then we can create a number of attestation authorities which satisfy particular security controls.

Attestation Examples:

• This container has been built by us
• This container comes from our (or a trusted) container repository
• This container does not run as root
• This container passes CI tests
• This container does not introduce any new vulnerabilities (scanned)
• This container is deployed with the appropriate security context

Given the attestation examples above, we can enable Kritis enforcement on our Kubernetes clusters that ensures we only run containers which are free from known vulnerabilities, have passed our CI tests, do not run as root, and have been built by us!

In addition to build time container security controls we can also generate Kritis attestations for the Kubernetes workload manifests with the results of kubeaudit during CI. This means we can ensure there are no regressions in the runtime security controls before the container is even deployed.

Using tools like Grafeas and Kritis has allowed us to inject security controls into the DNA of Shopify’s cloud platform to provide software governance techniques at scale alongside our developers, unlocking the velocity of all the teams.

We’re really excited about these new tools and hope you are too! Here are some of the ways you can learn more about the projects and get involved:

Try Grafeas now and join the GitHub project: https://github.com/Grafeas

Attend Shopify’s talks at Google Cloud Summit in Toronto on 10/17 and KubeCon in December.

See grafeas.io for documentation and examples.