The Shopify CLI (command line interface) is an essential tool for developers when building and deploying Themes, Apps, Hydrogen storefronts on the Shopify platform. It provides workflows to create new projects that follow best practices, integrate with the platform during development, and distribute the production artifacts to merchants. My team, CLI Foundations, is responsible for devising and building Shopify CLI’s foundation of best practices and core functionality. We know that developers live in the terminal as they develop their Shopify apps, and using the CLI wasn't always the most consistent and enjoyable experience. So we embarked on a complete rewrite of the Shopify CLI 2, which was written in Ruby, into Node, and launched it in Shopify Editions last summer. In this blog post, I'll explain past decisions, the team’s decision to do a rewrite and the tradeoffs, the principles we embraced in this new iteration, and the challenges and ideas ahead of us to explore.

Revisiting the Language Decision

Before Shopify CLI, theme developers were using another CLI of ours, ThemeKit, which we had been maintaining since October 2014. It was written in Ruby and built on Ruby gems like cli-kit, cli-ui, and theme-check that we use in internal CLIs and services.

When we started working on the first Shopify CLI to help App developers in December 2018, Ruby was a sensible choice considering the resources and knowledge that we had built around the language. Users would need to have a global Ruby installation to use the CLI, but we took care of that by providing installers for all the supported OSs (Windows, Linux, and macOS). In December 2020, we took the first step towards centralizing all the development in a single CLI by merging ThemeKit into the Shopify CLI. 

With the addition of UI extensions in June 2020 to allow developers to extend some areas of the platform with their own UI, the CLI started depending on Node tooling for transpiling and bundling extension code. The system requirements were increasing, which wasn’t desirable for users, but the ecosystem was trending towards JavaScript tooling implemented in compiled languages like Go and Rust, so we were hopeful that the dependency with Node would go away. That didn’t happen. Even though tools like ESBuild—which we used for bundling extensions—are portable binaries, their extensibility relied on plugins that were dynamically evaluated on a Node runtime. 

Moreover, the Hydrogen team, which had built some tools on Node, started to consider building a new CLI instead of building Hydrogen workflows into the Shopify Ruby CLI so their users didn’t need a Ruby runtime in their system. Hydrogen developers expect the npm install command to resolve all the dependencies that they need to work on a project. With Ruby as a dependency, that mental model breaks and they can easily run into issues where the CLI refuses to run because of additional steps needed. Having another CLI would break the unification effort that we started with the merge of ThemeKit into the CLI. It could potentially lead to inconsistent CLI experiences across different areas of the platform. 

And last but not least, Shopify leans more and more on web technologies and standards where JavaScript and the Node runtime are well positioned in terms of resources, tooling, and knowledge.  

All the above prompted us to think about whether Ruby was the most suitable language for the CLI, so we revisited the decision. We needed a technology that:

• has as few system requirements as possible (for example, not having to install multiple runtimes) 
• allows us to deliver a top-notch developer experience
• is easy for internal teams to contribute. 

We ultimately landed on the decision of rewriting the CLI into TypeScript to run on the Node runtime.

Moving From Ruby to Node

Of all the programming languages that are used at Shopify, Ruby is the one that most developers are familiar with, followed by Node, Go, and Rust. All of them support building a top-notch developer experience thanks to the rich ecosystem of packages to solve commonly-encountered problems. And from those choices, Go and Rust are ones that can easily ship a static binary that has no dependency with the runtime in which it runs. The same can be achieved with Node and Ruby by bundling the source code and the runtime dependencies (aka vendoring) with a more convoluted setup and potentially leaving some OSs unsupported.

We decided on Node for a few reasons. Go and Rust allow distributing a static binary, but at the cost of fewer people at Shopify being inclined to contribute due to the lack of familiarity with the language. This isn’t ideal because Shopify wants internal teams to contribute new ideas into the CLI. We were left with either Ruby or Node. 

Node has a feature that sets it apart from Ruby for building CLIs: its module system and the extensibility that it enables. Unlike Ruby, Node’s module system allows having multiple versions of the same transitive package without conflicting with each other. This allows us to build a modular architecture where we encapsulate different domains of the platform in NPM packages, and all of them are built on another package that contains shared functionality. There’s a caveat that we’re aware of—while Hydrogen and App developers only require one runtime (Node), Theme developers need two now: Ruby and Node. However, we’re already working on removing the Ruby dependency and we aim to complete the work later this year.

Building Great Terminal Experiences

We made the technology decision, but we still had to make some decisions around best practices, code architecture, patterns, and conventions. It was an exercise of learning from various groups and our experience building the Ruby CLI. I’ll share the seven decisions that have had the most positive impact for us when building great terminal experiences.

1. Creating A Foundation for Consistency

Ruby CLI contributions were loosely aligned and loosely coupled (as opposed to our desired state of highly aligned, loosely coupled), resulting in internal and external fragmentation that led to bad experiences. We had to do something different for the Node version. We needed a way to align contributions. We achieved so through:

• Code patterns: Model commands’ business logic. In projects powered by a framework, like Rails, these patterns are usually encouraged by the framework (for example, MVC), but there was no framework. So we had to develop our patterns and mechanisms to ensure developers followed them.
• UI patterns and components: We designed and built a design system on Ink to ensure all commands look and feel like Shopify.
• Conventions:  Make it easier for developers to navigate across their projects and commands.
• Principles: Build great experiences that make developers fall into the pit of success.

The above materialized in a shared NPM package, @shopify/cli-kit that all the domains (Themes, Apps, and Hydrogen) would build on, and a set of general principles to apply within the CLI and in any surface of our platform developers interact with (for example, docs and partners dashboard). Static analysis tools like ESLint became our platform to build automation to ensure contributions aligned with the foundation and direction of the CLI. We implemented custom rules like command-flags-with-env that ensures flags support being set through environment variables.

The example below shows an idiomatic API that feature developers can use to get a valid session to interact with the GraphQL APIs.

2. Ensuring Cross-OS Support

Ensuring code changes supported macOS, Windows, and Linux while developing in a macOS environment was a cumbersome process that led to testing being skipped and regressions arising. The Node runtime would add to the problem since some of the APIs are known for offering inconsistent behavior across OSs. The community is overcoming them with NPM packages. For example, pathe normalizes paths across OSs. To prevent the same thing from happening, we adopted the three strategies:

1. We provided modules from @shopify/cli-kit to interact with the environment (for example, IO operations) and ensure their APIs are cross-OS compatible. If we detect an OS incompatibility, we’ll fix it once and for all.
2. Our suite of tests is a mix of unit, integration, and end-to-end (E2E) tests that run in all the supported OS on continuous integration (CI). It surfaces issues in PRs before they get merged. We write integration tests for modules that contract with the environment (for example, a module that provides utilities for interacting with Git).
3. We provided instructions for testing changes in MacOS, Linux, and Windows environments. 

3. Moving to a Monorepo

Conway's law manifested in our organization with a proliferation of repositories containing different components of the CLI (for example, templates and internal CLIs). This multi-repo setup brought zero value to the users but made internal contributions and automated testing more cumbersome. We decided to take the rewrite as an opportunity to revert this and bring all the components into the same repository, shopify/cli. The monorepo setup allowed contributing changes that span multiple packages and templates atomically.

4. Embracing Functional Programming

Commands' business logic in Ruby CLI was very stateful, had many assumptions, and had side effects scattered across command lifecycles. This made code harder to reason about, contribute, and test. With Node CLI, we took a different approach.

We minimized side effects by designing logic more functionally and concentrating side effects, whenever possible, at the beginning of a command. For example, the first thing that commands do is load and validate the project in memory. It’s similar to what a web API does when it receives a request; it validates it before passing it down to the system in a potentially invalid state that might have cascading effects. We weren’t dogmatic about the functional paradigm, but we aimed for the logic to be a composition of functions that passes state around.

We use JavaScript objects and functions as units of composition, and we default to creating copies of objects over mutating passed instances. There are only a few scenarios where we resorted to classes to conform to the language, for example, for error types. We introduced a soft convention around the organization of functions that resembles the Model View Controller (MVC) architectural pattern:

• Models (Model): They’re TypeScript interfaces to model the state. Some examples are the internal representation of an App project, its configuration, and the session.
• Commands (View): They’re the surface users interact with by passing arguments and flags when invoking the CLI. Their responsibility is limited to parsing and validating arguments and flags, and providing the content for the help menu.
• Services (Controller): They’re the unit of encapsulation of business logic. All commands have a service containing the commands’ business logic, and some services aren’t tied to a specific command.

Aside from the above, we also have prompts, which contain functions for prompting the user through standard input, and utilities that group a set of functions under a particular domain. An example is all the functions to interact with Shopify GraphQL APIs.

5. Implementing an End-to-end Testing Strategy

Embracing functional programming and minimizing side effects made writing unit tests more accessible. To define and run them, we used Vitest, which had just been announced weeks before we started working on Node CLI. We decided on Vitest because it fully supported ES modules (the module system we adopted). Despite encountering some initial bumps expected from a tool going through a maturity process, we’re happy with the experience and the one-to-one mapping with Jest APIs.

Unit tests gave us confidence that our functions did what they were supposed to do in different scenarios, but it was not enough—a successful result running the unit tests suite wasn’t indicative of a workflow like “app build” running successfully with a recently created project. Therefore, we decided to invest in a suite of E2E tests with Cucumber to ensure various workflows worked end to end. Cucumber provided us with the tools and APIs to describe, run, and debug those tests. As you might know, E2E is famous for being cumbersome to maintain and potentially introducing flakiness. However, that’s not true in CLIs, where the setup is simpler. The execution can be isolated and scoped to the test scenarios, preventing the global state from leaking into other tests and causing them to behave oddly. Our tests look like the example below:

6. Leveraging TypeScript

TypeScript’s type system and compiler gives us the confidence that the contracts between units of code and with external dependencies match. Many NPM packages the CLI depends on and the modules provided in @shopify/cli-kit provide type definitions that significantly improve the ergonomics of contributing to the repository. For instance, we’re implementing the components of a new design system we’ve designed for the CLI, and we’re leveraging TypeScript extensively to ensure developers use the components the right way.

7. Building Community-tested Foundation

In one of our early conversations with developers working on CLIs outside of Shopify, oclif came up as an excellent framework of tools and APIs to build CLIs in Node. For instance, it was born from Heroku’s CLI to support the development of other CLIs. After we decided on Node, we looked at oclif’s feature set more thoroughly, built small prototypes, and decided to build the Node CLI on their APIs, conventions, and ecosystem. In hindsight, it was an excellent idea.

Oclif provides us with idiomatic APIs for declaring the CLIs interface and comes with outstanding customizable defaults. For example, the help documentation gets automatically generated from in-code declarations. Moreover, it has extensibility built in through a plugin system that we’re already leveraging with plugins for App, Theme, and Hydrogen development. It allowed us to organize the project into modules that had clear boundaries and well-separated responsibilities. We leveraged oclif’s hooks API to prevent @shopify/cli-kit from having knowledge on the dependent plugins using dependency inversion. Plugins and @shopify/cli-kit implementations depend on interfaces.

Looking to What’s Next for Node CLI

Node CLI is a significant improvement in developer experience: we unified and streamlined the development of apps, brought consistency across the board, and added new extensibility capabilities like functions. However, we still have a long way to go and a lot to learn. 

We want to align the Theme development experience with App and Hydrogen’s. Currently, Theme commands still run through the Ruby implementation presenting users with the Ruby CLI look and feel, and developers are required to have the Ruby runtime in their environment, which isn’t ideal. 

We’ll also continue iterating on the experience of developing apps, and provide developers with commands that are useful in the journey of creating, developing, and deploying an app to the platform. Since the announcement of a better developer experience for building apps, we’ve received valuable feedback that we’re iterating on, for example some difficulties moving to the unified app model from a multi-repo setup. Our teams also have a great foundation to prototype new ideas, and we’re excited to share our progress in the future.

And there’s a lot more to share about extensibility, but this is a topic for a future blog post.

We all get shit done, ship fast, and learn. We operate on low process and high trust, and trade on impact. You have to care deeply about what you’re doing, and commit to continuously developing your craft, to keep pace here. If you’re seeking hypergrowth, can solve complex problems, and can thrive on change (and a bit of chaos), you’ve found the right place. Visit our Engineering career page to find your role.