Migrating Large TypeScript Codebases To Project References
Share
In 2017, we began migrating the merchant admin UI of Shopify from a traditional Ruby on Rails Embedded RuBy (ERB) based front-end to an entirely new codebase, TypeScript paired with React and GraphQL. Using TypeScript enabled our ever-growing Admin teams to leverage TypeScript’s compiler to catch potential bugs and errors well before they ship. The VSCode editor also provides useful TypeScript language-specific features, such as inline feedback from TypeScript’s type-checker.
Many teams work independently but in parallel on the Shopify merchant admin UI. These teams need to stay highly aligned and loosely coupled to ensure they quickly ship reliable code. We chose TypeScript to empower our developers with great tools that help them ship with confidence. Pairing TypeScript with VSCode provides developers with actionable real-time feedback right in their editor without the need to run separate commands or push to CI. With these tools, teams leverage the benefits of TypeScript’s static type-checker right in the editor as they pull in the latest changes.
The Problems with Large TypeScript Codebases
Over the years as teams grew, many new features shipped, and the codebase dramatically increased in size. Teams had a poorer experience in the editor. Our codebase was taxing the editor tooling available to use. TS-Server (VSCode’s built-in language server for TypeScript) would seemingly halt, leaving developers with all the weight of TypeScript’s syntax and none of the editor tooling payoff. The editor took ~2-3 minutes to load up all of TypeScript’s great tooling, slowing how fast we ship great features for our merchants, and it also created a very frustrating development experience.
We need to ship commerce features fast, so we investigated solutions to bring the developer experience in the editor up to speed.
Understanding What VSCode Didn’t Like About Our Codebase
First, we started by understanding why and where our codebase was taxing VSCode and TypeScript’s type-checker. Thankfully, the TypeScript team has an in-depth wiki page on Performance, and the VSCode team has an insightful wiki page on Performance Issues.
Before implementing any solutions, we needed to understand what VSCode didn’t like about our codebase. To my pleasant surprise, around this time Taryn Musgrave, a developer from our Orders & Fulfillment team, had recently attended JSConfHi 2020. At the conference, she met a few folks from TC39 (the team that standardizes the JavaScript language under the “ECMAScript” specification) , one of them being Daniel Rosenwasser, the Program Manager for TypeScript at Microsoft. After the conference, we connected and shared our codebase diagnostics with the core TypeScript team from Microsoft.
The TypeScript team had asked us for more insight into our codebase. We ran some initial diagnostics against the single TypeScript configuration (tsconfig.json
) file for the project. The TypeScript compiler provides an --extendedDiagnostics
flag that we can pass to get some useful diagnostic info
Their team was surprised by the size of our codebase and mentioned that it was in the top 1% of codebase sizes they’d seen. After sharing some more context into our tools, libraries, and build setup they suggested we break up our codebase into smaller projects and leverage a new TypeScript feature released in 3.0 called project references.
Measuring Improvements
Before diving into migrating our entire admin codebase, we needed to take a step back to decide how we could measure and track our improvements. To confidently know we’re making the right improvements, we need tools that enable us to measure changes.
At first, we used VSCode’s TSServer logs to measure and verify our changes. For our Admin codebase the TSServer would spit out ~80,000 lines of logs over the course of ~2m30s on every bootup of VSCode.
We quickly realized that this approach wasn’t scalable for our teams. Expecting teams to parse through 80,000 lines to verify their improvement wasn’t feasible. So, for this reason we set out to build a VSCode plugin to help our teams at Shopify measure and track their editor initialization times over time. Internally we called this plugin TypeTrack.
TypeTrack in action
TypeTrack is a lightweight plugin to measure VSCode's TypeScript language feature initialization time. This plugin is intended to be used by medium-large TypeScript codebases to track editor performance improvements as projects migrate their code to TypeScript project references.
Migrating to Project References
In large projects, migrating to project references in one go isn’t feasible. The migration would have to be done gradually over time. We found it best to start out by identifying leaf nodes in our project’s dependency graph. These leaf nodes generally include the most widely shared parts of our codebase and have little to no coupling with the rest of our codebase.
Our Admin codebase’s structure loosely resembles this:
In our case, a good place to start was by migrating the packages and tests folders. Both of these folders are meant to be treated as “isolated projects” already. Migrating them to leverage project references not only improves their initialization performance in VSCode, but also encourages a healthier codebase. Developers are encouraged to break up their codebase into smaller pieces that are more manageable and reusable.
Starting At The Leaves
Once we’ve identified our leaf nodes, we start by creating an entrypoint for the project references.
Whenever a project is referenced, that respective folder is also migrated to a project reference, meaning it requires a project-level tsconfig specifying the dependencies that project references in turn. In the case of the above-mentioned project-references.tsconfig.json
, the ../../package
path reference looks for a packages/tsconfig.json
.
Once you’ve got a few folder with project references created, you run the TypeScript compiler to check if your project builds:
> Protip: Use the --diagnostics
flag to get insight into what the compiler is doing.
At this point it’s very likely that you’ll get tons of errors, the most common being that the compiler can’t find a module that your project is referencing.
To illustrate this, see the screenshot below. The compiler isn’t able to find the module @shopify/address-consts
from within the @shopify/address
package
Compiler can’t find a module the project is referencing
The solution here would be:
- Create a project-level
tsconfig.json
for the module being depended on. You can think of this as a child node in your project dependency graph. For our example error from above, we need to create atsconfig.json
inpackages/address-consts
. - Reference the new project reference (the
tsconfig.json
created from 1) in the consuming parenttsconfig.json
that requires the new module. In our example, we need to include the new project reference.
Keep in mind that you'll have to respect the following restrictions for project references:
- The include/files compiler option must include all the input files that the project reference relies on.
- Any project that’s referenced must itself have a references array (which may be empty).
- Any local namespaces you import must be listed in the
config/typescript/tsconfig.base.json
#compilerOptions#paths field.
General Steps to Migrating Your Codebase to Project References
Once the TypeScript compiler successfully builds the leaf nodes that you’ve migrated to project references, you can start working your way up your project dependency graph. Outlined below are general steps you’ll take to continue migrating your whole codebase:
- Identify the folder you plan to migrate. Add this to your project’s
config/typescript/project-references.tsconfig.json
- Create a
tsconfig.json
for the folder you’ve identified. - Run the TypeScript compiler
- Fix the errors. Determine what other dependencies outside of the identified folder need to be migrated. Migrate and reference those dependencies.
- Go to step 3 until the compiler succeeds.
For large projects, you may see hundreds or even thousands of compiler errors in step 4. If this is the case, pipe out those errors to a log file and look for patterns in the errors.
Here are some common solutions we’ve found when we’ve come across this roadblock:
- The folder is just too large. At the root of the folder create a
tsconfig.json
that includes references to child folders. Migrate the child folders with the general steps mentioned above - This folder contains spaghetti code. It’s possible this folder could be reaching into other dependencies/folders that it shouldn’t. In our case we’ve found it best to move shared code into our
packages
folder consisting of shared isolated packages.
We Saw Drastic Improvements in VSCode
Once we migrated our packages and tests folders in our Admin codebase, we made great improvements in the amount of time it takes VSCode to initialize the editor’s TypeScript language features in those folders.
File | Before | After |
packages/@web-utilities/env/index.ts | ~155s | ~13s |
packages/@shopify/address-autocomplete/AddressAutocomplete.ts | ~155s | ~8s |
tests/utilities/environment/app.tsx | ~155s | ~10s |
How Does This Make Sense? Why Is This 10x Faster?
When a TypeScript file is opened up in VSCode, the editor sends off a request to the TypeScript Server for analysis on that file. The TS Server then looks for the closest tsconfig.json
relative to that file for it to understand the types associated with that project. In our case, that’s the project-level tsconfig.json
which included our whole codebase.
With the addition of project references (i.e tsconfig.json
with references to its dependent projects) in the packages
and tests
folders, we’re explicitly separating the codebase into smaller blocks of code. Now, when VSCode loads a file that uses project references, the TS Server will only load up the files that the project depends on.
Scaling Migration Duties to Other Teams
In our case, given the size of our Admin codebase, my team couldn’t migrate all of it ourselves. For this reason we wrote documentation and provided tools (our internal VSCode plugin) to other Admin section teams to improve their section’s editor performance.
We're always on the lookout for talent and we’d love to hear from you. Visit our Engineering career page to find out about our open positions.