Optimizing multi-module setup behind fat Node.js projects

When building an application with Node.JS you usually reach a point where you want to extract logic from the application and put it into an isolated module. This is typically done when the code turns out to be usable, but not strictly coupled to the application. So instead of keeping it in the app, it gets moved to its own node module, with its own version control repository, where it can be treated separately from the rest of the application. By doing this you gain all kinds of benefits: the code can be shared across multiple consumers, the test runs are scoped to the module by default, the changes of the module are versioned independently from its consumer, open-sourcing becomes an option, which allows collaboration with external people — and so on.

A bit too many cross-functional modules

At Contentful we have various JavaScript applications that are running in the browser or are powering the backend. All of them are using various node modules that are either publicly available or hosted privately. Also the same modules are used in various places. For example we have a library that is connecting our apps to a shared logging system, or to an exception tracker, or to new relic, or to… you get the idea. Additionally, we have modules which bundle all kinds of validations or contain all kinds of error codes and are therefore used in almost every application. In order to prevent unpredictable breakages and problems with not-so-semver version bumps, we typically also have an npm-shrinkwrap file in place that pinpoints a specific version for every module and its sub-modules in the application.

Module management problems

So, while moving isolated logic into its own module generates a lot of advantages, it also makes it more complex to change them in the scope of the entire application and to ensure that such changes are not breaking the application. Typically this is the time when you start cloning the module repository to some place and start doing the npm link dance, which allows you to inject npm modules — that are located somewhere else on your hard drive — into an application. This is a good solution for smaller applications which only have small amount of modules, but it’s getting very frustrating when you have a lot of them, or if the module’s sub-modules are subject to change as well. It is getting even worse if your code contains instanceof checks which makes it crucial to only ever have a single installation of a module.

A possible solution

The following code snippet is about reading an entry via a node module and about handling its success and error case. In the error case it is furthermore checking if the error is of a specific error instance (which live in our errors module) and running different code depending on the result.

A typical problem with this approach can be seen when an application that uses this code and the module that allows reading content entries are running a different installation of the errors module – even though they might use the same version. Suddenly the errors are no longer inheriting from the same module but are two entirely independent objects in memory and therefore the instanceof check is about to fail. This happened to me (and my colleagues) so often when using npm link and running npm install inside the module’s code directory that we decided to search for a way that allows modification of the modules while staying inside the application.

Meet gitify-dependencies

Since the overall idea was about being able to modify the node modules used in an application while having full version control over the changes, we were thinking about replacing a node module with its respective repository. Essentially this meant:

• Go to the module directory that needs to be changed

• Move the dependencies of that module to somewhere else

• Remove the module’s directory

• Create a clone of the repository

• Checkout the right version of the repository

• Move the dependencies of that module back

Since doing this manually for a longer time was a no-go, we looked into automating this process. The result was gitify-dependencies, which is not only replacing the module directories, but also utilizes git-new-workdir to allow collaborative editing on the same repository across multiple working directories.

Let’s imagine the following application and its dependencies:

Using npm dedupe on this will remove all the duplications and reduces the dependencies to the following:

gitify-deps vs npm link

So let’s imagine a situation where a change needs to be introduced to the errors and the validations module. With npm link one would probably do this:

An interesting side effect of this approach can be seen when npm ls is called inside the application folder.

It states that suddenly gitify-dependencies-test-errors is installed as a child of the validations module again, although it was gone previously. This happens because of the npm install step. So while this can be fixed manually, it’s already obvious that this approach is generating all kinds of unwanted side effects.

With gitify-dependencies it looks like this:

Running this will generate the following output:

Gitified modules

After converting every matching module into its respective git repository, it is now possible to cd into them and make changes which are tracked by git.

By doing modifications directly inside the node_modules directory, it is now possible to change the application’s dependencies while keeping the possibility to run the (integration) tests of the app.

Options of gitify-deps

As seen in the example above gitify-deps can be called with some additional option flags. The only required and most important one is the -p or --gitify-url-pattern flag. While iterating through every of the installed packages gitify-deps is comparing the package’s URL property with the flag value and converts the module to its respective git repository if the comparison succeeds. In addition to this, you can use the following ones:

Speed up your workflow now

Installing and getting started with gitify-dependencies is simple and generates a huge value when working in complex applications with lots of dependencies. It allows you to manipulate dependencies as part of your application while keeping track of the changes via git. Just run npm install -g gitify-dependencies; gitify-deps -p <needle> and — profit!

Thanks for reading, and let us know what you think.

Subscribe for updates

Build better digital experiences with Contentful updates direct to your inbox.