Farewell require, you’re no longer needed…

This is a really technical article about front-end javascript programming. You have been warned.

Background

For most of the last two years, I have written my front-end code using my own implementation of require. It was inspired by the mostly delightful implementation of require used at Facebook (which I spent had over a month of my life in the bowels of). Initially, I just wanted to use something off-the-shelf, but every implementation I found required a build-phase during development (the Facebook version certainly did) and simultaneously tried to abstract away paths (because during the build phase, something would find all the libraries and map names to paths).

I wanted a require that did not depend on a build phase, did not abstract out paths (so if you knew a problem was in a required library, you also knew where that library was, was easy to use, and supported third-party libraries written to different variations of the commonjs “standard”.

What is require?

const foo = require('path/to/foo.js');

Require allows you to pull code from other files into a javascript file. Conceptually, it’s something like this (the first time).

1. get text from ‘path/to/foo.js’
2. insert it in the body of a function(module){ /* the code from foo.js */ )
3. create an object, pass it to the function (as module)
4. after the function executes, put whatever is in module.exports in a map of library files to output.

It follows that foo.js looks something like this:

module.exports = function() { console.log("I am foo"); }

The next time someone wants that file, pass them the same stuff you gave the previous caller.

Now, if that were all it needed to do, require would be pretty simple. Even so, there’s one nasty issue in the preceding when you want to allow for relative paths — if you require the same module from two different places you don’t want to return different instances of the same thing — both for efficiency and correctness.

But of course there are lots and lots of wrinkles:

• Every javascript file can, of course, require other javascript file.
• Circular references are easy to introduce accidentally, and can become unavoidable.
• Each of these things is a synchronous call, so loading a page could take a very long time (if you put this naive implementation into production).
• Javascript is a dynamic language. It’s already hard to statically analyze, require makes it harder. It’s particularly nasty because require is “just a function” which means it can be called at any time, and conditionally.
• And, just to be annoying, the different implementations of commonjs make weird assumptions about the behavior of the wrapper function and the way it’s called (some expect module to be {}, some expect it to be {exports:{}}, some expect it to be something else, and because require is just javascript can choose to behave differently based on what they see.

Under the hood, require ends up needing to support things like dependency-mapping, circular-reference detection, asynchronous require, inlining pre-built dependency trees, and pretending to be different versions of require for different modules. It’s quite a nasty mess, and it’s vital to the correct functioning of a web app and its performance.

So, good riddance!

ES6 Modules

So, all the major browsers (and nodejs — sort of) finally support ES6 modules and — perhaps more importantly — because this is a change to the language (rather than something implemented in multiple different ways using the language) various build tools support it and can transpile it for less-compatible target platforms.

So now, you can write:

import foo from 'path/to/foo.js';

And the module looks like:

export default function(){ console.log('I am foo'); }

A couple of months ago, I took a few hours and deleted require.js from the b8r source tree, and then kept working on the b8r source files would pass b8r’s (rather meagre) unit tests. I then ran into a hard wall because I couldn’t get any components to load if they used require.

The underlying problem is that components are loaded asynchronously, and import is synchronous. There’s an asynchronous version of import I didn’t know about.

const foo = await import ('path/to/foo.js'); // does not work!

To obtain default exports via import() you need to obtain its default property. So:

const foo = (await import ('path/to/foo.js').default;

In general, I’ve started avoiding export default for small libraries. In effect, default is just another named export as far as dynamic import() is concerned.

import — again because it’s a language feature and not just some function someone wrote — is doing clever stuff to allow it to be non-blocking, e.g.

import {foo} from 'path/to/foo.js';

Will not work if foo is not explicitly exported from the module. E.g. exporting a default with a property named foo will not work. This isn’t a destructuring assignment. Under the hood this presumably means that a module can be completely “compiled” with placeholders for the imports, allowing its dependencies to be found and imported (etc.). Whereas require blocks compilation, import does not.

This also means that import affords static analysis (unlike require, import must be at top level scope in a module). Also, even though import() (dynamic import) looks just like a regular function, it turns out it’s also a new language feature. It’s not a function, and can’t be called etc.

An Interesting Wrinkle

One of the thorniest issues I had to deal with while maintaining require was that of third-party libraries which adhered to mystifyingly different commonjs variations. The funny thing is that all of them turn out to support the “bad old way” of handling javascript dependencies, which is <script> tags.

As a result of this, my require provided a solution for the worse-case-scenario in the shape of a function — viaTag — that would insert a (memoized) script tag in the header and return a promise that resolved when the tag loaded. Yes, it’s icky (but heck, require is eval) but it works. I’ve essentially retained a version of this function to deal with third-party libraries that aren’t provided as ES modules.

b8r without require

Last night, I revisited the task of replacing require with import in b8r armed with new knowledge of dynamic import(). I did not go back to the branch which had a partial port (that passed tests but couldn’t handle components) because I have done a lot of work on b8r’s support for web-components in my spare time and didn’t want to handle a complicated merge that touched on some of the subtlest code in b8r.

In a nutshell, I got b8r passing tests in about three hours, and then worked through every page of the b8r demo site and had everything loading by midnight with zero console spam. (Some of the loading is less seamless than it used to be because b8r components are even more deeply asynchronous than they used to be, and I haven’t smoothed all of that out yet.)

This morning I went back through my work, and found an error I’d missed (an inline test was still using require) and just now I realized I hadn’t updated benchmark.html so I fixed that.

An Aside on TDD

My position on TDD is mostly negative (I think it’s tedious and boring) but I’m a big fan of API-driven design and “dogfooding”. Dogfooding is, in essence, continuous integration testing, and the entire b8r project is a pure dogfooding. It’s gotten to the point where I prefer developing new libraries I’m writing for other purposes within b8r because it’s such a nice environment to work with. I’ve been thinking about this a lot lately as my team develops front-end testing best practices.

Here’s the point — swapping out require for import across a pretty decent (and technically complex) project is a beyond a “refactor”. This change took me two attempts, and I probably did a lot of thinking about it in the period between the attempts — so I’d call it a solid week of work.

“5 Story Points”

I’ve done a bunch of refactors of b8r-based code (in my last month at my previous job two of us completely reorganized most of the user interface in about a week, refactoring and repurposing a dozen significant components along the way). Refactoring b8r code is about as pleasant as I’ve ever found refactoring (despite the complete lack of custom tooling). I’d say it’s easier than fairly simple maintenance on typical “enterprisey” projects (e.g. usually React/Redux, these days) festooned with dedicated unit tests.

Anyway, even advocates of TDD agree that TDD works best for pure functions — things that have well-defined pre- and post- conditions. The problem with front-end programming is that if it touches the UI, it’s not a pure function. If you set the disabled attribute of a <button>, the result of that operation is (a) entirely composed of side-effects and (b) quite likely asynchronous. In TDD world you’d end up writing a test expecting the button with its disabled attribute set to be a button with a disabled attribute set. Yay, assignment statements work! We have test coverage. Woohoo!

Anyway, b8r has a small set of “unit tests” (many of which are in fact pretty integrated) and a whole bunch of inline tests (part of the inline documentation) and dogfooding (the b8r project is built out of b8r). I also have a low tolerance for console spam (it can creep in…) but b8r in general says something if it sees something (bad) and shuts up otherwise.

Anyway, I think this is a pretty pragmatic and effective approach to testing. It works for me!

What’s left to do?

Well, I have to scan through the internal documentation and purge references to require in hundreds of places.

Also, there’s probably a bunch of code that still uses require that for some reason isn’t being exercised. This includes code that expects to run in Electron or nwjs. (One of the nice things about removing my own implementation of require is that it had to deal with environments that create their own global require.) This is an opportunity to deal with some files that need to be surfaced or removed.

At that point there should be no major obstacle to using rollup.js or similar to “build” b8r (versus using the bespoke toolchain I build around require, and can now also throw away). From there it should be straightforward to convert b8r into “just another package”.

Is it a win …yet?

My main goal for doing all this is to make b8r live nicely in the npm (and yarn) package ecosystem and, presumably benefit from all the tooling that you get from being there. If we set that — very significant — benefit aside:

• Does it load faster? No. The old require loads faster. But that’s to be expected — require did a lot of cute stuff to optimize loading and being able to use even cleverer tooling (that someone else has written) is one of the anticipated benefits of moving to import.
• Does it run faster? No. The code runs at the same speed.
• Is it easier to work with? export { .... } is no easier than module.exports = { .... } and destructuring required stuff is actually easier. It will, however, be much easier to work with b8r in combination with random other stuff. I am looking forward to not having to write /* global module, require */ and 'use strict' all over the place to keep my linters happy..
• Does it make code more reliable? I’m going to say yes! Even in the simplest case import {foo} from 'path/to/foo.js' is more rigorous than const {foo} = require('path/to/foo.js') because the latter code only fails if foo is called (assuming it’s expected to be a function) or using it causes a failure. With import, as soon as foo.js loads an error is thrown if foo isn’t an actual export. (Another reason not to use export default by the way.)