b8r goes pure(ish) Javascript

b8r now allows components to be implemented entirely in Javascript, with no eval

In an effort to make b8r work in the universe of “everything must be javascript” web development tools, I’ve finally added provision for pure javascript components in b8r.

Now, there’s four really good reasons for doing this.

First of all, the way b8r’s HTML-based components work intrinsically involves using eval (it actually uses the AsyncFunction constructor, but same difference). Whether this represents a real security threat or not depends a lot on how those components are in fact loaded, but it’s definitely an argument.

Second — and this has proven a small but persistent pain point with b8r development — it makes linters happy. To begin with, the <script> tag inside a component was actually the body of an async function with a ton of parameters. Using these parameters without annoying the lint gods involved /* global … */ to be added at the top of the scripts, while the fact that — for convenience — the code was async meant that linters would scream about use of import() and async/await.

Third — with HTML components, defining a single global controller for a family of components involved some jumping through hoops as did, for example, using computed bindings or lists where the method was registered in the component body (this works fine, but generates error messages if you don’t programmatically add the bindings after the necessary methods are registered). Now you can just do the necessary work outside the component definition (but in the same file!)

Fourth — while b8r has moved over to the world of ES6 Modules, it is distributed in cjs form for those still living with require, webpack, etc. The problem is that HTML components don’t play nicely with javascript bundlers, whereas javascript components are just javascript so if you really want to force a giant slab of javascript down your users’ throats you can. Yay.

So, these issues are fixed if you use pure javascript components.

You are still writing CSS as CSS and HTML as HTML, but now it’s likely going to be inside back-ticks.

In addition to the four big arguments for building components this way, there are some further implications (mostly good).

First of all, if and when you need to use import() inside a component, you can use relative paths without wondering “relative to what” (it used to be the location of b8r.js, because that’s where that code ends up executing).

Next, there was no convenient way to create multiple components in one source file. This is a mixed blessing — making it easy to define lots of small components in source files with no relationship between file name and component name presents the real risk of name-clashes. It may become necessary to introduce lint rules that prevent you from creating a component whose name does not match its source file in some way.

Finally, a really nice side-effect of this is that the boilerplate for a component is a lot more self-explanatory than it used to be. Yay!

Dark Mode with CSS Variables

All the cool kids are doing it, what’s the easiest way to support Dark Mode?

As little as four or so years ago, it was widely understood among front-end developers that CSS was a horrible problem, managing it was a nightmare, and Something Ought to be Done About It.

Back then, there were two popular “solutions”, LESS and SCSS, that both allowed you to transpile your CSS from a superset of CSS that allowed things like named constants and nesting rules. There was also the drum beat of the “all web development should be Javascript development” which advocated replacing CSS with Javascript (that created CSS). Incidentally, these were the same people who thought “all Javascript development should be some weird superset of Javascript that needs to be transpiled into Javascript”.

Anyway, the big problem LESS, SCSS, and the CSS-should-be-Javascript folks tried to solve was managing consistency across large sets of rules (no-one of course questioning the idea of having such large sets of rules). The downside was that this enabled even bigger sets of rules. (The best reason for using LESS or SCSS was the implementation of named constants. With these part of CSS, it seemed clear to me that their reason for existence was largely gone.)

If you reduce the cost of something, folks will use more of it. LESS, SCSS, and styling with Javascript reduced the apparent cost of adding CSS rules to web apps.

E.g. on a project I was working on at the time, about 10kB of LESS became 280kB of CSS — and this was on a team that was (a) small, and (b) ruthlessly minimized CSS whenever it could.

At this point, I can safely say “a pox on all your houses”. With bindinator’s approach of using a solid set of global styles and component-specific styles that are easy to scope to specific components (without the use of a “shadow DOM”) we never ran into any of the thorny CSS issues that plague other projects I’ve worked on, and CSS Variables are just icing on the cake.

@media (prefers-color-scheme: dark) {
  :root {  
    --content-bg-color: #222;

In fact, I was able to support “Dark Mode” in bindinator’s code base in a few hours over a couple of evenings by simply leaning hard into CSS Variables (essentially, cleaning the ones I was using up a bit and swapping out instances of hard-coded colors where I had missed them).

I find it particularly hilarious because the reason I did this was after seeing that one of our organization’s Storybooks (for our React-based UI library) had been updated to support Dark Mode (I assume, painfully) and it does not work very well (it looks like the problem is that it renders server-side and only starts respecting “Dark Mode” after the script “hydrates”).

Not only does implementing dark mode with CSS variables work flawlessly (in Chrome, Safari, and Firefox at least), but it involves literally zero lines of Javascript, and the UI transitions beautifully if you toggle dark mode in your UI settings.

I was so pleased with how easily all this worked that I quickly implemented another idea I had in the back of my mind for years, which is a CSS Variable powered theme editor (the link is to the github-hosted site because I haven’t deployed it to the bindinator home page yet).

It’s worth noting that the path of least resistance for implementing Dark Mode with LESS, or SCSS, or React Styletron, or whatever, is still going to be CSS Variables. It also makes theming controls (including web-components, even within their shadow DOMs) a breeze.

snake_case, camelCase, and other Misadventures in Coding Standards

some_programmers_like_snake_case while othersPreferCamelCase (and dont-talk-to-me-about-css-or-html) but life really becomes entertaining when different programmers — sorry “software engineers” — enforce their preferences locally.

Consider a service, implemented in python (where snake_case_is_popular) that passes data to a web application, i.e. javascript (whereCamelCaseIsPopular). You might end up with a data structure like:

{
    my_id: 17,
    my_name: 'seventeen'
}

In modern Javascript, you could write something like:

const { my_id, my_name } = await getSomeData()

And get on with your life.

But it won’t lint, because someone has enforced camelCase in your project lint rules, so now you write:

const data = await getSomeData()
const myId = data.my_id
const myName = data.my_name

And now your code lints.

No problems!

One day, you discover a problem with the data in myName and it stymies you for a bit until you find the assignment statement that renamed the value and you’re reminded that it gets populated from a service with the property my_name and you figure it out.

No problems, right? Chalk it up to experience.

After a lot of this, you kind of wish the problem would just disappear, so you do something like write snakeToCamel which converts an object with snake_case properties into an object with camelCase properties, and now you can write:

const { myId, myName } = snakeToCamel(await getSomeData());

And your code is nearly as clean as if you, say, had turned off the camelCase lint rule in the first place. But hey, snake_case is nauseating.

Later, you modify the helper library that builds async data methods (after all, snakeToCamel called on a conforming object is a no-op, isn’t it?!) so that snakeToCamel always gets called, and now you can finally write:

const { myId, myName } = await getSomeData();

You can now write a new lint rule that flags redundant inline calls to snakeToCamel so that every time a file gets touched, the redundant calls are flagged, or flex your awesomeness and write a code-transform to remove them all in one glorious diff.

Now, your code is beautiful, it’s all in camelCase, and it’s sleek and elegant and oh-so-2017…

Except that there’s no clue in the code for someone hunting down a reference to myName that it might be called my_name deeper in the pipe (e.g. in the Network tab), and now you have a bunch of folks bikeshedding about how things like my_uuid or inner_html should be treated. Because innerHTML does not work well in the other direction, and the python folks have their own lint rules…

Also, deep in your service layer, some dodgy regex is rewriting every property name you ever see, and it’s probably completely clueless about unicode (quick quiz — which unicode characters are OK in javascript variable names?! I looked up the answer and you don’t really want to know.) So, that’s definitely never coming back to bite you.

But hey, at least you don’t have to look at a mixture of snake_case and camelCase in your source code, because the cost of that is… zero?

A Brief Foray into Random Name Generation

I got a bee in my bonnet about name generation this morning, so here’s a simple Javascript module for randomly generating names:

/*
# NameGenerator

Usage:

  const starNameGenerator = new NameGenerator(['Ceti Alpha', 'Scorpio', 'Draconis'...]);
  starNameGenerator.generate(); // -> random name

Works better with a decent sized (hundreds) of thematically similar examples to work from.
*/
/* global module */

function pick(array) {
  return array[Math.floor(Math.random() * array.length)];
}

class NameGenerator {
// data is a map from character-pairs to observed successors,
// consider the examples "how", "now", "brown", "cow"
// the pair "ow" would have the following successors
// [undefined, undefined, "n", undefined] (undefined -> end of word)

  constructor(examples) {
    const data = {'': []};
    examples.
    map(s => s.toLowerCase()).
    forEach(example => {
      let pair = '';
      data[pair].push(example[0]);

      for(let i = 0; i < example.length; i++) {
        pair = pair.substr(-1) + example[i];
        if (! data[pair]) data[pair] = [];
        data[pair].push(example[i + 1]);
      }
    });

    console.log(data);
    this.data = data;
  }

  generate() {
    let s = pick(this.data['']);
    let next = pick(this.data[s]);
    while(next){
      s += next;
      next = pick(this.data[s.substr(-2)]);
    }
    return s;
  }
}

module.exports = NameGenerator;

I wrote a much more convoluted (but simple-minded) star name generator for my galaxy generator several years ago.  The approach I took was to take a collection of star names and break them up into syllables (starting, middle, and ending) and then given a range of syllable lengths, assemble a name out of random pieces.

Today it occurred to me that I’ve never explicitly implemented anything using Markov chains before, so how about I build something that way and see how it compares? If you follow the link, you’ll see examples of star names generated randomly the old way (names with “bad words” in them are rejected).

I took a list of proper star names from Wikipedia and cleaned it up (e.g. the Greek letter prefix of a star name simply indicates its brightness relative to other stars in the same constellation, while a Roman Numeral is simply an indicator of a star being part of a binary or trinary system). This gave me a bit over 600 star names with which to seed the generator, and the results are pretty nice. (Again, no need for bad world filtering.)

The major disadvantage of the new generator is that it can generate really long names pretty easily because of the way it terminates. If I implemented a more sophisticated generator that looked at things like overall length and length of current word in weighing the probabilities it would probably help here, but that might be overthinking it (it’s pretty easy just to reject overly long names).

In general, I think the new generator produces more pronounceable names than my earlier attempt, and some of the new names it generates seem like they should be real names, which is exactly what I’m hoping for.

Algorithm

The algorithm is very simple. The constructor looks at which characters follow a given pair of characters in the input data, so if you start with “how”, “now”, “brown”, “cow” you get the following data for the pair “ow”: [undefined, undefined, ‘n’, undefined]. So, using this data to generate names, 75% of names will end immediately after an ‘ow’ and 25% will have an ‘n’.

In this system, the first character of a name is following the empty string, while the second character of a name is following the first letter. It follows that using [“how”, “now”, “brown”, “cow”] all names will begin with ‘h’, ‘n’, ‘b’, or ‘c’. and most will end in ‘w’ and the rest will end in ‘n’. Not super interesting.

Let’s try slightly more interesting seed data: [‘Frodo’, ‘Peregrin’, ‘Meriadoc’, ‘Bilbo’, ‘Adalgrim’, ‘Bandobras’, ‘Celandine’, ‘Doderic’, ‘Erin’, ‘Griffo’]. This doesn’t seem like it will be enormously fruitful at first glance, but it immediately generates lots of new names that, to me, sound authentic: Adalgrin, Adobris, Grine, Bandine, Froderim.

And here’s a link to a jsfiddle to see it in action (with a bigger set of names from Middle Earth as the seed). One of the really nice things about it is that you don’t need to filter out bad words, because they pretty much don’t get created if they’re not in the source data.

It occurs to me that a lot of the random content generation stuff I’ve done in the past was, in effect, recreating Markov chains naively, and understanding what I’ve done in those terms is powerful and clarifying.

And with that, I leave you with a random Jabberwockish word generator. Don’t bewortle! And have a borpallith day.

Serving b8r

In a former life I worked on optimizing delivery of a fairly large website. I won’t pretend I understood a fraction of the detail, but I had a pretty good idea of the big picture and in a couple of places I drilled down to the bottom-most details.

This isn’t new to anyone who pays attention, but scale makes simple things hard.

The basic tricks to getting a web page to load fast are:

Do as little as possible:

  • Make everything as small as possible.
  • Make everything as simple as possible.
  • Be as asynchronous as possible.

Do it as infrequently, fast, and in parallel as possible:

  • Minimize the round-trip time from client to server.
  • Parallelize everything as much as possible.
  • Split stuff up enough to make it parallel, but not so much as to increase the number of round-trips. (To some extent, SPDY/http2 is solving this for us.)
  • Minimize the number of round trips.
  • Maximize cache utilization.

The grandparent of bindinator was bind-o-matic, which was not designed with all of these things in mind. In particular, it made no real attempt to leverage parallelism or asynchony. When Craig, Josiah, and I wrote bindinator, the “state of the art” was:

  • Figure out what your dependencies are.
  • Compile them into a big blob.
  • Minify the blob.
  • Give the client the blob.

Bind-o-matic’s approach was: “be so small and light that you don’t need to do clever shit to get good performance” (during development) because you can always do that later. We actually compiled our LESS on the client and it didn’t cause performance problems (once we forked less.js and sped it up a bit, and cached compiled CSS in localStorage).

While almost any javascript web application architecture can be served thus, more fine-grained optimization (e.g. trying to become interactive as fast as possible) is a tougher problem, especially when you have little or no ability to do less (e.g. suppose hypothetically that almost every person in your organization is incentivized to make your application bigger, slower, and more complex…)

And you might be committed to using a big, complicated framework that is virtually guaranteed to make anything it touches, bigger, more complex, and less asynchronous.

Anyway, I designed b8r to be as small, simple, and asynchronous as possible but left delivery optimization “for later”. I assumed I could just point webpack (or webpack2) or some more sophisticated tool (such as the stuff I had worked on) at anything I built later.

I did do one thing, though.

I wrote my own require implementation because I started reading the documentation for existing implementations and my eyes glazed over. In particular, none of them seemed as straightforward to use as the one I’d gotten used to in my former life (and with which I had deep familiarity).

I used to be able to write:

const foo = require('foo'); // don't even need to know the path to foo!

I think not needing to know the (relative) path to foo is an anti-feature by the way. The point is that this kind of thing normally needs a build-process to work, which adds a bunch of pain to development. I want the good part — require that just works — and no bad part.

Now, my require is purely client-side, which means that it is a big performance problem. Consider the following code:

const foo = require('path/to/foo.js');
foo(17);

The call to require must be synchronous. But what does require do behind the scenes?

  1. Use an XMLHttpRequest to pull “foo.js” from the server.
  2. Wrap a Function instance around the code inside it.
  3. Pass an object to the function.
  4. Return the “exports” property of the object when the function returns.

It really doesn’t matter how asynchronous your code is if every dependency involves halting execution while round-tripping to your server… recursively in the case of a file that, itself, has dependencies.

This behavior actually throws warnings in Chrome…

Chrome no likee
Chrome no likee

Chrome only complains about this the first time it happens, but it happens a lot.

Now the solution for this is to compile your javascript code on the server and deliver some kind of optimized blob — site.min.js say. This is exactly what tools like webpack do — they actually watch your code tree and trigger recompiles on-the-fly. Webpack offers a dev server that actually sets up a backchannel to the client and refreshes the browser automagically when there are code changes.

Sounds pretty good, right? — but it’s about 1/10 as responsive as using b8r and just forcing refresh. I fucking hate having to compile my code all the time, even if all the compiler does is walk a tree and concatenate a shitload of files wrapped in function calls and assignment statements and then call uglifyjs.

But that’s on a local dev server. What happens when you stick this code on a real server and the round-trip goes from ~0ms to ~100ms? It turns out that on the project I’m working on it changes my web application’s spin-up (with nothing cached) from ~600ms to ~1500ms. (Aside: this is a real web application with a shitload of functionality talking to production servers with no back-end optimization. In my past life, loading in ~1500ms from a real server would have caused spontaneous orgasms. When I told people that performance like this was achievable I was assumed to be a naive fool. No, I’m not bitter!)

So, how to do all this stuff on the client?

  1. Make b8r’s use of require asynchronous. E.g. b8r synchronously loads b8r.dom.js before it finishes loading, so load its dependencies asynchronously before loading b8r itself asynchronously.
  2. Get require to warn whenever it loads a module synchronously. Ick.
  3. OK, get require to return a JSON structure of synchronously loaded modules and load them asynchronously before doing anything else.
  4. Repeat step 3 until no warnings. (This took three iterations.)

Loading time from the stage server went from ~1500ms to ~600ms with no server-side optimization whatsoever. Not bad for a late night hack.

Now wouldn’t it be nice if this were all automatic?

I started writing this post on Friday evening, but my first stab at automating this didn’t work and my brain was too fried to fix it.

In order to function, require tracks all the modules it loads, and it already replaces itself recursively to handle nested requires (to allow for relative paths in require statements) so all I needed to do was track one level of dependencies, and then generate a list of preload “waves” where each wave comprises all modules with no unloaded dependencies. (Circular dependencies will be detected and throw errors.)

Oh, and this eliminates the need for b8r to do anything clever internally. The new solution is general and fixes b8r as well as everything else.

const preload_data =/* data from require.preloadData() */;
require.preload(preload_data).then(() => {
  /* ... */
});

So, now the steps are:

  1. Run require.preloadData() in the console, which spits out JSON data.
  2. Now call require.preload(), passing the data from step 1, which will generate a promise of everything loaded asynchronously.

If dependencies change, everything will still work, but dependencies that are force a synchronous request will generate console warnings.

As a nice bonus, this improves the load time of the b8r demo page by over 80%.

And One More Thing…

If you don’t mind your web app not loading asynchronously the first time, I’ve added another feature to bindinator (and it’s used in he demo page)…

require.autoPreload(10000).then(() => { 9/* ... */ });

This automates all of the above, storing preloadData() in localStorage (the number is the delay in milliseconds before preloadData is written to localStorage; the default is 2000). If code changes cause a file to load synchronously or break the page load, the preloadData is updated or destroyed as appropriate.

So, the first time you load the page it looks like this:

Browser network tab showing b8r files loading one-at-a-time

But from then on it looks like this:

network tab shows b8r files loading in parallel "waves"

You may notice that the last few items are sequential. These are chained promises. At some point I’ll add components to preloadData…