Must all data-tables kind of suck?

b8r's data-table in action
b8r now has a data-table component. It’s implemented in a little over 300loc, supports custom header and fixed headers, a correctly-positioned scrollbar, content cells, sorting, resizing, showing/hiding of columns.

Bindinator, up until a few weeks ago, did not have a data-table component. It has some table examples, but they’ve not seen much use in production because b8r is a rewrite of a rewrite of a library named bindomatic that I wrote for the USPTO that was implemented to do all the data-binding outside of data-tables in a large, complex project and then replaced all the data-tables.

It turns out that almost any complex table ends up being horrifically special-case and bindomatic, by being simpler, made it easier to build complex things bespoke. (I should note that I also wrote the original data-table component that bindomatic replaced.)

Ever since I first wrote a simple file-browser using b8r-native I’ve been wishing to make the list work as well as Finder’s windows. It would also benefit my RAW file manager if that ever turns into a real product.

The result is b8r’s data-table component, which is intended to scratch all my itches concerning data-tables (and tables in general) — e.g. fixed headers, resizable columns, showing/hiding columns, sorting, affording customization, etc. — while not being complex or bloated.

At a little over 300 loc I think I’ve avoided complexity and bloat, at any rate.

As it is, it can probably meet 80% of needs with trivial configuration (which columns do you want to display?), another 15% with complex customization. The remaining 5% — write your own!

Implementation Details

The cutest trick in the implementation is using a precisely scoped css-variable to control a css-grid to conform all the column contents to desired size. In order to change the entire table column layout, exactly one property of one DOM element gets changed. When you resize a column, it rewrites the css variable (if the number of rows is below a configurable threshold, it live updates the whole table, otherwise it only updates the header row until you’re done).

Another cute trick is to display the column borders (and the resizing affordance) using a pseudo-element that’s 100vh tall and clipped to the table component. (A similar trick could be used to hilite columns on mouse-over.)

Handling the actual drag-resizing of columns would be tricky, but I wrote b8r’s track-drag library some time ago to manage dragging once-and-for-all (it also deals with touch interfaces).

Next Steps…

There’s a to-do list in the documentation. None of it’s really a priority, although allowing columns to be reordered should be super easy and implementing virtualization should also be a breeze (I scratched that itch some time back).

The selection column shown in the animation is actually implemented using a custom headerCell and custom contentCell (there’s no selection support built in to data-table). I’ll probably improve the example and provide it as an helper library that provides a “custom column” for data-table and also serves as an example of adding custom functionality to the table.

I’d like to try out the new component with some hard-core use-cases, e.g. the example RAW file manager (which can easily generate lists of over 10,000 items), and the file-browser (I have an idea for a Spotlight-centric file browser), and — especially — my galaxy generator (since I have ideas for a web-based MMOG that will leverage the galaxy generator and use Google Firebase as its backend).

Apple Music… Woe Woe Woe

Meryn Cadell’s “Sweater” — I particularly recommend “Flight Attendant” and “Job Application” but they don’t have video clips that I can find.

We’ve been an Apple Music family pretty much from day one. I used to spend a lot of time and money in stores shopping for albums. Now I have access to pretty much everything there is (including comedy albums) for the price of a CD per month.

I love the fact that my kids can just play any music they like and aren’t forced to filter down their tastes to whatever is in our CD collection, or their friends like, or what’s on commercial radio (not that we listen to commercial radio).

I also love the fact that when I get a new Apple device it just effortlessly has everything in my library on the device the next day.

But…

As I said, I used to spend quite a bit of time buying music. So I have some unusual stuff. Also, I lived in Australia for a long time, so I have a lot of stuff that isn’t available in the US or is available in subtly different form in the US. Similarly, my wife is a huge David Bowie fan and has some hard-to-get Bowie albums, e.g. Japanese and British imports.

We’ve both been ripping CDs for a long time, and in 2012 we ripped everything we hadn’t already ripped as part of packing for a move.

So now we have music that isn’t quite recognized by iTunes. To some extent it gets synced across our devices, probably via a process that went like this:

  1. (Before Apple Music existed) explicitly send music to iPhone
  2. When we get new phone, restore phone from backup on Mac.
  3. (Later) Restore phone from cloud backup.
  4. (Apple Music arrives) Hey, as a service we’ll look through your music library and match tracks to the thing we think it is in iTunes and rather than waste backup space, we’ll simply give you copies of our (superior!?) version. Oh, yeah, it’s DRMed because we need to disable the tracks if you stop paying a subscription fee.

Now this mostly works swimmingly. But sometimes we encounter one of three failure modes:

  • You have something Apple Music doesn’t recognize
  • You have something Apple Music misrecognizes (this happened in the old days when the hacky way iTunes (using the CDDB et al) identified tracks would identify an album incorrectly and misname it and all your tracks, but you could fix it and rename them manually)
  • You have something Apple Music recognizes correctly as something it doesn’t sell in your current region, and disables your ability to play it!

The first failure means that Apple may be able to restore the track from a backup of a device that had it, but it won’t restore it otherwise. So you have to find a machine with the original (non-DRMed file and fix it).

The second failure means that Apple’s Music (iTunes on a Mac) application will start playing random crap. If you’re lucky you can find the correct thing in Apple Music and just play that instead, but now you’re stuck with Apple’s DRM and may even end up losing track of or deleting your (non-DRMed) version.

The third mode is particularly pernicious. I have a fantastic album by Canadian performance artist Meryn Cadell (Angel Food for Thought) that I bought after hearing a couple of the tracks played on Phillip Adams’ “Late Night Live” radio program many years ago. I freaking love that album. For years, Apple would sync the album from device to device because it had no freaking clue what it was…

But sometime recently, Apple added Meryn Cadell’s stuff to Apple Music. As far as I can tell, it makes a second album (that I didn’t know existed) to the Apple Music US region but not Angel Food for Thought. So it knows that Angel Food for Thought exists but it won’t let me play it.

Now, I happen to know where my backups are. I fired up iTunes on my old Mac Pro and there’s Angel Food for Thought. It plays just fine. Then I turned on “sync to cloud” and all the tracks get disabled. It’s magical, but not in a good way.

This is ongoing… I will report further if anything changes.

Update

After escalation, I’ve been told iTunes is working as intended. So, basically, it will play music it can’t identify or that is DRM-free and already installed, but what it won’t do is download and play music from iTunes that (it thinks) matches music that (it thinks) you have if it doesn’t have the rights to that music in your jurisdiction.

So, I have the album “Angel Food for Thought” which iTunes used not to know about, so it just worked. But, now iTunes knows that it exists BUT it doesn’t have US distribution rights, so it won’t propagate copies of “Angel Food for Thought” to my new devices (but it won’t stop me from manually installing them). Super annoying, but not actively harmful.

It does seem to mean that there’s a market for something that lets you stick all your own music in iCloud and play stream it for you.

Undo

If there’s one feature almost every non-trivial application should have which is overlooked, left as an exercise, or put aside as “too hard” by web frameworks, it’s undo.

I do not have much (any) love for Redux, but whenever I think about it, I wonder “why isn’t undo support built in out of the box”?

To its credit, the Redux folks do actually address Undo in their documentation. “It’s a breeze” the document says at the top, and it’s a mere 14 pages later that it concludes “This is it!”.

I’ve pretty much never seen a React/Redux application that implements undo. Since it’s such a breeze, I wonder why. My guesses:

  1. The words “service” and “back-end” do not appear in the document.
  2. It doesn’t look like a breeze to me. A 14 page document describing implementation of a core concept in a toy application sounds more like a tornado.
  3. Finally, every non-trivial Redux application I’ve ever seen is an incomprehensible mess. (Once multiple Reduxes get composed together, you’re left with multidimensional spaghetti code of the ‘come-from’ variety.) Adding undo into this mix as an afterthought strikes me as a Kafkaesque nightmare.

It’s since been pointed out to me that there are libraries, such as redux-undo, around that wrap the implementation described in the Redux documentation (or variants thereof) in libraries. The example application is a counter, I don’t know of nor have I seen any non-trivial examples.

“Come-from”? Come again?

Programmers by now fall into two groups. Folks who have either read or know by reputation the article Goto considered harmful — possibly the most famous rant in the history of computer science — or folks who have grown up in a world where the entire concept of goto has been hushed up as a dark family secret.

come-from means code that is triggered by a foreign entity, you don’t know when. You can think of it as an uncharitable description of the Observer Pattern. It’s the dual of goto. There are two common examples that are employed widely in user-interfaces — event-handling and pub-sub.

Event-handling is necessitated by the way human-computer interaction works — you’re talking to the analog world, it’s not going to be pretty. The user will do something, it’s generally unrelated to what, if anything, your code was doing. You need to deal with it.

Pub-sub can be self-inflicted (i.e. used when there’s no reason for it), but is often necessitated by the realities of asynchronous and unreliable communication between different entities. Again, that’s not going to be pretty. You make a request, you make other requests. Time passes. Things change. Suddenly you get a response…

Redux is a state management system, so it is natural that it implements the Observer Pattern, but it adds the additional garnish that the mapping between action names and what parts of your application’s state they modify can be — and often is — completely arbitrary.

implementing undo in b8r should be a breeze!?

If you stand back a bit, b8r‘s registry is a similar to redux except instead of your state being a quasi-immutable object which is swapped out by “actions” (with each action being a name linked to a function that creates a new object from the old object) with b8r your state is the registry (or some subset thereof) and your actions are paths.

In my opinion, b8r is wins here because you generally don’t need to implement your actions, when you do they still act on the registry in the expected way, and instead of having an arbitrary map of action names to underlying functions the path tells you exactly which bit of state you’re messing with.

For purposes of implementing Undo, however, Redux seems to have an advantage out of the gate in that it is (partially) cloning state every time something changes. So all you need to do is maintain a list of previous states, and you’re golden, right?

What should it require to implement Undo (in b8r)?

To my mind, it should be something like this:

import {UndoManager} from 'perfectUndoLib'
new UndoManager('path.to.foo', {
    path: 'foo-history',
    onUndo: ...
    unRedo: ...
    onUndoRedo: ...
    onHandleChange: ...
});
b8r.get('foo-history.undoQueue.length') // number of undoables
b8r.get('foo-history.redoQueue.length') // number of redoables
b8r.call('foo-history.undo') // undoes the last thing
b8r.call('foo-history.redo') // redoes the last thing
b8r.call('foo-history.reset') // clears queues (e.g. after saving a file)

The first parameter could allow a list of paths to include in the undo manager’s purview, while also allowing multiple independent undo chains using multiple undo managers (e.g. some advanced graphics applications separate out changes in viewport from changes in the underlying scene).

The path option would probably default to lastThing-history and so foo-path would be superfluous.

Ideally, the callback options would allow async functions to do things like perform a service call to keep the store consistent with the interface, curate the undo and redo queues (e.g. TextMate used to do single keystroke undo which was infuriating when going back through a large edit history, addressing this appears to have paralyzed development and sidelined what had been the darling text editor of the hipter screencast brogrammer crowd), handle errors, and so on.

Implementation Details

At this point in our story, I hadn’t written a single line of code.

I have, however, been thinking about this for several years, which is more important. I’ve implemented several different non-trivial applications (RiddleMeThis, two word-processors, and a non-destructive image editor) with undo support. I think it’s worthwhile when saying something like “I wrote this code in 2h, am I not a true 100x coder!?” to acknowledge, at least to oneself, that it crystallized thoughts collected over a longer period (in this case several years), or whatever.

E.g. I wrote the core of b8r in less than two weeks, but it was a rewrite from scratch of a library I had written over a period of several months while at Facebook which was itself a rewrite of a rewrite of code I had written at USPTO.

b8r — or rather its registry — already has a robust observe method that informs you when anything happens to a particular path (or its subpaths), so something like:

import b8r from '../source/b8r.js'

Of course we’re going to need to deep clone state because — unlike with Redux — we can’t just grab the current state object.

On the plus side, we can just keep a copy of the subtree that changed. Keeping a clone of the entire state in memory every time something changes can get hairy.

Aside: two of the three serious implementations of undo I’ve been responsible for have been (a) a long form word-processor that handled both annotations and multi-user-edit-history and (b) a non-destructive image-editor). And, in fact two things I’ve been thinking about putting together as side projects are a long-form book-publishing system (which aims to publish on web and ePub vs. print/pdf/etc.) and a photography-workflow application, neither of which are going to play nicely with naive “every undo step is the state of your app” approaches.

Here is the code actually committed to the repository in the course of writing this post. Or you can see it in the inline documentation system.

After a quick visit to StackOverflow and Google, it looks at first approximation like the most robust option is “the dumb thing”:

const deepClone = (thing) => JSON.parse(JSON.stringify(thing));

OK so now for the meat!

export class UndoManager {
  constructor (watchPaths, options={}) {
    watchPaths = [watchPaths].flat()
    const {
      historyPath,
      onUndo,
      onRedo,
      onUndoRedo,
      onHandleChange,
    } = {
      historyPath: '_' + watchPaths[0].split('.').pop() + '_history',
      ...options
    }

We expose the buffers and methods in the registry, which makes binding controls to undo and redo super simple. The observer is how we get notified state we care about changes.

    b8r.register(historyPath, {
      undoBuffer: [],
      redoBuffer: [],
      undo: this.undo,
      redo: this.redo,
      observer: this.handleChange.bind(this)
    });

    Object.assign(this, {
      watchPaths,
      historyPath,
      onUndo: onUndo || onUndoRedo,
      onRedo: onRedo || onUndoRedo,
      onHandleChange
    })

    this.reset()
    this.observe()
  }

  reset () {
    this.undoBuffer.splice(0)

    this.undoBuffer.push(this.watchPaths.reduce(
      (state, watchPath) => {
        state[watchPath] = deepClone(b8r.get(watchPath))
        return state;
      },
      {}
    ));

    b8r.touch(this.historyPath)
  }

b8r.observe informs you of state-changes after they happen. (Because, for perf reasons, changes to the registry are queued and handled asynchronously, so if one or more registry entries are changed rapidly, each relevant observer will only be fired once.) So if we wait to collect state until the first user action, we’re not going to be able to restore our original state.

  observe () {
    this.watchPaths.forEach(path => b8r.observe(path, `${this.historyPath}.observer`))
  }

  unobserve () {
    this.watchPaths.forEach(path => b8r.unobserve(path, `${this.historyPath}.observer`))
  }

  get undoBuffer () {
    return b8r.get(`${this.historyPath}.undoBuffer`)
  }

  get redoBuffer () {
    return b8r.get(`${this.historyPath}.redoBuffer`)
  }

restore doesn’t care where state is coming from, its job is just to restore it without triggering handleChange and notify anything bound to the the instance that its state has changed.

  async restore () {
    this.unobserve()
    // const state = b8r.last(this.undoBuffer)
    // b8r.forEachKey(state, (value, key) => b8r.set(key, value))
    this.undoBuffer.forEach(state => {
      b8r.forEachKey(state, (value, key) => b8r.replace(key, deepClone(value)))
    })
    this.observe()
    b8r.touch(this.historyPath)
  }

It turns out there’s a couple of subtle bugs in restore both of which become obvious if the UndoManager (now called HistoryManager) is handling changes on more than one path. So restore() now replays all changes in the undo queue.

I’ve commended out the bad lines of code — the loop below fixes both bugs. There are obvious optimizations here which I have not yet implemented.

When the state of the world changes owing to user action, we need to put the new state on the undoBuffer and clear the redoBuffer (which is now obsolete).

  async handleChange (pathChanged) {
    if (this.onHandleChange && ! await this.onHandleChange()) return
    this.undoBuffer.push({
      [pathChanged]: deepClone(b8r.get(pathChanged))
    })
    this.redoBuffer.splice(0)
    b8r.touch(this.historyPath);
  }

  undo = async () => {
    if (this.undoBuffer.length === 1) return
    if (this.onUndo && ! await this.onUndo()) return false
    this.redoBuffer.push(this.undoBuffer.pop())
    await this.restore()
    return true
  }

  redo = async () => {
    if (this.onUndo && ! await this.onRedo()) return false
    this.undoBuffer.push(this.redoBuffer.pop())
    await this.restore()
    return true
  }

  destroy() {
    this.unobserve()
    b8r.remove(this.historyPath)
  }
}

And undo/redo are now super simple to implement. E.g. this example is now in the documentation of undo.js:

<button 
  data-event="click:_simple-undo-example_history.undo"
  data-bind="enabled_if=_simple-undo-example_history.undoBuffer.length"
>undo</button>
<button 
  data-event="click:_simple-undo-example_history.redo"
  data-bind="enabled_if=_simple-undo-example_history.redoBuffer.length"
>redo</button><br>
<textarea data-bind="value=simple-undo-example.text"></textarea>
<script>
  const {UndoManager} = await import('../lib/undo.js');
  b8r.register('simple-undo-example', {
    text: 'hello undo'
  })
  const undoManager = new UndoManager('simple-undo-example')
  set('destroy', () => {
    b8r.remove('simple-undo-example');
    undoManager.destroy()
  })
</script>

And that’s it!

I’ve walked through the general-purpose implementation of undo in b8r and it looks to me like you can use this to implement undo in a typical b8r application in a single line of code (plus whatever extra lines it takes to put in and add the widgets to your view and bind them). I’d say that qualifies as a breeze — at least until you get to dealing with services.

b8r goes pure(ish) Javascript

b8r now allows components to be implemented entirely in Javascript, with no eval

In an effort to make b8r work in the universe of “everything must be javascript” web development tools, I’ve finally added provision for pure javascript components in b8r.

Now, there’s four really good reasons for doing this.

First of all, the way b8r’s HTML-based components work intrinsically involves using eval (it actually uses the AsyncFunction constructor, but same difference). Whether this represents a real security threat or not depends a lot on how those components are in fact loaded, but it’s definitely an argument.

Second — and this has proven a small but persistent pain point with b8r development — it makes linters happy. To begin with, the <script> tag inside a component was actually the body of an async function with a ton of parameters. Using these parameters without annoying the lint gods involved /* global … */ to be added at the top of the scripts, while the fact that — for convenience — the code was async meant that linters would scream about use of import() and async/await.

Third — with HTML components, defining a single global controller for a family of components involved some jumping through hoops as did, for example, using computed bindings or lists where the method was registered in the component body (this works fine, but generates error messages if you don’t programmatically add the bindings after the necessary methods are registered). Now you can just do the necessary work outside the component definition (but in the same file!)

Fourth — while b8r has moved over to the world of ES6 Modules, it is distributed in cjs form for those still living with require, webpack, etc. The problem is that HTML components don’t play nicely with javascript bundlers, whereas javascript components are just javascript so if you really want to force a giant slab of javascript down your users’ throats you can. Yay.

So, these issues are fixed if you use pure javascript components.

You are still writing CSS as CSS and HTML as HTML, but now it’s likely going to be inside back-ticks.

In addition to the four big arguments for building components this way, there are some further implications (mostly good).

First of all, if and when you need to use import() inside a component, you can use relative paths without wondering “relative to what” (it used to be the location of b8r.js, because that’s where that code ends up executing).

Next, there was no convenient way to create multiple components in one source file. This is a mixed blessing — making it easy to define lots of small components in source files with no relationship between file name and component name presents the real risk of name-clashes. It may become necessary to introduce lint rules that prevent you from creating a component whose name does not match its source file in some way.

Finally, a really nice side-effect of this is that the boilerplate for a component is a lot more self-explanatory than it used to be. Yay!

Dark Mode with CSS Variables

All the cool kids are doing it, what’s the easiest way to support Dark Mode?

As little as four or so years ago, it was widely understood among front-end developers that CSS was a horrible problem, managing it was a nightmare, and Something Ought to be Done About It.

Back then, there were two popular “solutions”, LESS and SCSS, that both allowed you to transpile your CSS from a superset of CSS that allowed things like named constants and nesting rules. There was also the drum beat of the “all web development should be Javascript development” which advocated replacing CSS with Javascript (that created CSS). Incidentally, these were the same people who thought “all Javascript development should be some weird superset of Javascript that needs to be transpiled into Javascript”.

Anyway, the big problem LESS, SCSS, and the CSS-should-be-Javascript folks tried to solve was managing consistency across large sets of rules (no-one of course questioning the idea of having such large sets of rules). The downside was that this enabled even bigger sets of rules. (The best reason for using LESS or SCSS was the implementation of named constants. With these part of CSS, it seemed clear to me that their reason for existence was largely gone.)

If you reduce the cost of something, folks will use more of it. LESS, SCSS, and styling with Javascript reduced the apparent cost of adding CSS rules to web apps.

E.g. on a project I was working on at the time, about 10kB of LESS became 280kB of CSS — and this was on a team that was (a) small, and (b) ruthlessly minimized CSS whenever it could.

At this point, I can safely say “a pox on all your houses”. With bindinator’s approach of using a solid set of global styles and component-specific styles that are easy to scope to specific components (without the use of a “shadow DOM”) we never ran into any of the thorny CSS issues that plague other projects I’ve worked on, and CSS Variables are just icing on the cake.

@media (prefers-color-scheme: dark) {
  :root {  
    --content-bg-color: #222;

In fact, I was able to support “Dark Mode” in bindinator’s code base in a few hours over a couple of evenings by simply leaning hard into CSS Variables (essentially, cleaning the ones I was using up a bit and swapping out instances of hard-coded colors where I had missed them).

I find it particularly hilarious because the reason I did this was after seeing that one of our organization’s Storybooks (for our React-based UI library) had been updated to support Dark Mode (I assume, painfully) and it does not work very well (it looks like the problem is that it renders server-side and only starts respecting “Dark Mode” after the script “hydrates”).

Not only does implementing dark mode with CSS variables work flawlessly (in Chrome, Safari, and Firefox at least), but it involves literally zero lines of Javascript, and the UI transitions beautifully if you toggle dark mode in your UI settings.

I was so pleased with how easily all this worked that I quickly implemented another idea I had in the back of my mind for years, which is a CSS Variable powered theme editor (the link is to the github-hosted site because I haven’t deployed it to the bindinator home page yet).

It’s worth noting that the path of least resistance for implementing Dark Mode with LESS, or SCSS, or React Styletron, or whatever, is still going to be CSS Variables. It also makes theming controls (including web-components, even within their shadow DOMs) a breeze.