Apple Music… Woe Woe Woe

Meryn Cadell’s “Sweater” — I particularly recommend “Flight Attendant” and “Job Application” but they don’t have video clips that I can find.

We’ve been an Apple Music family pretty much from day one. I used to spend a lot of time and money in stores shopping for albums. Now I have access to pretty much everything there is (including comedy albums) for the price of a CD per month.

I love the fact that my kids can just play any music they like and aren’t forced to filter down their tastes to whatever is in our CD collection, or their friends like, or what’s on commercial radio (not that we listen to commercial radio).

I also love the fact that when I get a new Apple device it just effortlessly has everything in my library on the device the next day.

But…

As I said, I used to spend quite a bit of time buying music. So I have some unusual stuff. Also, I lived in Australia for a long time, so I have a lot of stuff that isn’t available in the US or is available in subtly different form in the US. Similarly, my wife is a huge David Bowie fan and has some hard-to-get Bowie albums, e.g. Japanese and British imports.

We’ve both been ripping CDs for a long time, and in 2012 we ripped everything we hadn’t already ripped as part of packing for a move.

So now we have music that isn’t quite recognized by iTunes. To some extent it gets synced across our devices, probably via a process that went like this:

  1. (Before Apple Music existed) explicitly send music to iPhone
  2. When we get new phone, restore phone from backup on Mac.
  3. (Later) Restore phone from cloud backup.
  4. (Apple Music arrives) Hey, as a service we’ll look through your music library and match tracks to the thing we think it is in iTunes and rather than waste backup space, we’ll simply give you copies of our (superior!?) version. Oh, yeah, it’s DRMed because we need to disable the tracks if you stop paying a subscription fee.

Now this mostly works swimmingly. But sometimes we encounter one of three failure modes:

  • You have something Apple Music doesn’t recognize
  • You have something Apple Music misrecognizes (this happened in the old days when the hacky way iTunes (using the CDDB et al) identified tracks would identify an album incorrectly and misname it and all your tracks, but you could fix it and rename them manually)
  • You have something Apple Music recognizes correctly as something it doesn’t sell in your current region, and disables your ability to play it!

The first failure means that Apple may be able to restore the track from a backup of a device that had it, but it won’t restore it otherwise. So you have to find a machine with the original (non-DRMed file and fix it).

The second failure means that Apple’s Music (iTunes on a Mac) application will start playing random crap. If you’re lucky you can find the correct thing in Apple Music and just play that instead, but now you’re stuck with Apple’s DRM and may even end up losing track of or deleting your (non-DRMed) version.

The third mode is particularly pernicious. I have a fantastic album by Canadian performance artist Meryn Cadell (Angel Food for Thought) that I bought after hearing a couple of the tracks played on Phillip Adams’ “Late Night Live” radio program many years ago. I freaking love that album. For years, Apple would sync the album from device to device because it had no freaking clue what it was…

But sometime recently, Apple added Meryn Cadell’s stuff to Apple Music. As far as I can tell, it makes a second album (that I didn’t know existed) to the Apple Music US region but not Angel Food for Thought. So it knows that Angel Food for Thought exists but it won’t let me play it.

Now, I happen to know where my backups are. I fired up iTunes on my old Mac Pro and there’s Angel Food for Thought. It plays just fine. Then I turned on “sync to cloud” and all the tracks get disabled. It’s magical, but not in a good way.

This is ongoing… I will report further if anything changes.

Update

After escalation, I’ve been told iTunes is working as intended. So, basically, it will play music it can’t identify or that is DRM-free and already installed, but what it won’t do is download and play music from iTunes that (it thinks) matches music that (it thinks) you have if it doesn’t have the rights to that music in your jurisdiction.

So, I have the album “Angel Food for Thought” which iTunes used not to know about, so it just worked. But, now iTunes knows that it exists BUT it doesn’t have US distribution rights, so it won’t propagate copies of “Angel Food for Thought” to my new devices (but it won’t stop me from manually installing them). Super annoying, but not actively harmful.

It does seem to mean that there’s a market for something that lets you stick all your own music in iCloud and play stream it for you.

Photo-backup, Hubic, and Other Stories

So, a year ago I started backing up to HubiC after my previous backup service decided to stop servicing retail customers. At the time, HubiC seemed on paper like a great option — potentially offering the simplicity and utility of DropBox with effectively unlimited storage capacity.

In practice, HubiC is useless. I’ve had a 2012 Mac Pro constantly connected to HubiC via a fast cable connection for 12 months and managed to back up only about 1/4 of the files I’ve pointed at it. The damn thing breaks down constantly. Every time I log in it wants me to change passwords. They just billed me for renewal, but I’m two days late and I can’t cancel my account without paying for another year. Good luck with that, guys.

Now, I haven’t been sitting on my hands while I watch HubiC fail to deliver on any of its promises. Most of my stuff continues to be backed up locally via Time Machine. The stuff I work on is stored in the cloud (iCloud, GitHub, Google Drive, and/or DropBox).

My big problem is photographs and video.

The disaster that is my Flickr account
Flickr’s auto-uploader turned my Flickr account into an unfixable mess

Now, when Flickr raised its “free” tier to offer 1TB of cloud storage for JPEGs, I jumped on that. It may not be RAW storage, but it’s better than nothing, and 1TB is enough space for a huge number of JPEGs. The big problem, Flickr’s (since discontinued) auto-uploader was so stupidly designed that it successfully rendered my Flickr account borderline useless (it created an album for every folder it found an image in, and it uploaded every image it found, including things like UI images inside applications and development trees, so I have “albums” comprising sprites from sample game development projects and logos for PHP templates) and Flickr’s account management tools look and work like something an intern abandoned in 2005, so just deleting stuff is an exercise in frustration. It looks to me like Flickr’s abandonware API isn’t really up to the task of even supporting a third party application to untangle the mess.

And of course, since Yahoo changed hands and various security scandals unfolded just logging into Yahoo accounts is a pain, and you need to navigate ads to even get into your account. Yahoo is the GeoCities of 2018.

Google photos in "action"
When I scroll to some arbitrary point on my timeline, this is what I see for some random (long) amount of time…

Recently, Google raised the “free” tier of photos.google.com to unlimited storage of photos where RAW files are JPEGs are processed into high-but-not-full-quality JPEGs on-the-fly. I’ve tried it and it’s pretty damn good. The uploader is smart enough to skip files that are clearly not important photos (e.g. too small, wrong format) and ignore obvious duplicates. The problems are (i) that the uploader application periodically just hangs and needs to be manually killed and restarted (ii) the web app seems to be weirdly slow and unreliable (I can log on with two machines side-by-side and they’ll see different subsets of my photos), (iii) no Apple TV support, and (iv) online photo editor seems to need one or two extra clicks to accomplish anything (but it’s a lot better than nothing). I’m pretty confident that my stuff is there, just not in my ability to see a given photo from a given machine on a given day. It’s certainly the most complete, easy-to-navigate, and shareable archive I’ve ever managed to create of my photographs. And if I can find a photo there, I can locate the original RAW image pretty easily.

Now, the absolute best system for dealing with my photographs thus far is iCloud. If I could simply rent 10TB from iCloud for a reasonable price (let’s say, $25/month) and get my Mac to automatically sync multiple volumes to iCloud, my problem would be solved. Obviously, I’m a happy Apple customer. If I were a more-than-casual Windows or Linux user then this would not be a useful option to me, and I’m not sure what I’d do, because I’m pretty sure there’s no equivalently seamless option for people who don’t want to pay the “Apple Tax”. Google Drive isn’t even a tolerable substitute for DropBox (although I think it has Sharepoint beaten).

Here’s where iCloud beats all other options:

  • I don’t need to think about it or do anything. (Well, on a desktop device, I need to NOT avoid storing my data in iCloud) If I take a photo, then it ends up in the cloud pretty quickly (basically, when the device gets recharged while on a LAN, if not sooner).
  • By default, full-resolution images are not propagated to all my devices (as would be the case for DropBox, or Hubic if it actually worked). Instead, as with everything in iCloud it’s available on-demand. (Indeed, it’s a bit reminiscent of the way iTunes deals with movies… superficially less convenient than pure streaming, but a lot more flexible and useful in practice).
  • If I ingest a RAW photo from a camera onto a device, then it’s in the cloud and available from any device on-demand (but it’s not wasting space on all my devices).
  • If I want to work on a photo, I can use the best native tools that are available on the device I’m using — seamlessly (although I’m inclined to actively avoid Adobe applications because Adobe’s workflow involves use of Adobe’s barely functional Cloud ecosystem).

The big problem — of course there has to be one — is that Apple’s highest storage tier is 2TB. I’m currently on 200GB which is plenty for the stuff I need that isn’t photos and videos, but hopelessly inadequate for photos and videos. 2TB (the next tier up, and it’s competitively priced) would be sufficient for my photos and videos if I were to curate them, but I don’t want to curate shit. I want to dump it in the cloud and not think about it.

Missing in Action

All of this adds up to a bunch of pretty disappointing non-solutions. Even though Apple provides a file sync system that works pretty well for personal photographs, it wouldn’t work for say a small photography business. (I guess you could use some kind of “family plan” but I’m pretty sure that would run you into weirdness pretty fast.) And it’s not like we’re talking advanced workflow support here — I just want my photos backed up and available.

Where is a tool that automatically detects blurred, underexposed, or overexposed photos and flags them as less worthy of backup? (Google’s photos app does a pretty good job of automatically correcting exposure, I wonder if it’s smart enough to task the uploader with going back to the RAW and reprocessing and re-uploading the photo?)

Where is the tool that remembers which photos have been opened or zoomed in and flags them as more interesting or worthy of backup?

Where is the tool that correlates the GPS location data of your iPhone photos and tentatively applies them to your corresponding camera photos?

Aperture used to collect photos from bursts into a single set and represent them with what it guessed was the best one. Where did this idea disappear to?

There’s a ton of low-hanging fruit here. Someone, please do something. I’m busy.

b8r 1.0

I decided to make the current revision of b8r “1.0” (it’s still marked “prerelease”) based on the fact it’s proven solid and usable during a year of constant use and improvement. It has been at least as robust and easy to work with as the jQuery-dependent library we developed at USPTO. I’ve just updated bindinator.com and my galaxy demo.

Recently, I made the first deliberately breaking changes and the difficulty of adapting various codebases that use b8r was pretty minor. So, I’m pretty confident that b8r is in good shape.

b8r's "fiddle" component in action
b8r’s “fiddle” component in action — b8r’s fiddle.component.js currently weighs in at 272 loc including markup, css, and comments.

I also improved the appearance of the inline fiddle and test components, and added prism.js code-rendering to all the various inline code examples to make the documentation pages look snazzier. A nice change to the test library makes sure that async test results are consistently ordered, and added a visible “pending” state so you can see tests that somehow failed to complete.

b8r has some pretty nice stuff. (Although much of this nice stuff needs to be documented.) You can download b8r, put nwjs in your Applications directory and/or npm install electron and double-click a .command file to see the b8r documentation inside a desktop app. Or you can install nodejs and double-click a .command file and serve it locally via https (I also provide a .command file will generate local ssl keys). (The .command stuff is currently Mac only, for which I apologize. I imagine it would be very easy to do it for Linux and Windows, but I haven’t tried.)

"electron-file-browser" component (running inside nwjs)
“electron-file-browser” component (running inside nwjs)

There’s a cute feature if you load the b8r documentation in nodejs or electron and command-click on a component — the component is loaded in a new window. I’m planning on leveraging this functionality to let the documentation app function as an IDE.

I’m currently working on convenience methods for multi-window desktop applications (it would be super cool if you could transparently bind across windows and browser tabs). I’ve also written a new version of foldermark that uses a very simple PHP back end (nodejs servers are still a much bigger pain to deal with than PHP) combined with b8r on the client-side.

The biggest shortcoming of b8r remains the fact that my team is the only group really using it. Because we’re developing a desktop app using Electron, we aren’t constantly testing on Edge, Firefox, Safari, etc.. I know for a fact that it has problems in IE and Edge, and that some of the example components aren’t touch-friendly, and we’re definitely doing more stuff for Electron than for nwjs (nwjs is much simpler to work with, but it’s becoming increasingly irrelevant, I fear). But if you’re working in reasonably recent releases of Chrome or Chromium, b8r should be very solid.

So, that’s the way it is: b8r 1.0.

Replacing Crashplan

I’ve been pretty happy with Crashplan for five years or so, although lately not so much. Obviously, one reason to be unhappy with Crashplan is that they’re no longer supporting the family plan I was using to back up all our Macs. Worse, their quality of service seems to have (understandably) slipped as my plan approaches its termination.

Exhibit 1: my wife bought a new laptop. When she tried to restore files from her old laptop, they were MIA.

So, here we are at a nasty juncture where our supposedly bulletproof fire-and-forget backup system is (a) not working terribly well and (b) shortly going to not work at all.

Possible Crashplan Replacements

The obvious replacement is Carbonite. The problem with Carbonite is that it’s going either (a) be a lot more expensive than Crashplan was (because of our family plan — which is the equivalent of, effectively, n $100/year plans where n is the number of computers you back up; the small business plan works out as being stupidly expensive if you have more than 250GB of data) or (b) require me to do a bunch of work (i.e. set up one computer as the family server, have all the other devices back up to it, and then back it up via Carbonite. So Carbonite will either cost $300 (say) per year or $100 per year but require me to be my own network engineer. Oh, and forget mobile backups.

If I want to do a whole bunch of work I might as well just use Amazon S3. For 2TB of data that’s a mere $46/month. Ouch. I could probably use their cheaper long term storage, but now I’m basically starting my own Crashplan / Dropbox implementation and that sounds kind of like hard work. Forget that.

So, on to the “consumer” options (I’ve tried to pick 2TB plans as this is the absolute minimum I can live with):

  • Dropbox — $99 p.a. 1TB (there is no 2TB plan)
  • Box — $540 p.a. for unlimited (3 computers)
  • Hubic — $60 p.a. 10TB
  • Amazon — $120 p.a. 2TB
  • Google — $240 p.a. 2TB
  • Microsoft — $99 p.a. 5 users, 1TB/user
  • Apple — $120 p.a. 2TB (no Android client)

(Edit: TablePress was a disaster.)

Note that all these services support web browser access, so you can get at your files from any device with a web browser, but I’m talking about file-system integration where you can just save your file in the usual place and it’s seamlessly backed up to the cloud. I should note that Dropbox and Hubic even provide Linux clients. Everyone supports Mac, Windows, and iOS.

At first glance, Hubic looks like the standout value-for-money option. (Hubic is essentially like Dropbox, except it’s run by a gigantic hosting company). The problem is that I’ve found Hubic to have a poor user experience, especially with regard to performance. (My own experience is quite limited — and I’m kind of shocked that my blog post on Dropbox vs. Box vs. Hubic ranks quite high on Google searches for “Hubic Review”.) This may be a result of server location, or simply under-provisioning (10TB is so close to “unlimited” that I imagine it attracts a lot of abusive users).

(Edit: I should mention that Hubic offers both Dropbox-like services and backup services. It also has what looks like a pretty robust API. Also there are advantages to storing your data in France. On the other hand — no 2FA.)

At second glance, Microsoft OneDrive seems like a great deal at 5TB for $100/year. The problem is that it’s 5x 1TB per user, which is effectively 1TB. Still, a great deal if you’re happy with 1TB and want Microsoft Office for your family.

The shock comes at third glance — Apple’s product is competitively priced (cheaper than Dropbox!). It doesn’t support Android (surprise!) but aside from that it’s a great deal, you can share it with your family (with each person having segregated storage), it requires no real configuration and — this is the kicker — it’s smart about mirroring stuff to your devices. Want a 1TB photo library in the cloud? Great. Want it on every one of your family computers? Not so much.

Conclusion

Hubic is the clear winner in terms of price per unit storage. Apple is the clear winner in terms of functionality and comes second in price. When I take into account the fact that I pay Apple $3/month for a lower tier of storage just for convenient mobile backups it’s an even better deal (hooray for Opportunity Cost). After some consideration I’m thinking of doing both. Hubic for volume backup and iCloud for convenience.

I’ll let you know how it works out.

Followup…

I nearly gave up on Hubic. After paying for the 10TB plan I received no response and my account didn’t change… Well, until the next day. Sacré bleu! I’m still a bit concerned that the app hasn’t been updated since 2015, but it seems to work.

Anyway, I’m going to start backing up to Hubic and we’ll see how things go.

 

Further research has shown that there are two more reasonably-priced alternatives to Carbonite, notably iDrive and Backblaze. Backblaze seems very compelling for a single computer ($5/month unlimited; $4 with a two year plan), and not bad for a small number of computers (it works out as roughly the same as Crashplan for 3 computers). iDrive is offering some interesting discounts (and lets you handle any number of computers with one account) but I find the website poor to the point of suspecting the competence of the company.

System Update Weirdness

I’m simply recording this for the benefit of others who may have similar weird experiences updating to 10.13.6.

I recently updated macOS via the App Store from 10.13.5 to 10.13.6 and it was unusual in that it got about two-thirds of the way through the install fast (while saying it had 16 minutes to go) and then got stuck.

When I noticed the problem a couple of hours later, I

  • forced power off and restarted. It then quickly went up to about the same point it previously hung, rebooted, got to about 90% of the way done, rebooted again, then got to about the same spot and the screen went black and stayed black.
  • forced power off and restarted (I might have had to do this more than once at this point) and then it got to the faux sign-in screen you now get when Macs aren’t fully booted (e.g. after the battery runs flat during stand-by).
  • I logged in and got to the proper login screen, and was was relieved to find that everything from there proceeded normally — the OS asked me if I wanted to send feedback to Apple, etc., which seems to happen after some OS updates but not others, but also wasn’t the “new user” experience where you’re asked to log into iCloud again.

As a side-note, this is an experimental first post via Ulysses.