Photo-backup, Hubic, and Other Stories

So, a year ago I started backing up to HubiC after my previous backup service decided to stop servicing retail customers. At the time, HubiC seemed on paper like a great option — potentially offering the simplicity and utility of DropBox with effectively unlimited storage capacity.

In practice, HubiC is useless. I’ve had a 2012 Mac Pro constantly connected to HubiC via a fast cable connection for 12 months and managed to back up only about 1/4 of the files I’ve pointed at it. The damn thing breaks down constantly. Every time I log in it wants me to change passwords. They just billed me for renewal, but I’m two days late and I can’t cancel my account without paying for another year. Good luck with that, guys.

Now, I haven’t been sitting on my hands while I watch HubiC fail to deliver on any of its promises. Most of my stuff continues to be backed up locally via Time Machine. The stuff I work on is stored in the cloud (iCloud, GitHub, Google Drive, and/or DropBox).

My big problem is photographs and video.

The disaster that is my Flickr account
Flickr’s auto-uploader turned my Flickr account into an unfixable mess

Now, when Flickr raised its “free” tier to offer 1TB of cloud storage for JPEGs, I jumped on that. It may not be RAW storage, but it’s better than nothing, and 1TB is enough space for a huge number of JPEGs. The big problem, Flickr’s (since discontinued) auto-uploader was so stupidly designed that it successfully rendered my Flickr account borderline useless (it created an album for every folder it found an image in, and it uploaded every image it found, including things like UI images inside applications and development trees, so I have “albums” comprising sprites from sample game development projects and logos for PHP templates) and Flickr’s account management tools look and work like something an intern abandoned in 2005, so just deleting stuff is an exercise in frustration. It looks to me like Flickr’s abandonware API isn’t really up to the task of even supporting a third party application to untangle the mess.

And of course, since Yahoo changed hands and various security scandals unfolded just logging into Yahoo accounts is a pain, and you need to navigate ads to even get into your account. Yahoo is the GeoCities of 2018.

Google photos in "action"
When I scroll to some arbitrary point on my timeline, this is what I see for some random (long) amount of time…

Recently, Google raised the “free” tier of photos.google.com to unlimited storage of photos where RAW files are JPEGs are processed into high-but-not-full-quality JPEGs on-the-fly. I’ve tried it and it’s pretty damn good. The uploader is smart enough to skip files that are clearly not important photos (e.g. too small, wrong format) and ignore obvious duplicates. The problems are (i) that the uploader application periodically just hangs and needs to be manually killed and restarted (ii) the web app seems to be weirdly slow and unreliable (I can log on with two machines side-by-side and they’ll see different subsets of my photos), (iii) no Apple TV support, and (iv) online photo editor seems to need one or two extra clicks to accomplish anything (but it’s a lot better than nothing). I’m pretty confident that my stuff is there, just not in my ability to see a given photo from a given machine on a given day. It’s certainly the most complete, easy-to-navigate, and shareable archive I’ve ever managed to create of my photographs. And if I can find a photo there, I can locate the original RAW image pretty easily.

Now, the absolute best system for dealing with my photographs thus far is iCloud. If I could simply rent 10TB from iCloud for a reasonable price (let’s say, $25/month) and get my Mac to automatically sync multiple volumes to iCloud, my problem would be solved. Obviously, I’m a happy Apple customer. If I were a more-than-casual Windows or Linux user then this would not be a useful option to me, and I’m not sure what I’d do, because I’m pretty sure there’s no equivalently seamless option for people who don’t want to pay the “Apple Tax”. Google Drive isn’t even a tolerable substitute for DropBox (although I think it has Sharepoint beaten).

Here’s where iCloud beats all other options:

  • I don’t need to think about it or do anything. (Well, on a desktop device, I need to NOT avoid storing my data in iCloud) If I take a photo, then it ends up in the cloud pretty quickly (basically, when the device gets recharged while on a LAN, if not sooner).
  • By default, full-resolution images are not propagated to all my devices (as would be the case for DropBox, or Hubic if it actually worked). Instead, as with everything in iCloud it’s available on-demand. (Indeed, it’s a bit reminiscent of the way iTunes deals with movies… superficially less convenient than pure streaming, but a lot more flexible and useful in practice).
  • If I ingest a RAW photo from a camera onto a device, then it’s in the cloud and available from any device on-demand (but it’s not wasting space on all my devices).
  • If I want to work on a photo, I can use the best native tools that are available on the device I’m using — seamlessly (although I’m inclined to actively avoid Adobe applications because Adobe’s workflow involves use of Adobe’s barely functional Cloud ecosystem).

The big problem — of course there has to be one — is that Apple’s highest storage tier is 2TB. I’m currently on 200GB which is plenty for the stuff I need that isn’t photos and videos, but hopelessly inadequate for photos and videos. 2TB (the next tier up, and it’s competitively priced) would be sufficient for my photos and videos if I were to curate them, but I don’t want to curate shit. I want to dump it in the cloud and not think about it.

Missing in Action

All of this adds up to a bunch of pretty disappointing non-solutions. Even though Apple provides a file sync system that works pretty well for personal photographs, it wouldn’t work for say a small photography business. (I guess you could use some kind of “family plan” but I’m pretty sure that would run you into weirdness pretty fast.) And it’s not like we’re talking advanced workflow support here — I just want my photos backed up and available.

Where is a tool that automatically detects blurred, underexposed, or overexposed photos and flags them as less worthy of backup? (Google’s photos app does a pretty good job of automatically correcting exposure, I wonder if it’s smart enough to task the uploader with going back to the RAW and reprocessing and re-uploading the photo?)

Where is the tool that remembers which photos have been opened or zoomed in and flags them as more interesting or worthy of backup?

Where is the tool that correlates the GPS location data of your iPhone photos and tentatively applies them to your corresponding camera photos?

Aperture used to collect photos from bursts into a single set and represent them with what it guessed was the best one. Where did this idea disappear to?

There’s a ton of low-hanging fruit here. Someone, please do something. I’m busy.

b8r 1.0

I decided to make the current revision of b8r “1.0” (it’s still marked “prerelease”) based on the fact it’s proven solid and usable during a year of constant use and improvement. It has been at least as robust and easy to work with as the jQuery-dependent library we developed at USPTO. I’ve just updated bindinator.com and my galaxy demo.

Recently, I made the first deliberately breaking changes and the difficulty of adapting various codebases that use b8r was pretty minor. So, I’m pretty confident that b8r is in good shape.

b8r's "fiddle" component in action
b8r’s “fiddle” component in action — b8r’s fiddle.component.js currently weighs in at 272 loc including markup, css, and comments.

I also improved the appearance of the inline fiddle and test components, and added prism.js code-rendering to all the various inline code examples to make the documentation pages look snazzier. A nice change to the test library makes sure that async test results are consistently ordered, and added a visible “pending” state so you can see tests that somehow failed to complete.

b8r has some pretty nice stuff. (Although much of this nice stuff needs to be documented.) You can download b8r, put nwjs in your Applications directory and/or npm install electron and double-click a .command file to see the b8r documentation inside a desktop app. Or you can install nodejs and double-click a .command file and serve it locally via https (I also provide a .command file will generate local ssl keys). (The .command stuff is currently Mac only, for which I apologize. I imagine it would be very easy to do it for Linux and Windows, but I haven’t tried.)

"electron-file-browser" component (running inside nwjs)
“electron-file-browser” component (running inside nwjs)

There’s a cute feature if you load the b8r documentation in nodejs or electron and command-click on a component — the component is loaded in a new window. I’m planning on leveraging this functionality to let the documentation app function as an IDE.

I’m currently working on convenience methods for multi-window desktop applications (it would be super cool if you could transparently bind across windows and browser tabs). I’ve also written a new version of foldermark that uses a very simple PHP back end (nodejs servers are still a much bigger pain to deal with than PHP) combined with b8r on the client-side.

The biggest shortcoming of b8r remains the fact that my team is the only group really using it. Because we’re developing a desktop app using Electron, we aren’t constantly testing on Edge, Firefox, Safari, etc.. I know for a fact that it has problems in IE and Edge, and that some of the example components aren’t touch-friendly, and we’re definitely doing more stuff for Electron than for nwjs (nwjs is much simpler to work with, but it’s becoming increasingly irrelevant, I fear). But if you’re working in reasonably recent releases of Chrome or Chromium, b8r should be very solid.

So, that’s the way it is: b8r 1.0.

Replacing Crashplan

I’ve been pretty happy with Crashplan for five years or so, although lately not so much. Obviously, one reason to be unhappy with Crashplan is that they’re no longer supporting the family plan I was using to back up all our Macs. Worse, their quality of service seems to have (understandably) slipped as my plan approaches its termination.

Exhibit 1: my wife bought a new laptop. When she tried to restore files from her old laptop, they were MIA.

So, here we are at a nasty juncture where our supposedly bulletproof fire-and-forget backup system is (a) not working terribly well and (b) shortly going to not work at all.

Possible Crashplan Replacements

The obvious replacement is Carbonite. The problem with Carbonite is that it’s going either (a) be a lot more expensive than Crashplan was (because of our family plan — which is the equivalent of, effectively, n $100/year plans where n is the number of computers you back up; the small business plan works out as being stupidly expensive if you have more than 250GB of data) or (b) require me to do a bunch of work (i.e. set up one computer as the family server, have all the other devices back up to it, and then back it up via Carbonite. So Carbonite will either cost $300 (say) per year or $100 per year but require me to be my own network engineer. Oh, and forget mobile backups.

If I want to do a whole bunch of work I might as well just use Amazon S3. For 2TB of data that’s a mere $46/month. Ouch. I could probably use their cheaper long term storage, but now I’m basically starting my own Crashplan / Dropbox implementation and that sounds kind of like hard work. Forget that.

So, on to the “consumer” options (I’ve tried to pick 2TB plans as this is the absolute minimum I can live with):

  • Dropbox — $99 p.a. 1TB (there is no 2TB plan)
  • Box — $540 p.a. for unlimited (3 computers)
  • Hubic — $60 p.a. 10TB
  • Amazon — $120 p.a. 2TB
  • Google — $240 p.a. 2TB
  • Microsoft — $99 p.a. 5 users, 1TB/user
  • Apple — $120 p.a. 2TB (no Android client)

(Edit: TablePress was a disaster.)

Note that all these services support web browser access, so you can get at your files from any device with a web browser, but I’m talking about file-system integration where you can just save your file in the usual place and it’s seamlessly backed up to the cloud. I should note that Dropbox and Hubic even provide Linux clients. Everyone supports Mac, Windows, and iOS.

At first glance, Hubic looks like the standout value-for-money option. (Hubic is essentially like Dropbox, except it’s run by a gigantic hosting company). The problem is that I’ve found Hubic to have a poor user experience, especially with regard to performance. (My own experience is quite limited — and I’m kind of shocked that my blog post on Dropbox vs. Box vs. Hubic ranks quite high on Google searches for “Hubic Review”.) This may be a result of server location, or simply under-provisioning (10TB is so close to “unlimited” that I imagine it attracts a lot of abusive users).

(Edit: I should mention that Hubic offers both Dropbox-like services and backup services. It also has what looks like a pretty robust API. Also there are advantages to storing your data in France. On the other hand — no 2FA.)

At second glance, Microsoft OneDrive seems like a great deal at 5TB for $100/year. The problem is that it’s 5x 1TB per user, which is effectively 1TB. Still, a great deal if you’re happy with 1TB and want Microsoft Office for your family.

The shock comes at third glance — Apple’s product is competitively priced (cheaper than Dropbox!). It doesn’t support Android (surprise!) but aside from that it’s a great deal, you can share it with your family (with each person having segregated storage), it requires no real configuration and — this is the kicker — it’s smart about mirroring stuff to your devices. Want a 1TB photo library in the cloud? Great. Want it on every one of your family computers? Not so much.

Conclusion

Hubic is the clear winner in terms of price per unit storage. Apple is the clear winner in terms of functionality and comes second in price. When I take into account the fact that I pay Apple $3/month for a lower tier of storage just for convenient mobile backups it’s an even better deal (hooray for Opportunity Cost). After some consideration I’m thinking of doing both. Hubic for volume backup and iCloud for convenience.

I’ll let you know how it works out.

Followup…

I nearly gave up on Hubic. After paying for the 10TB plan I received no response and my account didn’t change… Well, until the next day. Sacré bleu! I’m still a bit concerned that the app hasn’t been updated since 2015, but it seems to work.

Anyway, I’m going to start backing up to Hubic and we’ll see how things go.

 

Further research has shown that there are two more reasonably-priced alternatives to Carbonite, notably iDrive and Backblaze. Backblaze seems very compelling for a single computer ($5/month unlimited; $4 with a two year plan), and not bad for a small number of computers (it works out as roughly the same as Crashplan for 3 computers). iDrive is offering some interesting discounts (and lets you handle any number of computers with one account) but I find the website poor to the point of suspecting the competence of the company.

System Update Weirdness

I’m simply recording this for the benefit of others who may have similar weird experiences updating to 10.13.6.

I recently updated macOS via the App Store from 10.13.5 to 10.13.6 and it was unusual in that it got about two-thirds of the way through the install fast (while saying it had 16 minutes to go) and then got stuck.

When I noticed the problem a couple of hours later, I

  • forced power off and restarted. It then quickly went up to about the same point it previously hung, rebooted, got to about 90% of the way done, rebooted again, then got to about the same spot and the screen went black and stayed black.
  • forced power off and restarted (I might have had to do this more than once at this point) and then it got to the faux sign-in screen you now get when Macs aren’t fully booted (e.g. after the battery runs flat during stand-by).
  • I logged in and got to the proper login screen, and was was relieved to find that everything from there proceeded normally — the OS asked me if I wanted to send feedback to Apple, etc., which seems to happen after some OS updates but not others, but also wasn’t the “new user” experience where you’re asked to log into iCloud again.

As a side-note, this is an experimental first post via Ulysses.

A better RAW workflow for $35

Processed and cropped in RAWPower
Processed and cropped in RAWPower

As my frequent reader knows, I have been grappling with my RAW workflow for as long as I have had a RAW workflow. I’m hardly a pro or even much of an enthusiast, and I find dealing with all these files exhausting (it also consumes a stupid amount of disk space, etc.)

A user of Photoshop since it was called Barneyscan, and Illustrator since it was 88, I’ve been ambivalent about Adobe’s products ever since they started renting them; this was actually before they switched to monthly fees — the Creative Suites essentially forced you to upgrade on a constant basis simply to not have your software mysteriously stop launching when Adobe’s authentication servers were down.

Today, despite paying Adobe’s tax (albeit the lesser “Photographer’s” tax of $100/year) I remain unhappy with their products. Lightroom is slow, constantly wants patching, requires me to sign in (often more than once) to Adobe’s stupid services just to launch, and on and on. But, until recently, I had no credible alternative that was fast and produced even vaguely decent results.

But now there are two inexpensive, lightweight products that together may mean I don’t need Adobe’s crap any more (I’ll get back to you!):

A quick press of the P key and I can see that I nailed focus on the grass
A quick press of the P key and I can see that I nailed focus on the grass

FastRawViewer — endorsed by no less than Thom Hogan and Nasim Mansurov — is a terrific program that does exactly what it says on the can. It’s simple and lets you browse and rate photos really, really fast. You can customize its keyboard shortcuts to your pleasure (e.g. I have ratings mapped to the 0-5 keys, and P toggles high pass filtering so you can see exactly what, if anything, is in focus without pixel-peeping. Rather than having its own proprietary catalog system, it leverages your file system and XMP metadata (“sidecar” files that are compatible with Lightroom if that still floats your boat). It costs $20, you can get it here.

RAWPower lets me make RAW adjustments, straighten, and crop faster than I can launch Photoshop
RAWPower lets me make RAW adjustments, straighten, and crop faster than I can launch Photoshop

RAWPower — developed by former Aperture engineers (or a former Aperture engineer; I’m not sure) — gives you most of Aperture’s non-destructive RAW processing in a fast, lightweight app that also provides the same functionality via Apple’s Photos app. I like the Photos app except for the whole slower-than-treacle-in-a-walled-garden thing, so there’s that too. It costs $15 in the App Store. My only issue with RAWPower is that its crop-and-rotate tool is clumsy if you want to both crop AND rotate, which I usually do (and I’ve been told that addressing this issue is a priority).

(If you’re a Windows user, FastRawViewer is still great, but RAWPower is Mac only.)

FastRawViewer lets me view a folder with thousands of RAW files with no waiting (just dragging the folder info Lightroom, Photos, or Aperture would be agony), and RAWPower lets me adjust exposure, shadow recovery, straightening, and so forth faster and just as competently as Lightroom. (Photoshop still wins for any major surgery, obviously — RAWPower has no dodge, burn, layers, healing brush, perspective correction, stitching, etc.)