Adventures in Mobile Development

My initial foray into mobile development was to create an iPhone app from scratch using XCode. I wrote a simple mortgage calculator in an evening, and then spent a ridiculously long time trying to figure out how to arrange my signing certificates and whatnot so as to let me actually run it on my iPhone. (I never did get it running on anyone else’s.) Given that I am only superficially familiar with Obj-C and Cocoa, this was not terribly painful. If I could write apps for iOS and Android using XCode and Interface Builder, I’d be very happy. (And, from my point of view, XCode 4 is looking like a huge step forward from XCode 3.)

More recently, I ported my Manta game to iOS.

(Actually, I just realized that my first foray into mobile development, back in 2001, was an attempt to build a Mobile Flash-based multimedia detailing tool for the Compaq iPaq, but I had successfully repressed that memory. You think Flash sucks on mobile devices today? Shudder.)

Lately, I’ve been trying to figure out a good strategy for rapidly prototyping, developing, deploying, and supporting mobile apps for a large enterprise, and it’s been … educational. (“Experience,” Randy Pausch noted, “is what you get when you didn’t get what you wanted.”)

The first step in this latest adventure was an excellent tutorial: How to Create an HTML5 iPhone App (Alex Kessinger). I didn’t bother with his Tetris clone, as I was trying to build a data-driven productivity app, so I simply used his tutorial as a shortcut to figuring out the basics of manifests and the stupid HTML incantations necessary to get a non-scrolling full-screen web app working. This persuaded me that pure HTML5 web apps are a perfectly viable way to go for most business stuff, but you’ll still need to build a minimal wrapper to deploy via an App Store.

I should state at this point that this does not qualify as a review so much as a disjoint set of observations based on generally shallow experience of a number of mobile development options. It is, in effect, a quick brain dump of my current impressions rather than any kind of careful analysis.

Mobile development options range between two extremes: mobile-friendly web pages and Native development using platform-specific SDKs. (It should be noted that if you want to provide a “real” app through an “app store” the platform-specific SDK is pretty much unavoidable, but you might be able to avoid writing all your code in an IDE you despise.)

Mobile-Friendly Web Development

The purest web option is, of course, to simply create a website. You don’t need to do anything except test to see that your website isn’t pathological on a mobile platform (e.g. do you use Flash for UI elements, are your article’s coerced into bizarre multi-column layouts or otherwise broken up in a way that makes them hard to zoom in on and read?).

You can be a bit more mobile-friendly by creating custom CSS to cater for mobile web browsers, including a mobile (or at least iOS) friendly favicon, and possibly even providing manifests and so forth to make some or all of your pages work offline (although working with manifests is an absolute nightmare during development). It’s possible to make web pages look arbitrarily similar to a native app just using a bit of CSS and Photoshop magic, but it’s very hard to create native behavior and responsiveness, and you won’t have access to platform-specific functionality (e.g. the camera or GPS).

Personally, I think that all websites should be mobile-friendly, but making mobile-specific websites is generally a Bad Idea. (Even when done well, e.g. ESPN.com, I think I’d rather just get the “real” site if the real site weren’t horribly heavy.)

PhoneGap + Your Favorite UI Library

PhoneGap allows you to build a “native” app out of HTML and JavaScript by essentially building a mini-site into a simple custom app framework that simply provides an interface between the JavaScript runtime and the underlying mobile OS. PhoneGap doesn’t attempt to solve the problem of providing native look and feel — that’s between you, Photoshop, and the JavaScript programming deities.

Luckily, the JavaScript programming deities have smiled on us and provided both free/open source and non-free solutions to the other side of the problem in the form of jQuery MobilejqTouch, and Sencha Touch. (Sencha being the folk who produce ExtJS, which is a superb JavaScript widget library.)

I should mention that while jQuery Mobile and jqTouch are implemented as jQuery extensions, Sencha Touch is perfectly compatible with jQuery (I actually first learned jQuery and ExtJS together, since I was working on a site that used both). This is hardly surprising since both jqTouch and Sencha Touch are from Sencha! From what I can see (and from this response on StackOverflow) jqTouch was an experiment that got open-sourced, Sencha Touch is what was built based on what the devs learned building jqTouch (and it’s bigger, more complex, and actively maintained library), and jQueryMobile is the official jQuery mobile platform (which may or may not have borrowed stuff from jqTouch, I don’t know). More importantly, working with Sencha Touch is more like pure coding, while jqTouch and jQueryMobile apps are, at the bottom, web pages.

jQueryMobile has, unquestionably, the most impressive website of the three — go to the Docs and Demos pages and then try resizing your browser Window. (Here’s a link, but note that the url seems to be version-specific and may stop working.)

Visual Basic Reloaded

A little further along the spectrum there are Appcelerator Titanium and Ansca Mobile’s Corona SDK which essentially remove HTML from the equation and replace it with a native canvas (in essence, Appcelerator is “Visual JavaScript” and Corona is “Visual Lua”). Appcelerator Titanium seems to have quite a bit of traction (supporting JavaScript is probably a good marketing move) but Corona has probably produced more commercially successful results (Lua is generally more performant than JavaScript).

Corona is more 2d-game focused (your ability to use native widgets is pretty limited, and there are lots of iOS-centric examples of faux native controls) while Appcelerator is more enterprise-app focused (and makes a point of exposing platform-native controls). As a personal aside, Appcelerator is a huge pain in the ass to get going (it is Eclipse-based but requires an appropriate version of XCode to be installed for iOS development and the Android SDK to be installed “just right” for Android development) with while Corona SDK simply provides a very nice simulator that “just works” (it doesn’t do platform native stuff, but works fine for many purposes) that lets you use whatever development tools you like.

If you don’t need a lot of OS-level widgets (and don’t mind learning Lua) Corona is much easier to get up-and-running with. But if you want to create native productivity apps that run on both iOS and Android, Appcelerator is much more capable.

The 800lb 3D Gorilla

Unity differs from Appcelerator and Corona in three key ways. First, its runtime is 3d-game oriented rather than 2d or productivity oriented. Second, its runtime is both more capable in some ways than its competitors because it contains a chunk of Mono (i.e. .NET) functionality and less capable in others because not all native platform capabilities (e.g. browser panels and other UI widgets) are supported. Finally, it runs compiled code against its own APIs. This means that Unity is going to be useless for certain kinds of development (e.g. want Google Maps integration? Forget it) and vastly superior, perhaps even to native development, for others (i.e. 3d games and other realtime 3d stuff).

HTML PhoneGap+ Corona Appcelerator Unity
IDE n.a. XCode for iOS, Eclipse for Android n.a. Eclipse Unity
Native Widgets Kind Of Kind Of No Yes No
Best Feature You’re already done Free Lightweight Native Widgets Awesome Dev Tools
Worst Feature DIY Still kind of DIY Need to build app to see supported native widgets Fiddly to get going Not intended for non-games
Language JavaScript JavaScript Lua JavaScript JavaScript[1], C#, Boo[2]
Price free free $199 p.a.+ $49/month+ $400+
Personal Opinion manifests are a pain to deal with (because stuff gets cached on your iPhone and it’s hard to dislodge) haven’t gotten too far down this road, but Sencha Touch looks great very click, but productivity apps seem to be a poor relation absolute pain in the ass in every conceivable respect love unity but it’s kind of heavy and for productivity stuff you’re on your own

Notes: [1] Unity JavaScript is not quite the same as JavaScript, but it’s easy to get up-and-running if you know JavaScript. [2] Boo is a Python-like language with curly braces instead of parens, and just enough dynamism removed to allow it to compile.

I need a Mobile App tomorrow!

If all you want is a mobile-friendly landing page (and you’re willing to pay), Widgetbox has a tool for creating a mobile “web app” with point-and-click simplicity. I’ve played around with their tools enough to verify that the work, but I haven’t actually tried to deploy their tool so I don’t know how well it works offline (e.g. can you bookmark the site on your iPhone and have it work offline correctly?) It would be pretty neat to have a tool along those lines that actually let you build a proper app.

Going Native

Finally, there’s native development, which I won’t go into. (I like XCode and loath Eclipse so I’m pretty biased.) Aside from producing a truly native app, the iOS SDK has a bunch of functionality that none of these tools match (I don’t know enough about Android to know whether the Android SDK does too). E.g. that way iOS lets you handle screen orientation and user interface transitions is simply light years ahead of most of these tools (where you’re pretty much on your own) and solidly ahead of the best (which would be Sencha Touch).

The Story So Far…

At this point, I should mention two of my biases and get them out of the way. I don’t care for Eclipse, and I don’t like subscription-based models, and Appcelerator Titanium has both. Now, to be fair, my opinion of Appcelerator was perhaps unduly harshed by my having been using Mac OS X [redacted for NDA reasons] on my Macbook Pro which meant nothing much seemed to work (although I also spent a huge amount of time not getting it working on my Mac Pro, which is running 10.6.x). Basically, in order to work, Appcelerator seems to need all the stars to be in alignment and I have yet to get them so aligned. I like the idea of Appcelerator, but thus far I’ve found the execution leaves much to be desired.

Corona SDK on the other hand (which is not without OS X [redacted] issues of its own) was a joy. The download was small and I was able to get the simulator up and running in minutes, and complete the excellent tutorial with no problems. I immediately grokked how things worked and was able to dive into their API documentation and add extra functionality to my toy tutorial app immediately. (Despite not really knowing Lua.) I should add that this is the experience I’d hoped Appcelerator would deliver, but no. (I’d also add that the Appcelerator site has a bunch of getting started video tutorials that never actually seem to get to the point of doing anything interesting. In terms of “five minute demo” Corona SDK has Appcelerator beat.)

The problem with Corona (for me) is that if I’m going to use a game-centric development tool, I might as well use Unity. Corona does have the advantage of seeming lighter (and it’s certainly cheaper). I guess if the right project came along I’d probably consider Corona, but business productivity apps just doesn’t seem to be Corona’s strong suit.

Frustrated by Appcelerator’s lengthy and fiddly installation process (and being forced to use Eclipse) and Corona’s lack of support for native widgets, I started looking at the PhoneGap+ options once more, and I have to say that I’m very impressed by jQueryMobile and Sencha Touch. Simply based on the jQueryMobile doc site (and its “eat your own dogfood” approach), I’m leaning towards PhoneGap + jQueryMobile. That said, I finally managed to get Titanium Developer to launch the “Kitchen Sink” app in my iPhone Simulator, which means I might be able to code with Accelerator the way I was able to code in Corona (modulo a much more cumbersome build/debug cycle).

So, to sum up, I like Corona. It’s lightweight, really easy to pick up and use, and fun. Needing to learn a new and not especially widely used language is a bit annoying but it’s no big deal. But I really can’t use it — it’s not a general purpose app development platform but a 2d game platform (e.g. it has no selector widget, but it does provide a system for managing virtual currency).

I’ve not yet built a PhoneGap app but, assuming it all works as advertised (which is clearly a huge assumption for mobile development), it looks like this is a useful option, but it’s probably never going to be as refined as a truly native application (e.g. none of Sencha’s widgets look native on an iPhone, and a few of jQueryMobile’s widgets are downright ugly).

Unity continues to rule the roost for game development, but it’s virtually useless for what I’m currently trying to do.

And finally, there’s Appcelerator. In some respects it seems like the Holy Grail, but it’s been such a pain in the ass so far that I’ve just spent the last two days scouring Google for alternatives (culminating in this blog post). The fact that getting up and running was so painful isn’t promising, the documentation on the website has strange gaps (“hello world”, anyone?), I find the instructor in their video tutorials to be infuriatingly slow, and I’ve read nightmare stories about bugs in the platform that have brought projects to a screeching halt.

I also don’t see a single example of a “cool app” developed with Appcelerator. What I see is a bunch of giant corporations that are using Appcelerator for reasons that probably make no sense to end-users. (What App does NBC Universal ship and why do I care?)

So, right now, I’m probably going to experiment with PhoneGap and Sencha Touch or jQueryMobile. If not, I guess I’ll bit the bullet. At this point, if I could ignore other platforms and just code in Obj-C for iOS I would avoid Appcelerator altogether. Oh well.

Postscript

I came across this blog post (Martin Kou) in my meanderings. The main takeaway is that the writer finds jQueryMobile to be pretty flaky, and hasn’t tried Sencha Touch but says that it is “purportedly, much more mature and is faster”.

How to use HTML5 <video> tags everywhere and have them Just Work™

Chrome's HTML5 video player interface
Chrome's HTML5 video player interface (just a screenshot, it doesn't actually work)

I’ve been coding raw html since 1994 (when I learned just enough to create a home page for Prince of Destruction — check out the all the <BR>s). I can remember all kinds of random CSS properties and half the jQuery API, but when I need to embed video in a web page I still need to google a bunch of references to get anything that works halfway decently.

Until Google pulled the rug out from under H264 there was some hope that embedding video inside a web page might soon become as simple as embedding images has been since the days of NCSA Mosaic. It’s hard to imagine a non-trivial piece of code simpler than a basic image tag. (If you have any doubts as to whether Google really wants the video tag to succeed, take a look up at how horrible the video player UI in Chrome still is as of version 5. I might add that it can hardly play video without skipping and flickering.)

Whether or not Chrome (and Android) support H264, we’re going to be stuck with a ton of legacy users (Firefox, IE6/7) for a long time, and the only viable solution if we want to encode video in as few formats as possible is Flash.

Well, Flash supports H264 and Webkit supports H264, so what I want is to:

  • encode my video once, in some flavor of H264 that “just works” — ideally I want to be able to open a movie in QuickTime, Save As… and I’m done.
  • embed my video using the simplest, easiest-to-remember html possible, i.e. <video src="foo.m4v" width="800" height="450" controls="true">
  • and have my video play back everywhere

So, here’s my solution. Use HTML5 video tags, and use a tiny bit of JavaScript to automagically convert the video tag into a Flash embed where necessary using the information already in the web page (which can be found in the video tag’s attributes).

I’ve built a simple prototype here. It uses video and h264 detection code from Dive into HTML5 (which means that for now Chrome plays back the video natively) and jQuery for cross-browser DOM manipulation, but hand-coding the relevant bits wouldn’t be that difficult if I wanted to make the code self-contained.

In essence, all I need is: $( function(){ $('video').replaceWith( flash_embed_crap ) } );

Most of the code is simply regurgitating boilerplate html for Flash embedding. (Incidentally, the code will have issues in IE7 thanks to the “Eolas bug”, but if I were to put the script in an external JavaScript file that issue would disappear. Any weird IE behavior should be gone.)

The Flash video player I’m using is very simple — it pretty much uses Flash’s built-in video components and has some simple glue code allowing the embedding web page to talk to it once it’s embedded (not that I’m taking advantage of this here).

It works.

P.S. As per the comments in my source code, I plan to extend this code to provide a number of useful convenience functions and support <audio> tags the same way, and finally to provide a simple but robust open source Flash media player. Aside from anything else, this will help me implement cross-platform media playback in Acumen.

P.P.S. My initial attempts to do for <audio> tags what I’ve just done for <video> tags have not met with much success.

My big problem is that Flash’s FLVPlayer component doesn’t seem to want to play MP3s for me. I’m not sure what’s going on there; it may be some odd combination of Flash security settings hosing locally hosted playback and my ISP’s server configuration hosing remote playback, or FLVPlayer may just be fragile. I’d like the audio and video player UIs to be as identical as possible and for the player to be as simple as possible — having two completely different code paths for audio and video seems like hard work (mostly from a UI point of view).

So the upshot is I can’t change this article’s title to include <audio> tags (yet). Sigh.

Fanboys 1. Pundits 0

Cosmic Encounter could be a great iPad game

So Apple is running out of iPads, Dan Lyons has changed his tune, magazine publishers are jumping on the bandwagon. I guess sometimes the fanboys are right. (Indeed, I’d have to say that Apple fanboys tend to be pretty accurate — certainly more accurate than the professionals — on Apple’s actual announced products, they just suck at predicting anything about unann0unced products. But doesn’t everyone?)

Board Games

It took a while, but some commentators have figured out that the iPad will be great for boardgames (unlike any other platform except Microsoft’s Surface, which is more of a tech demo than a product). For $3.95 on launch day you can get BoardBox — a collection of classic boardgames rendered very nicely (by the looks). I haven’t seen it working (for obvious reasons) but I’m going to take a wild guess that it may have some competition in price, quality, and polish. Travel Scrabble costs $20 on Amazon, and if you’re like me it’s a product you use once and never again.

I’d love to see some classic board wargames converted to the iPad and their — often horribly complex — rules systems automated. SPI’s old freebie Napoleon at Waterloo would be a great start. It’s something I’d love to do myself, but I simply don’t have time. A really great, obvious candidate would be Victory Games’s Ambush! Any game that involved a ridiculous collection of expansion packs or incredibly fiddly components (e.g. Cosmic Encounter or Illuminati score on both) also has a shot.

RPGs also have potential, perhaps more than any of the above. Ideally, each player would have their own iPad, but one would be enough. At last, one purchase could get you the game system, character sheets, “miniatures”, dice, and content.

Children’s Books

Another killer app is children’s books and interactive “games”. A typical children’s book costs $10 and has a lifespan of approximately … well if your child likes the book, about a month. I imagine we’ll quickly see a whole bunch of famous old titles, such as the old Broderbund “Living Books” ported (they’ll need to figure out the whole mouseover thing, although if I recall correctly they were more driven by clicks than mouse activity).

The Opportunities are Endless…

The iPad is, essentially, interactive paper. (The Kindle is, essentially, paper. Big difference.)

It’s the gizmo Bill Atkinson envisaged when he was trying to come up with the perfect computing device (not quite cheap enough to lose and not care, but getting there) and then gave up and produced HyperCard. Anyone who deals with maps (pilots, navigators, field geologists, the military) will want one because it can replace maps (imagine a map with interactive “find” or automate great circle navigation or automatic magnetic north correction), which are expensive and a pain in the ass. If you look at the way computers are used in hospitals, iPads are a huge, obvious win (both cheaper and better suited to hospital environments, where keyboards are a hygiene and usability disaster) — indeed healthcare loved the Newton and has been one of the strongest proponents of tablets in general.

Fewer Things to Lug Around?

If I may return briefly to one of my hobby horses: new tech devices succeed when they reduce the amount of crap you need to carry around with you, the iPad is poised to succeed in ways we simply haven’t figured out yet. These days, I carry around a backpack that usually contains (a) my laptop, (b) a notebook for sketching and taking notes, (c) as many useful/interesting books as I can bear to carry, (d) miscellaneous cables (including multiple sets of headphones for my iPhone because I’m always misplacing them). If we make the reasonable assumption that my laptop is going to stay there for the time beingand that the iPad’s battery life will be sufficient to permit its use for notes and sketches then this will change from laptop + notebook + books + cables to laptop + iPad + fewer books + cables. (Sadly, most of the books I am likely to carry around are not available electronically.) So, not a huge win for me. Oddly enough, the big win is that I’ll end up taking more stuff with me (e.g. I get the New Yorker via email — now I can read it conveniently) and having less stuff at home (e.g. I will cancel all our physical magazine deliveries). Similarly, I’m going to bite the bullet and rip our DVD collection. If only book scanners were a little more practical…

It could be a huge win for me when traveling — if I trusted the TSA not to steal my laptop from checked baggage and/or I didn’t have to pull out the laptop at airport security if I packed it in a carry-on. LogMeIn and its competitors are available for the iPhone, so I wonder whether I might be able to use an iPad as a terminal for a “real” computer on the road (1024×768 is hopeless for “real” computers these days though).

The thing I don’t anticipate doing, but I may yet be surprised, is giving up carrying a laptop around. Here’s what I use my laptop for: 3d modeling (using Cheetah 3d and Blender), 2d graphics, web development, software development (occasionally), game development (using Unity), email, web surfing, games (very occasionally). The thing is, I basically travel between two locations — home (where I have a very nice desktop system to work on) and work (where I have a tolerable system to work on). I can easily imagine decent sketch tools being available for the iPad (both 2d and 3d) and quite possibly a Blender port (since it runs on modest hardware). I think it’s safe to say that web, software, and game development on an iPad is not on the cards for the foreseeable future, but the iPad may prove to be a great tool for planning development vs. doing it.

Adventures with <canvas>

acumen_asset_viewer
Acumen's Asset Viewer

Acumen is at 1.0 beta, and we finally settled on a hybrid image viewer. Google’s excanvas.js is used to fake a canvas in IE (getting it to work in IE8 was the trickiest part) but because excanvas.js runs pretty slowly and doesn’t support text (there are third-party text implementations of text in excanvas, but they’re also slow and don’t use standard fonts) I ended up using canvas just to render images. All main UI elements are implemented in HTML/CSS using jQuery.

Based on my googling, successfully creating a canvas that transparently works in either decent browsers or IE is not common knowledge, so I thought I’d share the method I came up with. I originally wrote this code for the pure canvas asset viewer (which forced IE users to install Chrome Frame); when I decided to reuse the code, I tried to simplify it, and everything I did to make it simpler broke it. I guess the me who wrote the original code wasn’t as big an idiot as the me who borrowed it thinks he was.

var c; // will contain a working canvas when we're done
var d = $('#div_that_will_contain_canvas');
var ie_version = some_expression_that_is_truthy_for_ie();
if( ie_version ){
	c = document.createElement('canvas');
	c.id = viewer_id;
	c.style.width = d.width() + 'px';
	c.style.height = d.height() + 'px';
	d.append( c );
	G_vmlCanvasManager.initElement( c );
} else {
	d.html('<canvas id="' + viewer_id + '" width="' + d.width() + '" height="' + d.height() + '"/>');
	c = document.getElementById( viewer_id );
}

The main takeaway is that creating the canvas using a blob of html does not work for IE (because it doesn’t recognize the canvas element, and thus ignores its attributes) and using the standard DOM methods does not work for proper canvas elements (because using styles to resize a real canvas stretches the canvas).

If you have a cleaner way of achieving the same result please let me know—I’m not proud of this code, it just happens to work.

Google, On2, and Onward for <video>

Google’s recently announced acquisition of On2 paves the way for the possibility of a Flash-free future (F-cubed?).

With Adobe’s desire to push all of its most annoying technologies into Flash (witness the recent vulnerabilities caused by the interplay between PDF and Flash), the unpleasantness of Flash and the need to have it in order to play FLV video streams, and the way FLV is presented to the browser (as a URL to a SWF file along with completely arbitrary information telling that SWF file where to find the FLV), are all enough to give anyone trying to present videos nicely in a browser something of a stomach ache.

But now, Google owns On2 who, in turn, own VP6 (the most popular Flash video codec).

Ideally, this means that Google might release an FLV plugin for popular browsers that allows them to play Flash video directly without the need for a SWF player, or — better yet — simply release open source playback code for VP6 (if necessary providing the glue code for Webkit and Mozilla to play FLVs encoded with VP6 inside <video> tags). Without an enormous amount of effort, the need for Flash to play FLV video could be hugely reduced. Most FLV video is played using YouTube or JWPlayer and similar products. It wouldn’t be hard to infer the FLV url and ignore the player (certainly, it would be an easy thing to do as a Firefox Plugin).

Now, if Flash isn’t necessary to play video, exactly why would we want it? Oh, right, banner ads and games.