Many have been killed in the attempt…

The fundamental idea behind verification is that one should think about what a program is supposed to do before writing it. Thinking is a difficult process that requires a lot of effort. Write a book based on a selection of distorted anecdotes showing that instincts are superior to rational judgment and you get a best seller. Imagine how popular a book would be that urged people to engage in difficult study to develop their ability to think so they could rid themselves of the irrational and often destructive beliefs they now cherish. So, trying to get people to think is dangerous. Over the centuries, many have been killed in the attempt. Fortunately, when applied to programming rather than more sensitive subjects, preaching rational thought leads to polite indifference rather than violence. However, the small number of programmers who are willing to consider such a radical alternative to their current practice will find that thinking offers great benefits. Spending a few hours thinking before writing code can save days of debugging and rewriting.
— Leslie Lamport (most noted for creating LaTeX) in an interview here.

The interview starts with a lot of failed attempts to summarize the thrust of Lamport’s various papers, but eventually gets quite interesting.

Digital Archeology and Markup Options

I’m currently developing a tool for archivists which will allow them — or indeed anyone — to “publish” a repository of finding aids (xml documents containing metadata about collections of stuff, e.g. the papers of a 19th century lawyer) by, essentially, installing the tool (making some changes to config) and pressing “go”. Actually you don’t need to press go. There are a bunch of similar tools around, but most of them have the dubious virtue of storing and maintaining valuable data inside a proprietary or arbitrary database, and being a lot more complicated to set up.

E.g. the typical workflow for one of these tools, we’ll call it “Fabio”, goes like this:

  1. Archivist creates a complicated XML file containing metadata about some collection of stuff.
  2. Archivist creates or uses an existing XSL file to transform this into data Fabio understands.
  3. Fabio loads the data, and there are almost certainly errors in it because (a) the XML had mistakes in it and/or (b) the XSL had errors in it.
  4. Archivist discovers some errors and fixes either (a) the XML (and reimports), (b) the XSL (and reimports), or (c) the data in Fabio. Probably (c) because Fabio reimporting data into Fabio is a pain, and the whole point of Fabio is it’s supposed to be an “all in one” solution once you get your data munged into it.
  5. Later, more errors are discovered and fixed by the process listed in 4.

Now, the problem with all of this, as I see it, is that it’s completely nuts.

  • People who already know (to some extent) how to create and maintain XML and XSL files now need to learn how to use Fabio. Knowledge of Fabio is relatively useless knowledge (when you change jobs, the new place probably doesn’t use Fabio, but a completely different but equally stupid product).
  • Corrections may be made in either the raw data (XML), reusable scripts (XSL), or Fabio’s black box database (???). Later corrections can easily overwrite earlier corrections if, for example, one mistake is fixed inside Fabio and then another is fixed in the XML.
  • If Fabio stops being maintained or you just want to stop using it, all (or much) of your valuable data is in Fabio’s black box database. Even if you know how it’s structured you may lose stuff getting your data back out.
  • The XML repository needs separate version control anyway, otherwise what happens if you make a change to your XSL to fix one import and then need to reimport another file that worked before but doesn’t work now.
  • Data may be lost during the import process and you won’t know.
  • Fabio needs to provide an API to expose the data to third parties. If it doesn’t expose a particular datum (e.g. because it was lost on import, or Fabio’s developers haven’t gotten around to it yet, you’re out of luck).
  • Fabio may have stupid flaws, e.g. provide unstable or ugly URLs — but that’s between you and your relevant committee, right?

My solution is intended to be as thin as possible and do as little as possible. In particular, my solution wraps a user interface around the raw data without changing the workflow you have for creating and maintaining that data. My solution is called metaview (for now) but I might end up naming it something that has an available .com or .net address (metaview.com is taken).

  • I don’t “import” the data. It stays where it is… in a folder somewhere. Just tell me where to find it. If you decide to stop using me tomorrow, your data will be there.
  • If you reorganize your data the urls remain unchanged (as long as you don’t rename files and files have unique names — for now)
  • Unless you’re maintaining the UI you don’t need to understand it.
  • You fix errors in your data by fixing errors in YOUR data. If you stop using me tomorrow, your fixes are in YOUR data.
  • I present the raw data directly to users if they want it (with the UI wrapped around it) so that there’s no need to wait for an API to access some specific datum, or worry about data loss during import processes.
  • Everything except base configuration (i.e. what’s my DB password and where’s the repository) is automatic.
  • I don’t try to be a “one stop solution”. No I don’t provide tools for cleaning up your data — use the tools you already use, or something else. No I don’t do half-assed version control — use a full-assed version control solution. Etc. I don’t even think I need to do a particularly good job of implementing search — since Google et al do this already. I just need to make sure I expose everything (and I mean everything) to external search engines.

This has me thinking quite a bit about markup languages and templating systems. At first, I tried to decide what kind of template system to use. The problem for me is that there seems to a number of templating systems for any given web development stack that is some multiple of the nth power of people using that stack and the number of components in the stack. So if you’re looking at PHP (roughly two bazillion users) and MySQL (three bazillion) and Apache (four…) that’s a metric frackton of options, and knowledge of any one of those is way over on the bad side of the Knowledge Goodness Scale.

Aside: my unoriginal Knowledge Goodness Scale. This isn’t original, but I try to acquire “good” knowledge and try to avoid “bad”. The more times, contexts, and situations in which knowledge is accurate, useful, and applicable, the better it is. So knowledge about how to understand and evaluate information (such as basic logic, understanding of probability and statistics, understanding of human nature) is incredibly far over on the good end. Knowledge of how to build templates for a “content management system” that only works with PHP 5.2.7 with MySQL 5.x and Apache 2.x is closer to the bad end.

It follows that if you are going to force your users to learn something, try to make it good stuff, not bad stuff. So, let’s continue…

Archivists try to preserve knowledge and/or create new — preferable good — knowledge. We don’t produce index information or metadata about something because we want to have to do it again some day. The knowledge they’re dealing with is often very specific, but to make it good it can still be accurate and applicable across time. Developing an engine based on a bunch of technologies which themsevles are unlikely to be useful across a span of time and contexts is not a good start. (Card indexes have lasted a long time. If your electricity goes out or your server crashes you can still use them today.)

So, my solution involves requiring users to change their lives as little as possible, learn as little as possible, and build on their existing good knowledge rather than acquire new bad knowledge. Instead of figuring out the foibles of Fabio, they can learn how to better create and maintain raw data.

So, what’s the approach?

  • XML is rendered using XSL — on the browser or the server. If you want the XML, click the XML link. (It looks the same on modern browsers — folks with primitive browsers will need server-side rendering.)
  • The templating system is XSL.
  • The database contains index information, but is built dynamically from the repository (as needed).

Of all the different markup languages around, XML is probably the best. It satisfies much of the original intent of HTML — to truly separate intention from presentation (order is still much to important in XML — it’s quite a struggle to reorder XML content via XSL in a nice, flexible way). It’s very widely used and supported. Many people (my target audience in particular) already know it. And it’s not limited to a particular development stack.

XSL is part of XML, so it’s easy for people who already use XML to grok, and again it’s not limited to a particular development stack.

There’s no escapting binding oneself to a development stack for interactivity — so metaview is built using the most common possible free(ish) technologies — i.e. MySQL, PHP, JavaScript, and Apache. Knowledge of these tools is probably close to the least bad knowledge to force on prospective developers/maintainers/contributors.

Less Bad Options

I do have some misgivings about two  technical dependencies.

XML has many virtues, but it sucks to write. A lot of what I have to say applies just as much to things like blogs, message boards, and content management systems”, but requiring users of your message board to learn XML and XSL is … well … nuts. XML and XSL for blog entries is serious overkill. If I were making a more general version of metaview (e.g. turning it into some kind of content management system, with online editing tools) I’d probably provide alternative markup options for content creators. Markdown has many virtues that are in essence the antithesis of XML’s virtues.

Using Markdown to Clean Up XML

Markdown — even more than HTML — is all about presentation, and only accidentally discloses intention (i.e. the fact you’re making something a heading might lead one to infer that it’s important, etc.). But unlike HTML (or XML) Markdown is easy and intuitive to write (anyone who has edited ASCII files is already 75% of the way there) and the marked up text looks good as is (one of its design features). There are a ton of “similar” markup languages, but they are all either poor clones of HTML (the worst being bbcode) or just horrible to look at (e.g. Wiki syntax). Markdown also lets you insert HTML making it (almost) supremely flexible should the need arise. So, if I wanted to create an alternative method for maintaining content, Markdown seems like a nice option.

Markdown also seems like a nice way of embedding formatted text inside XML without polluting the XML hierarchy… e.g. rather than allowing users to use some toy subset of HTML to do rudimentary formatting within nodes, which makes the DTD and all your XML files vastly more complex, you could simply have <whatever type=”text/markdown”> and then you can have the XSL pass out the text as <pre class=”markdown”> which will look fine, but can be made pretty on the client-side by a tiny amount of JavaScript. In a sense, this lets you separate meaningful structure from structure that purely serves a presentational goal — in other words, make your XML cleaner, easier to specify, easier to write correctly, easier to read, and easier to parse.

My other problem is PHP. PHP is popular, free, powerful, and it even scales. It’s quite easy to learn, and it does almost anything I need. I’m tired of the “PHP sucks” bandwagon, but as I’ve been writing more and more code in it I am really starting to wonder “why do I hate it so much?” Well, I won’t go into that topic right now — others have covered this ground in some detail — (e.g. I’m sorry but PHP sucks and What I don’t like about PHP) — but there’s also the fact that it’s popular, free, powerful, and scales. Or to put it another way, PHP sucks, but it doesn’t matter. It would be nice to implement all this in Python, say, but then how many web servers are set up to serve Python-based sites easily? While it may be painful to deal with PHP configuration issues, the problem is fixable without needing root privileges on your server.

So while I think I’m stuck with PHP, I can at least (a) stick to as nice a subset of it as possible (which is hard — I’m already using two different XML libraries), and (b) write as little code as possible. Also I can architect the components to be as independent as possible so that, for example, the indexing component could be replaced with something else entirely without breaking the front end.

Mac Text Editors

One of the most baffling gaps in Mac third-party software is programming-oriented text editors with autocomplete. Now, obviously there’s XCode, vim, and emacs (which, being free, are all something of a problem for would-be competitors), and a bunch of cross-platform tools like Eclipse and Komodo, but XCode is really oriented towards Cocoa development and is kind of like using Microsoft Word to edit a shopping list, while the others are all non-Mac-like (and kind of suck anyway).

The Mac heavyweight text-editing champs — BBEdit and TextMate — have many great features, but don’t do autocompletion very well or at all, respectively. (There’s some kind of third-party hack add on for TextMate, but you need to compile it yourself and few seem to be able to get it working.) A lot of Unity developers were using PCs or Virtual Windows boxes to run Visual Studio just because it has an autocompleting text editor that doesn’t suck — that’s how bad it was. (Now that Unity runs on Windows their lives have gotten much easier, which is a pretty sad statement.)

Before you scream at me about how, say, Subethaedit has some kind of autocomplete support, or mention Coda (which has OK autocomplete support) or Espresso (which may one day have halfway decent extensible autocomplete support but right now is a work-in-progress), go and try Visual Studio or, if you’re allergic to Windows how about Realbasic. Realbasic‘s built-in text editor has autocomplete that doesn’t suck, e.g. it recognizes context, knows about intrinsic language constructs as well as library functions and the stuff you’ve declared, and doesn’t incorrectly complete stuff you need to fix constantly or fail to offer “function” when I type in “fun” in a .js file.

I will say this: TextMate’s macro facility is truly awesome (you can type swi->TAB and get a switch statement template, and then tab through the bits you need to change, almost like a context-sensitive-in-place-dialog-box), if this were paired with a proper autocomplete system (like RealBasic’s) it would be the best I’ve seen and then some — maybe 2.0 will have this — but right now the situation on the Mac remains pretty dismal.

Realbasic Alternatives

So, you want to develop apps quickly and easily but you are sick of Realbasic’s subscription model, or are annoyed by the forced switch to “Studio”. What to do? The latest news from Real Software has me thinking (again) about what I should do rather than continue to use Realbasic, and here are the alternatives I’ve come up with.

XCode — Free (Mac). On the downside, you can’t deploy on Windows (or Linux, if you care). On the upside, actually supports all Mac UI widgets properly, has far fewer stupid bugs, can actually load images with alpha channels, produces apps which you don’t need to be ashamed were created with Realbasic, allows you to deploy on the iPhone, and actually has pretty good tools for building web apps.

Cocotron looks to offer the Holy Grail of cross-platform development. Develop your apps in XCode and Cocoa and simply compile to Windows. I haven’t tried it yet, but it certainly seems intriguing and it appears to be under pretty active development.

Unity — $199 and up (Mac/Win as of 2.5, also iPhone/Wii). On the downside, doesn’t produce standard desktop apps. On the upside, very good for game and multimedia development (far better than Realbasic); generally superior performance to Realbasic; your programs can run in web browsers, the iPhone, and even the Wii; one license allows you to code on both Mac and Windows; actually has a superior GUI for OO development than Realbasic (once you get your head around it); supports three languages (Boo, C#, and JavaScript), each of which is an easy move for RB coders; no subscription model.

BlitzMax — $80 (Mac/Win/Linux). Very fast, modern BASIC with full cross-platform GUI support (available as a $30 add-on). Designed with 2D game development in mind, but perfectly capable of being used for app development. Downside: bare-bones IDE which does not include visual GUI tools or handle bindings between UI elements, events, properties, and code. Visual GUI tools (which do do these things) are available from third parties.

Python — Free (Mac/Win/Linux). Python is not only a ridiculously nice language, it’s also hip and cool and highly marketable. It’s kind of like JavaScript without the negative associations (but also without the ability to run in Web browsers). For GUI development, Tkinter looks interesting and PythonCard actually seems pretty compelling.

Java — Mostly Free (Mac/Win/Linux). Well, Eclipse is pretty nice, and I assume that by now it’s probably possible to produce vaguely decent UIs. I’ll need to look into this. Java is definitely not my favorite language, but it’s very marketable.

Netbeans — Free (Mac/Win/Linux). Free and open source IDE and runtime that lets you code in Java, JavaScript, Ruby, PHP, C/C++, and (shortly) Python. OK that sounds too good to be true. (I downloaded 6.5 and messed with it a bit. It falls under the category “I guess it probably seems pretty neat if you think Solaris is pleasant to use”, so — yeah — too good to be true.)

Web-based — Mostly Free, and some amazingly good, cheap stuff (e.g. Coda). On the downside, you can’t deploy standalone desktop apps via the web. Oh wait, you can. And you are living with whatever functionality you get from browsers (i.e. JavaScript, canvas, Flash, etc.). On the upside, web apps are where the action is. And there’s always Cappuccino and Atlas.

Adobe AIR — Free? (Mac/Win/Linux). Essentially a runtime that bundles Webkit, Flash, and other functionality, allowing you to build web apps that run like applications (including being able to avoid the usual sandbox restrictions). Of course, you’re essentially trapped inside the functionality provided by Webkit (and Flash, if you choose to use it).

Flex Builder Standard 3 — $249 (Mac/Win). On the downside, produces non-standard (Flash-like) UIs. On the upside, your software runs inside browsers (OK, not on iPhones, but neither does Realbasic); you don’t pay a subscription, and Adobe will provide free bug-fixes even for outdated versions of its software. Also, Flash-like UIs are all the rage anyway, and at least you’ll have a consistent user experience on all platforms. Oh, and ActionScript 3 is not going to be hard to learn for Realbasic developers.

Runtime Revolution — $49/$249/$499 (Mac/Win/Linux). On the downside, produces non-standard (sometimes ugly) UIs, and the language is a bit outmoded (although nice in many ways). On the upside, there’s no subscription model.

Qt SDK — Free or Expensive (Mac/Win/Linux). Built on top of the well-known Qt UI library. On the down side, requires you to code in C++. On the upside, produces robust, cross-platform apps. Builds skills that get you better paid jobs than RB experience. The free version is only useful for producing free apps, but that’s a lot of what I do with Realbasic. Correction: the free Qt version can be used for proprietary apps. (And frankly, no-one cares if you open-source a RB Project.)

Lazarus — Free (Mac/Win/Linux). Very interesting looking open source recreation of Delphi. If it works it could be fabulous — I love Object Pascal (although it’s hardly a popular language these days). It appears to let you compile both native Cocoa and X11 apps on OS X.

It’s worth noting that it’s not easy to replace Realbasic for cross-platform development. I can whip up a cross-platform utility with a native UI in Realbasic with almost ridiculous ease, and that’s simply not true for any of these options.

I’ll probably end up keeping an active Realbasic license for as long as I make money from contract programming with it. But, I’ll be moving all the projects I can move, along with any new projects, away from Realbasic ASAP. RiddleMeThis, for example, may well be rewritten using web technology, with desktop deployment via Adobe AIR or something similar.

Other Useful Links

Wikipedia’s compilation of RAD tools

Microsoft’s “Express” (i.e. free) development tools

Apple’s Accidental Game Strategy

When Apple released the Macintosh its primary focus was on competing with the IBM PC as a “serious” computer. The Apple II was widely seen as being a “games machine”. It shipped with game paddles, and pretty much every classic computer game started out as an Apple II game. Ultima I-IV, Wizardry, Sierra’s first titles, EA’s first titles, Broderbund’s first titles… all started life on the Apple II.

Macs are not games machines, dammit

For computer games enthusiasts, the Mac was extremely frustrating. It had excellent graphics, but its CPU really wasn’t up to updating all those pixels really fast. And on top of that there were no decent native development tools for the first couple of years. Worst of all, Apple refused to provide standardized game controller specifications or even to allow games to go full screen without using evil hacks.

Apple’s anti-game bias was probably a result of Steve Jobs’s desire to have Macs taken seriously by business, but after he was forced out, this policy was continued mindlessly for over a decade. It’s highly unlikely that Jobs would have persisted with such a self-destructive policy had he remained in charge of Apple. (There are persistent rumors that Jobs hates games. Maybe so… even though he and Woz created games together. But he’s enough of a businessman not to piss in the hand that feeds him.)

But if we pretend really hard…

Since about 1993, Apple has periodically tried to recapture its lost opportunity in games, notably by employing games “evangelists” who tried to convince developers that Apple was serious about games and convince Apple to be serious about games, the introduction of “Game Sprockets” (libraries of code to make life easier for game developers, something Microsoft arguably imitated with DirectX, and which Apple spectacularly failed to follow through on, eventually abandoning the whole concept in the switchover to MacOS X), the ill-fated and underpowered Pippin games console, and, more recently, the highlighting of games in Mac marketing events and literature (starting with Steve Jobs bringing the Bungie folks on stage to show off early alpha code of HALO, shortly before they were bought by Microsoft and HALO was turned into a flagship launch title for the X-Box).

Accidental Victory

I don’t know if it’s an accident or not, but the iPhone turns out to be a very nice piece of game hardware with a very odd game control interface. Fortunately for Apple, the Nintendo Wii and DS have whetted people’s appetites for games which use weird control interfaces. As for the raw specifications, compared to the Sony PSP, it has more memory, a far better CPU, and a somewhat inferior GPU. Compared to the Nintendo DS it is grotesquely superior in every way except battery life. Unlike the original Mac, it can push pixels as fast as needed for games, and it has excellent developer tools that Apple gives away to anyone interested. Anyone complaining about the iPhone being a closed platform should try to get a PSP or DS dev kit.

So, suddenly, Apple has a game platform that is dominant in its obvious category (smart phones) and a serious contender in its less obvious category (handheld games consoles). Certainly in terms of available titles, the iPhone has more games available for it now than the PSP, DS, and all earlier Gameboys combined. Most iPhone games suck ass, but guess what, so do most PSP, DS, and Gameboy titles, and they’re way cheaper. What’s more, Apple can sell a $5 or $10 title direct to a consumer, collect 30% and pass on 70% — no middlemen involved. The developer of a $10 iPhone game probably gets more money per sale than the developer of a $29 DS title. (Last I heard, the developer of a AAA $40 PC game title nets about $14 per game sold at full price, but consoles take more money off the top, especially titles that ship on cartridges.)

This is all pretty cool, but there’s more: 

 

  • There’s no reason why iPhone games can’t run with a simple recompile on a Mac. Possibly without a recompile (via CPU emulation). iPhones are, after all, Macs running on an ARM CPU. We should start to see games developed for both platforms simply because it’s easy to do.
  • There’s no reason why iPhone games can’t run with a simple recompile on an AppleTV. AppleTV’s are, after all, Macs. It follows that Apple could reposition the AppleTV as a games platform with a system update.
  • The Unity game development system is going to be able to target iPhones as of this month. (It’s currently in beta.) The development process isn’t quite seamless with Windows/Web/Mac/Wii development, but there’s no reason not to expect to be able to target Windows/Web/Mac/Wii/iPhone from a single codebase in Unity within six months.

 

So, Apple could be in the process of becoming a serious contender in the game platform wars almost accidentally as a result of its success with the iPhone. And Apple has done this without flushing a ton of money down the toilet the way Microsoft has with both the XBox and XBox 360. And, unlike Sony, it doesn’t need to offset hardware losses with software profits — it made money from the iPhone from day one.