Javascript Dick Waving FTW

Ars Technica’s rather gushy coverage of Squirrelfish Extreme (and the somewhat critical comments on that coverage) prompted me to do some of my own testing. Certainly, Squirrelfish Extreme keeps Webkit (and, ultimately, Safari) among the contenders for best endowed web browser, but my results were a little less definitive.

Google’s V8 benchmark suite is here. The Sunspider benchmark suite is here. Domaeo (the Mozilla test suite) is here. You might like to check them out in a few browsers. I just tried the current Safari release, the current Firefox release, and the Webkit Nightly Build. As I understand it, the Webkit speed claims are based on testing the JavaScript engines in isolation from the command line. Of course, very few folks run JavaScript this way…

The following are my very slipshod results obtained by firing up browsers and trying each benchmark once or twice. (I wasn’t careful about shutting stuff down, minimizing background activity, closing extra tabs, etc.)

Google’s V8 Benchmark Suite

This set of benchmarks takes less time to run and appears to be less thorough. (It certainly produces less immediately intelligible results… you just have to accept that big numbers are a good thing.)

Firefox 3.0.1 (OS X 10.5.5)

  • Score: 213
  • Richards: 226
  • DeltaBlue: 238
  • Crypto: 205
  • RayTrace: 162
  • EarleyBoyer: 245

Safari 3.1.2 (OS X 10.5.5)

  • Score: 218
  • Richards: 124
  • DeltaBlue: 173
  • Crypto: 187
  • RayTrace: 309
  • EarleyBoyer: 401

WebKit r36685 (OS X 10.5.5)

  • Score: 1312
  • Richards: 1317
  • DeltaBlue: 990
  • Crypto: 2553
  • RayTrace: 575
  • EarleyBoyer: 2035

This is just wonderful. (I tried to download a Firefox Nightly to check out Tracemonkey for myself but figuring out how took more than ten seconds, so I gave up. Here are some stats from Windows (note that these are running under VMWare Fusion, so it’s not fair to compare to the Mac figures directly).

Firefox 3.0.1 (Windows XP Professional/VMWare Fusion)

  • Score: 208
  • Richards: 207
  • DeltaBlue: 264
  • Crypto: 157
  • RayTrace: 180
  • EarleyBoyer: 251

Google Chrome (Windows XP Professional/VMWare Fusion)

  • Score: 2117
  • Richards: 2593
  • DeltaBlue: 2482
  • Crypto: 1692
  • RayTrace: 1456
  • EarleyBoyer: 2685

Internet Explorer 8.0 Beta 2

Note: IE8 interrupts each run with a “script is running slowly” dialog. I tried to click this off as quickly as I could but it definitely hurt the benchmark slightly. I ran it three times, dismissing the dialog as quickly as possible and took the best score.

  • Score: 58
  • Richards: 47
  • DeltaBlue: 52
  • Crypto: 50
  • RayTrace: 69
  • EarleyBoyer: 75


Sunspider’s results are really detailed, and I’m not going to post the whole things here, but I got the following bottom line results from a single run of each. In the Webkit case, this was with half a dozen tabs open (including this blog entry), while Chrome had two tabs open, one of which was displaying the results of its earlier benchmark (not very taxing, I expect).

Webkit r36685: 1614.6ms +/- 7.5% (running native under OS X)

Google Chrome: 1458.2ms +/- 0.4% (running under XP/VMWare; yes, faster is better)

Dromaeo Results

Again, the results are very detailed and also seem very real world (they use code from a lot of widely used JavaScript libraries such as ExtJS, Dojo, and Yahoo Widgets as part of the suite). You can see them for yourself here (Webkit) and here (Chrome). During the Chrome run, Chrome kind of seized up, whichwas probably Windows XP’s fault. So the overall result may be garbage, but if you look at the details you’ll find that V8 is clearly faster at some things and Squirrelfish at others. What this says to me is that both have quite a bit of headroom even yet.

Webkit r36685: 6652.60ms (running native under OSX)

Google Chrome: 10540.80ms (running under XP/VMWare)

So on both Google’s and Apple’s preferred benchmarks, I’m seeing better performance out of Google Chrome, but not by much. Dromaeo, which is a very DOM-intensive suite and also appears the most comprehensive, favors Webkit. (And, yes, IE8 is pitiful. No, I am not going to try Sunspider or Dromaeo on IE8.)

The really annoying thing is that Apple isn’t very aggressive about pushing advances in webkit out to the masses. Or, to put it another way, they use new versions of Safari to sell new versions of Mac OS X*. Chances are Squirrelfish Extreme‘s performance will be a key selling point of Snow Leopard when it ships, instead of a free upgrade to Safari, which is what it ought to be. In the mean time, we’ll probably need to use some other Webkit-based browser or Webkit Nightly, or simply use Firefox to enjoy better performance.

Note: as a comment — rather rudely — points out, Apple doesn’t charge for Safari upgrades. But it does hold new features back so it can announce them as though they were new OS features. (Edit: and, actually, it doesn’t support Safari on versions of Mac OS X before 10.4.x, so technically I was completely correct and the aptly named “bs” was full of bs.) A case in point is the debugging functionality added to Webkit over a year before it became available in Safari as part of Leopard. (And yes, you can get it for Tiger, but the marketing would have you believe it was a Leopard feature, and you simply can’t get it for Panther or earlier.)

And the really puzzling thing is that Microsoft seems to be happy to allow JavaScript performance to suck dog’s balls under IE8.

So, Javascript is fast now?

This site is the most interesting attempt I’ve found to seriously compare execution speed across languages. It should obviously be taken with more than a grain of salt since, apart from all the usual issues with benchmarks (artificiality, etc.), the quality of the code written in each language may vary greatly, as may the actual algorithms, and different algorithms may favor different languages, and different languages are used for different things and, obviously, benchmark comparisons try to force everything to do the same thing. Still, it is interesting.

Based on some quick spelunking, JavaScript (Spidermonkey) is comparable in speed to PHP or Ruby, perhaps a third as fast as Python, a fifth as fast as Lua, about a hundred times slower than Java, C# Mono, or Lisp, and a hundred and fifty or so times slower than C++.

What that says to me is that JavaScript should easily be able to get two or three times faster than Spidermonkey (because it has a lot in common with Python), but then it will get tough. It also speaks rather well of Lisp (which manages to be truly dynamic and still competitive with the big guns). Since it looks like Squirrelfish Extreme and V8 are both around twice as fast as Spidermonkey, and that if you took the best bits out of each from the detailed benchmarks and combined them you’d maybe get 50% faster again, it looks like we’re nearly out of low-hanging fruit in the JavaScript engine race.