BT

Latest Dart VM Beats JVM in DeltaBlue Benchmark

by Abel Avram on May 11, 2013 |

Shortly after Dart2js outperformed hand-written JavaScript, the latest Dart VM now beats JVM 7 64-bit Server, and other versions, in the DeltaBlue benchmark.

Nikolay Botev has ported the DeltaBlue benchmark from Dart to Java and run the benchmark against Dart-22416 (latest stable release) and Dart-22577 (a nightly build), and against several Java client versions, posting the following results on his blog:

image

The fist column contains the time in us needed to run one iteration of the test while the second column represents the number of runs/sec, being the DeltaBlue score charted above.

We have also run the same tests on a 8-core AMD FX-8350 machine with SSD, but this time against JVM 7 64-bit Server, getting the following average results after 10 runs of each test:

VM Average Time (us) Score
Java HotSpot(TM) 64-Bit Server VM 1.7.0_21 23.21-b01 1808.18 553.04
Dart-22416 1885.02 530.5
Dart-22610 1610.48 620.93

image

In our test, Dart VM shows 12% better performance than the JVM, which is consistent with Botev's findings.

Greg Lowe posted his results on running DeltaBlue on JVM and Dart VM a couple of months ago, but his benchmark uses Sun’s original DeltaBlue code written in Java which includes a warm-up time:

image

Lowe’s results show better performance for Dart up to 1,000 iterations of the test, while Java performs better after 10,000 iterations. In the light of Botev’s latest tests, which uses a newer Dart VM, Lowe commented on a Dart group:

I had a quick look at this. It appears the DartVM is now beating the JVM even at 100,000 iterations (Previously Java edged out Dart at higher iterations).

100,000 iterations takes ~45 seconds to run on my computer.

At a lower number of iterations the DartVM beats the JVM by a significant margin.

While we cannot draw final conclusions based on a single benchmark, we can safely say that Google has enhanced Dart VM’s performance to the point where it becomes a serious contender.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Shocker! by Cameron Purdy

I tried following some of the links to learn more, but it took me to random unrelated posts on some Google site that kind of looks like an lame copy of facebook.

But let me see if I got this right: Some Google guy working on Dart at Google has ported a benchmark made for Dart to Java, and his results show that Dart is faster?

Peace,

Cameron Purdy | Oracle
The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.

Re: Shocker! by Tom M

Re: Shocker! by Abel Avram

Hi Cameron,
sorry for the wrong link. Here is the DeltaBlue link again: www.cs.washington.edu/research/constraints/delt...


But let me see if I got this right: Some Google guy working on Dart at Google ...



Who's working for Google?

... has ported a benchmark made for Dart to Java, and his results show that Dart is faster?


DeltaBlue was not made for Dart. It's a one-way constraint solver focused on polymorphism and OOP, originally written in Smalltalk, Lisp and C. And Lowe, which I quoted in the article, used Sun's version of the benchmark, not the one ported from Dart.
I understand if there is criticism regarding the benchmark, but it has to be specific. That would help creating and running a better benchmark with more accurate results. Saying that someone works for Google is not enough to refute the results.

Re: Shocker! by Faisal Waris

The gene pool available for VM optimization is very small.

The same people have been at it since Smalltalk days (e.g. Lars Bak).

I would not be surprised if Dart performance ends up being in the same ballpark as other established VMs.

Re: Shocker! by Sean Hsien

Yes, what was that famous quote about statistics and lies?:)

Here are 2 links for more info:

www.reddit.com/r/programming/comments/1e2jhr/da...

benchmarksgame.alioth.debian.org/u64q/benchmark...

Well informed readers should make up their own minds.

Cheers,
Sean

Re: Shocker! by Cameron Purdy

Thanks for the link. I'll check it out.

(The assumption on "working for Google" isn't any entirely unreasonable one, since the links were to Google+, and in general, only Google employees use Google+. Sounds like I got that wrong.)

Anyhow, I was more frustrated by not being able to follow the links and learn more of the story, than I was about the results that were posted. It just seemed a bit squirrely.

Peace,

Cameron Purdy | Oracle
The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.

Re: Shocker! by Abel Avram

and in general, only Google employees use Google+.

Good joke :) There are some 400 M people on G+.

Re: Shocker! by arnaud m

I wonder how the Dart VM could always compensate for the lack of static type info at runtime, on call sites ("in production mode, type annotations are ignored"), even with dynamic code execution analysis.
?

Re: Shocker! by Faisal Waris

The compiler is separate from the runtime. The runtime can use dynamic type info for optimization.

Very few people know this but the Hotspot JVM is based on Strongtalk - an optionally typed Smalltalk, very similar to Dart.

See: www.strongtalk.org/

If you look at the history page, some of the same people - Lars Bak, Gilad Bracha - are also behind Dart.

Has DeltaBlue been ported to golang? by Dean Schulze

It would be great to benchmark Dart and Java against golang.

Re: Shocker! by Dean Schulze

Are you saying that the Dart language is similar to Smalltalk? I thought it was similar to Java or C#, which if I'm not mistaken are quite different from Smalltalk (I've never written a line of Smalltalk).

Here's the history of HotSpot and StrongTalk:

en.wikipedia.org/wiki/HotSpot

Not only DeltaBlue by Tomasz Wysocki

www.dartlang.org/performance/

Also tracer benchmark shows +26% performance improvement on 22480:22488 which correlates nicely to
r22485 | vegorov@google.com | 2013-05-08 01:40:42 +0200 (Wed, 08 May 2013)

codereview.chromium.org//14935005
wiki.luajit.org/Allocation-Sinking-Optimization

Polimorphic inlining by Tomasz Wysocki

After some more reasearch I came to conclusion that deltablue benchmark benefited more from something called "polimorphic inlining" (+20% performance improvement)

r22434 | kmillikin@google.com | 2013-05-06 19:22:18 +0200 (Mon, 06 May 2013) | 13 lines
Initial support for polymorphic inlining.
codereview.chromium.org//14740005

Re: Allocation sinking by Andrew McVeigh

Probable cause?

codereview.chromium.org//14935005
wiki.luajit.org/Allocation-Sinking-Optimization


Luajit is astonishingly good. I benchmarked the spectral norm computer shootout this morning, limiting both Java and Lua to a single thread. Java took 20s, Luajit 18s. Looking through and playing with Luajit has convinced me that we will actually arrive at a time when dynamically typed languages are faster than statically typed languages because the backend can infer types and optimizations better than humans. I know the smalltalk people predicted this back in the late 80's, but it looks to finally be possible...

Re: Allocation sinking by Osvaldo Doederlein

Looking through and playing with Luajit has convinced me that we will actually arrive at a time when dynamically typed languages are faster than statically typed languages because the backend can infer types and optimizations better than humans.


There's no reason to believe that -- static-typed languages can benefit from the same optimizations, and a JIT compiler doesn't need to infer any information that was provided by the programmer for free :) though obviously, in cases where the programmer has chosen a sub-optimal type, a dynamic-typed language with a good optimizer can do better.

Static vs. dynamic typing is not the only trait of languages that impact performance; Java is static-typed (and also class-based and other nice things from the compiler's POV... most shared by Dart!), but it certainly includes its fair share of performance screwups: overly-strict FP math, no value types, erasure-based generics, etc. This also allows Java to lose in some benchmarks where dynamic languages give the JIT compiler more freedom; for example an ArrayList<Integer> in Java will always be backed by a dog-slow array of references to overweight boxed objects, while similar JS or Dart code could benefit from a VM that's smart enough to notice that all elements are integers and compile that to a dense array of in-place, unboxed ints. But then again, you cannot blame static typing for the problem, in this specific example a language like C++ or C# would produce optimal code, and would do that every time without depending on optimizers.</integer>

Re: Allocation sinking by Faisal Waris

Idiomatic java code is actually very 'dynamic' because of heavy use of interface types (which are mapped to concrete types at runtime) so polymorphic inline caching is essential for performance.

There are languages such as StandardML and Haskell where the type system is much stronger than Java's and consequently can be optimized better at compile time as all types are known upfront. Also because of type inference the type system does not seem like a straight jacket.

MLTon is a 'whole program optimizing complier' (www.mlton.org/) for StandardML that can produce very small and fast executables.

Re: Allocation sinking by Andrew McVeigh

...and a JIT compiler doesn't need to infer any information that was provided by the programmer for free :)


well, sure ;-P

let me rephrase my comment: "Luajit has convinced me that we will actually arrive at a time when ... high performance runtime techniques will find static typing & type hinting useless, and even worse a hindrance (because they require the runtime to undo the work that the programmer had to do to work within the type system)"

Re: Allocation sinking by remco greve

What about the simple fact that a compiler relatively has a lot of time to do optimization and does it in a way that is mathematically correct so to speak where a dynamic language does it heuristically and it also takes time normally spent on runtime. I think you can even say that any optimization found by a jit compiler that is better than a optimization found by a compiler at compile time probably introduces a bug.

Re: Allocation sinking by Faisal Waris

There is a movement back toward native code - Objective-C for Apple and C++'11 for Microsoft - to better support small portable devices (smartphones, tablets, wearables, automotive infotainment systems).

Native code is better for memory consumption, power consumption and cold-start responsiveness than JIT compiled code.

The issue is that none of promoted languages are very appealing for general application development.

There are perfectly acceptable alternatives such as OCaml which can be compiled via MLTon to executables approaching C-like size and speed. Sadly these alternatives remain underdeveloped.

Re: Allocation sinking by Andrew McVeigh

I think you can even say that any optimization found by a jit compiler that is better than a optimization found by a compiler at compile time probably introduces a bug.




I don't think that's true. There is far more information available during execution than available purely with a static analysis of a program. That information can be used to selectively tune the system (a la Hotspot) at runtime.

Try turning Hotspot and the Java server runtime heuristics off if you want to see what i mean.

Re: Shocker! by Cameron Purdy

Good joke :) There are some 400 M people on G+.


No, there are not 400 million people "on G+". What a ridiculous claim.

p.s. I am not implying that I have never made ridiculous claims. However, I can still call "bull sh1t" when someone else makes a ridiculous claim.

Peace,

Cameron Purdy | Oracle
The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.

Re: Shocker! by Abel Avram

CNET, Sept. 2012: “Google said today that 100 million people are using Google+ each month, a significant milestone for the year-old social network. The company said it now has 400 million total users.”


GlobalWebIndex, Jan 2013: G+: 343M active users, defined as “Used or contributed to in the past month”


Do you have more accurate numbers than that?

my simple revisit to this benchmark by Mingjian Jin

my result: Java 8 is ~15% faster than Dart nightly in my DeltaBlue testing(poorman's porting)! The key here is, the author of benchmark still compares apples and oranges rather than “1-to-1 port”. see more here if you are interesting: jinmingjian.github.io/2013/05/15/simple-revisit...

Re: Shocker! by Werner Schuster

"used or contributed" is weasel-speak for "was logged into their GMail account and tripped over one of the many Google+ links or buttons Google places all over the web and their other products".

Or how about this: if I use Google Hangouts to talk to someone, does that make me a Google+ user?

All those millions of users are mainly the result of creative accounting (sentence may contain puns).

Re: Not only DeltaBlue by Tomasz Wysocki

Nikolay had also compared two other benchmarks on his blog bonovox.be/blog/?p=127 - the results for 32-bit vms are again better for dartvm

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

26 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT