BT

Your opinion matters! Please fill in the InfoQ Survey!

Google Retires Octane JavaScript Benchmark

| by David Iffland Follow 3 Followers on Apr 26, 2017. Estimated reading time: 2 minutes |

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

Google has announced that they are retiring support for their Octane JavaScript benchmark, citing lack of real-world benefits; there are also signs that a big change is in the works for Chrome.

In a blog post, Google says that the benchmark was useful early in its life, but that it has become outdated since the benchmark doesn't reflect how web pages are built today.

Octane helped engine developers deliver optimizations that allowed computationally-heavy applications to reach speeds that made JavaScript a viable alternative to C++ or Java.

By 2015, however, most JavaScript implementations had implemented the compiler optimizations needed to achieve high scores on Octane. In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.

The cynic's view of this news is that they were getting beat by competitors on their own benchmark. On a marketing page, Microsoft claims Edge is faster than Chrome on the Octane 2.0 score, explicitly saying "Google's own performance benchmark shows that Microsoft Edge is faster than Chrome and Firefox."

For their part, however, Microsoft seems to agree that the synthetic benchmark doesn't reflect the real world:

We are still frequently asked about JavaScript benchmark performance, and while it doesn’t always correspond directly to real-world performance, it can be useful at a high level and to illustrate improvement over time.

Brian Terlson, program manager for Microsoft's Chakra JavaScript engine, wrote in a comment on Hacker News that "Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one, e.g. when we realized having an interpreter really helps startups on lots of real-world apps but doesn't do much for Octane score)".

It's clear that the browser vendors feel that they've tackled most of the big items on the JavaScript performance checklist. With features such as WebAssembly, performance improvements may come to us differently than they have for the past few years. What's more important now is improving what the user is actually experiencing, so a new class of benchmarking tools is required.

Multiple signs point to Speedometer as a good choice for now. Jeff Atwood has endorsed it, and a blog post from the Chromium team said the same thing. Speedometer uses the popular TodoMVC sample to exercise the browser and measure its performance. Browser Bench also has a browser graphics test (Motion Mark) and a JavaScript benchmark (JetStream).

The Chromium post also says that they plan on improving JavaScript performance based on modern JavaScript patterns and teased that something big is in the works:

Stay tuned for updates about our new engine architecture, designed for real-world performance.

Octane is still available on GitHub and developers can continue using it.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT