Microsoft’s HTML5 Compliance Test Results Are Disputed by Google, Mozilla, and Opera [UPDATED]
Microsoft has posted the results for 192 tests grouped in 8 categories for HTML5, SVG 1.1, CSS3, and DOM Level 2&3 showing that IE9 Preview passes all of them with flying colors while Chrome, Firefox, Opera, and Safari have mixed results varying from 0% to 100% depending on the category. The conclusion, that IE9 is the most compliant with W3C standards, is contested by Google, Mozilla, Opera.
Microsoft has created 192 “test pages that we [Microsoft] developed in conjunction with the World Wide Web Consortium (W3C) working groups” and the results of those tests for the major browsers are:
The results summary is followed by detailed results with links to web pages containing the respective tests, so anyone could perform the respective tests. The conclusion is: IE 9 Preview is 100% compliant with W3C standards covered by tests while the other browsers lag behind. The best result for the other browsers is for Firefox with 100% for DOM 2 tests, but most of the browsers show results in yellow and red for many categories. InfoQ has talked to Google, Mozilla, and Opera to find out what is their opinion on tests, results and conclusion.
Google has pointed to Google I/O Keynote Day 1 where they covered the issue of HTML5 in browsers and their expectancy for the end of 2010 is for all browsers except IE to fully implement the HTML5 standard and some other specs (9min:35secs in the keynote):
Mozilla considers Microsoft’s tests as having limited coverage, being inaccurate and even misleading:
While we are glad that Microsoft is keen on using test suites to measure standards compliance, these particular tests on MSDN only cover a very limited spectrum of capabilities, and statements about browser compliance made here are frankly misleading.
The SVG-in-HTML tests cited by MSDN are known to be broken, do not conform to the HTML5 parsing rules or the DOM specification, and in fact make use of an IE bug. Microsoft's engineers know about this bug *and* the fact that the test itself is wrong , so it is surprising that this test is touted as an important one in furthering interoperability between browsers. Firefox's beta builds, on the other hand, actually do have an HTML5 parser in place , and nightly builds  will pass legitimate tests for SVG-in-HTML parsing.
Additionally, some of the CSS tests listed on MSDN are inaccurate, and browsers that actually pass these tests are not compliant with the standard. There is an official CSS3-selectors test suite where we pass all the relevant pieces  and we think that for web developers, sites like caniuse.com  are much more useful when discussing HTML5, CSS3, and SVG support in browsers. Mozilla will continue to contribute to open and fair test suites that adhere to the specifications in question , and encourage other browser vendors to do so as well.
Update: Mozilla has updated their statement with the following short but meaningful piece of information:
We've heard back from Microsoft and are now working with them to fix the tests.
Opera is aware of Microsoft’s tests and results, but they are using a different test suite having different results:
We haven't run these specific tests internally, but we do run a broad gamut of tests all the time to ensure that our contribution to the HTML5 standard does get implemented in our browser products. Opera is a big contributor to the HTML5 standard, it was after all Opera who proposed the Video tag that got the ball rolling on the standard. We constantly strive to improve our products and make sure the follow the standards as best possible.
This test is very selective in what elements it chooses to test for, and the results seem to favor one over the others. When evaluating HTML5 tests, the criteria should be the same as for all other kinds of tests. The test authority should be independent and the tests should reflect the nature of the specification. It's easy to point to other HTML5-relates tests where Opera do much better than IE like this one: http://www.codedread.com/svg-support.php. …
We run tests all the time to see how compatible our browser is, but HTML5 is a work in progress. So regarding the web pages of Microsoft, they seem to test a very selective part of HTML5 while leaving out large parts like the Canvas element.
Interestingly, Opera pointed to test results showing that the Opera 10.53 browser implements 94.89% of the SVG 1.1 test, while IE9 Preview 2 implements only 30.55% of the same test. Opera 10.52 does about the same in Microsoft’s test, but IE9 has 100% in that test which is far away from 30.55%.
Jeff Schiller, the software developer who ran the tests mentioned by Opera, commented on Microsoft’s test results:
While I am very excited about the direction IE9 is taking and have been impressed with the level of participation in the SVG Working Group, I have a problem with the “IE Testing Center” as I feel it is very misleading. Even Microsoft admits that they are only including tests that they wrote (for which they claim IE9 to pass) not the entire test suite.
For instance: the SVG Test Suite has 275 tests. Microsoft only talks about 31 tests on that page. As an example, IE9 does not even support gradients yet.
As another example, their HTML5 tests that talk about inline SVG markup are actually incorrect. They only pass in IE9 due to a browser bug dealing with childNodes. If I correct those tests they pass in Firefox nightlies and fail in IE9.
A test suite where IE fails all tests
From that page:
Based on the tests that we have performed, it is very clear that there is a very big difference between the best and worst browsers. Therefore we can only conclude that the results are valid and true.
Re: A test suite where IE fails all tests
I was aware of that paragraph but I did not include it because it is followed by this one:
Update June 6: Support the IE9 Canvas Campaign. The results in this test are mostly imaginary to get the point across. Choose your next web browser carefully.
Re: A test suite where IE fails all tests
My Chrome build scored 232
"C:\Users\Your-User-name\AppData\Local\Google\Chrome\Application\chrome.exe" --enable-webgl --no-sandbox --enable-fastback --enable-indexed-database --enable-gpu-plugin --enable-gpu-rendering --enable-native-web-workers --enable-preparsed-js-caching
All that gave me a score of 232 on html5test.com but also running WebGL tests located at khronos.org/webgl/wiki/Demo_Repository enjoy :)