Nearby: less-than-2, complete-fails, all results,
NOTE: this analysis is pretty final (2014-09-01).
Historically, W3C test suites have altogether too often been subpar in their coverage, barely scraping the surface of interoperability. In such a context, with only a few hundred simple tests, it is naturally expected that one would expect to hit the 100% pass mark.
With the more recent focus on testing however, notably through the Web Platform Tests and Test The Web Forward projects the quality of test suites has improved massively. And since we don't live in a perfect world, if one looks for trouble hard enough one will find it.
The HTML test suite, which is nearing 100,000 tests (and growing), certainly did set out to look for trouble. Given that, it seems useful to accompany the test results with this this note explaining where the failures come from.
In total, 2875 tests fail to pass in two implementations, which amounts to 2.93% of the total (down from 3.30% in May 2014). The sections below provide context in which to appraise these failures.
2437 of the failures (84%) or less are linked to WebIDL-related problems. Essentially, they are attributes being present on instances instead of prototypes, wrong exception types, and other similar issues.
These will get better over time as WebIDL compliance progresses. It is a slow process, but also one that does not threaten to break HTML (or other specifications depending on WebIDL).
These failures do not adversely affect regular everyday development. Removing them leaves us with just a 0.44% failure rate.
HTML5 introduces a number of newer features, and some of those have not entirely finished
stabilising yet. This concerns mostly forms (9.6%), the template element (4.9%), and
the new media elements, most of all <track> (2.4%).
That these are partially failing should not hide the fact that they otherwise overwhelmingly pass their tests. It is only to be expected that more recent additions to the stack have their darker corners for which not everyone is yet on the same page. These do not seem to be pointing at implementability issues (the features are by and large known to be usable already).
The haphazard manner in which HTML was for much of its lifetime developed means that there are some features that never really were interoperable. HTML5 takes great strides in the direction of fixing this, but while some core aspects have been the subject of driven alignment (e.g. parsing) defining the behaviour of historical features which have never been reliable naturally ends up entering an area of diminishing returns.
These are failures for features that will likely get fixed eventually but that aren't high on anyone's priority list at this point. Tests in this segment represent 2.2% of all failures.
The remaining 138 failing test cases (0.14% of total) do not appear to subject themselves to easy categorisation. Many of these are corner cases that have little impact on everyday development, if any.
Considering that:
We contend that the specification being proposed for advancement through the process has demonstrated interoperability.
HTML5 uses the interfaces File, FileList, and Blob, including the type attribute and the close method. The following tests were used to check implementations:
While not being fully intereoperable in the details, those 3 interfaces are supported by at least 2 browser implementations.
HTML5 uses the attributes innerHTML and outerHTML. The following tests were used to check implementations:
While not being fully intereoperable in the details, those 2 attributres have been around for a while.
HTML5 mentions the interface MessagePort from the Web Messaging specification but does not make use for it. The following test was used to check implementations:
While not being fully intereoperable in the details, this interface is supported by at least 2 browser implementations.
HTML5 mentions the interface URLUtils, href and protocol attritutes which are all exposed through the URL interface. The following test was used to check implementations: