DOM: Analysis of failed tests

Nearby: less-than-2, complete-fails, all results,

Introduction

Historically, W3C test suites have altogether too often been subpar in their coverage, barely scraping the surface of interoperability. In such a context, with only a few hundred simple tests, it is naturally expected that one would expect to hit the 100% pass mark.

With the increased focus on testing however, notably through the Web Platform Tests and Test The Web Forward projects the quality of test suites has improved massively. And since we don't live in a perfect world, if one looks for trouble hard enough one will find it.

The DOM test suite, which is over 52,000 tests (and growing), certainly did set out to look for trouble. Given that, it seems useful to accompany the test results with this this note explaining where the failures come from.

In total, 743 tests fail to pass in two implementations, which amounts to 1.43% of the total. The sections below provide context in which to appraise these failures. This is an increase compared to the last time the DOM test results were reported upon; it is due to the test suite expanding in scope, counteracting actual improvements in implementations.

createNodeIterator()

555 of the failures (75%) come from this single method. This method is actually already in DOM Level 2, but its interoperability in browsers has only recently started improving.

Given that this is existing functionality, already Recommended, that mostly needs to be improved upon, the volume of failures is not particularly worrying.

createDocument()

81 of the failures (11%) come from this method. The failures are actually mostly for fields that need to be set or not to specific values on the created Document object.

While it would be useful to sort these problems out, the interoperability issues around this topic are well-known and there has been little interest from implementers in resolving them. The specification features explicit warnings about these.

WebIDL

43 of the failures (6%) or less are linked to WebIDL-related problems. Essentially, they are attributes being present on instances instead of prototypes, wrong exception types, and other similar issues.

This is marked improvement on previous reports in which WebIDL heavily dominated the interoperability landscape. While still not perfect, this area is moving in the right direction, and the failures are unlikely to impact regular development much.

Miscellaneous

The remaining 64 failing test cases do not appear to subject themselves to easy categorisation. Many of these are corner cases that have little impact on everyday development, if any.

Conclusion

Considering that:

  1. the volume of failures is really low to start with (we have over 98% success over 52,000 tests);
  2. the vast majority of failures are circumscribed to limited areas with well-known issues;
  3. the percentage of issues that point to problems that could affect developers is so vanishingly small as to be negligible and none of the remaining tests indicates a failure in interoperability;

We contend that the specification being proposed for advancement through the process has demonstrated interoperability.