W3C Testing How To

https://w3c.github.io/testing-how-to/

Content

  1. What do you test in a specification?
  2. How to write a test?
  3. W3C Test repositories
  4. How to add a test suite into the W3C framework?
  5. Licenses For W3C Test Suites
  6. Strategies for testing a specification

What do you test in a specification?

  1. Look for conformance requirements in the spec
  2. Some statements are unambiguously requirements
  3. Some statements are candidate requirements
  4. Requirements are often stated within algorithms
  5. A single requirement usually requires multiple test
  6. A single test case may test a combination of requirements
  7. In short, it’s complicated :-)

Look for conformance requirements in the spec

In the simplest case, a conformance requirement is any single statement that uses RFC 2119 language (MUST, MUST NOT, SHOULD, SHOULD NOT, MAY), that specifies required behavior for a particular conformance class defined elsewhere in the spec (e.g., a Web browser or other user agent). However, some conformance requirements may not use RFC 2119 language.

Here’s a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

Some statements are unambiguously requirements

Here again is a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

That paragraph contains three statements that use RFC 2119 “must” language and so are unambiguously requirements that need testing:

  1. On getting, it must return its current value.
  2. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value.
  3. Other values must be ignored.

Some statements are candidate requirements

Here again is a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

That paragraph contains one statement that may or may not actually be a requirement that needs testing:

When the DataTransfer object is created, the dropEffect attribute is set to a string value.

Although that statement does not use RFC 2119 language, it still states something that could possibly be tested. Determining whether or not it should be tested is something that requires reading other parts of the spec. For this particular example, no test is necessary for the statement above, because the actual requirement for setting the value of the dropEffect when the DataTransfer object is created is defined in another part of the spec.

Requirements are often stated within algorithms

Some algorithms in a spec may contain many individual statements or steps that don’t use each use RFC 2119 but that nonetheless state conformance requirements. Such algorithms are typically prefaced by a statement that does use RFC 2119 language.

Here’s an excerpt from an algorithm in the HTML5 spec:

When a user agent is required to fire a DND event named e at an element, using a particular drag data store, the user agent must run the following steps:

  1. Let dataTransfer be a newly created DataTransfer object associated with the given drag data store.

  2. Set the effectAllowed attribute to the drag data store's drag data store allowed effects state.

  3. Set the dropEffect attribute to "none" if e is dragstart, drag, or dragleave

Notice that the algorithm is prefaced by a statement saying, “the user agent must run the following steps”.

A single requirement usually needs multiple tests

There usually isn’t a one-to-one relationship between a conformance requirement in a spec and the number of test cases needed to test that requirement adequately. Instead, each requirement typically needs multiple test cases. For example, the following statement from the HTML5 spec would need at least four test cases—one for each of the possible values given.

On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value.

That requirement likely in fact would actually need even more test cases. Determining how to test it adequately would require reading other parts of the spec.

A test case may test a combination of statements

It’s always preferable for a test case to focus as narrowly as possible on isolating a test for one specific case (though that case will almost always have dependencies on other major features: the parsing algorithm, the entire DOM, etc.) But often it’s not that simple to narrow it down that specifically. For example, in the case of testing complex algorithms, a single test case may necessarily be testing a combination of requirements from different parts of the algorithm.

In short, it’s complicated :-)

For any particular spec, determining things like which statements in the spec require testing and how many test cases are needed to test them is something that requires a very close reading of the spec and very close familiarity with it.

Different spec editors take different approaches—and though the style and conventions of W3C specs for browser technologies have “normalized” and improved considerably over the years in terms of the degree of rigor and precision and lack of ambiguity with which the specs state requirements (for example, using RFC 2119 language consistently to clearly identify requirements), isolating exactly what to test and how to test it necessitates first spending a lot of time reading the spec.

Test the Web Forward mascots: dinosaurs.

Test the Web Forward

W3C's one stop shop for Open Web Platform testing.

Why follow Test the Web Forward?

  1. They know how to test the Web
  2. They want the Web to work, no exceptions
  3. A growing number of browser vendors are running the TTWF tests
  4. They have a lot of test infrastructure ready to use
  5. Your lifespan is too short to reinvent it
  6. They have happy and cool dinosaurs on their landing page

How to write a test?

Start with Documentation

  1. Script Test
  2. Reftests
  3. Self-Describing Test
  4. WebIDL Test
  5. Client-Server Test

See Key Aspects of a Well Designed Test.

Script Test

testharness.js: When the result of your test could be determined by a script.

<script
  src="/resources/testharness.js">
</script>
<script
  src="/resources/testharnessreport.js">
</script>

testharness.js

<head>
<title>Document.title exists</title>
</head>
<body>
<div id='log'></div>
<script>
test(
 function() {
  assert_equals(document.title, "Document.title exists");
 },
 "Document.title exists"
);
</script>
</body>

See Demo

testharness.js: assertions

assert_true, assert_false, assert_equals, assert_not_equals, assert_approx_equals

assert_in_array, assert_array_equals, assert_regexp_equals

assert_own_property, assert_inherits, assert_idl_attribute, assert_readonly, assert_throws, assert_unreached, assert_any

See the full list of assertions with their documentation

asynchronous testing

Intended for:

testharness.js: async test

var t = async_test("Load event fires");

window.onload = function(e) {
  t.step(
    function () {
      assert_equals(e.type, "load");
    }
  );
  t.done();
};

See also:

testharness.js: promise test

promise_test(function() {
  return Promise.resolve("foo")
    .then(function(result) {
      assert_equals(result, "foo", "foo should return 'foo'");
    });
}, "Simple example");

See also Promise Tests documentation

testharness.js: metadata

In the <head> element:

<title>title exists</title>
<link rel="author"
 title="John Doe" href="mailto:john.doe@example.com"/>

testharness.js

testharness.js contains additional functionalities:

testharness.js: Further reading

Reftest

When the result of your test is visual.

bdo-000.html bdo-000-ref.html
HEBREW WERBEH
<bdo dir="rtl">HEBREW</bdo>
<span>WERBEH</span>

reftest: metadata

As for testharness.js, put it in the <head> element:

<title>rtl value on dir attribute on bdo element</title>
<link rel="help"
 href="http://www.w3.org/TR/html5/global-attributes.html#attr-dir-rtl"/>
<link rel="help"
 href="http://www.w3.org/TR/html5/the-bdo-element.html#the-bdo-element"/>
<meta name="assert"
 content="The dir attribute on bdo element could invert the text direction"/>
<link rel="author"
 title="John Doe" href="mailto:john.doe@example.com"/>

Reftest: further readings

Manual test :(

When everything else wouldn't work

visibilitychange

Iconify this window and restore it.

If the square is still red, please indicate that the test has failed.

See also Demo

WebIDL test

Test your Web IDL interfaces!

<script
  src="/resources/testharness.js">
</script>
<script
  src="/resources/testharnessreport.js">
</script>
<script src="/resources/WebIDLParser.js"></script>
<script src="/resources/idlharness.js"></script>

WebIDL test

<pre id='idl'>
partial interface Document {
  readonly attribute boolean hidden;
  readonly attribute DOMString visibilityState;
};</pre>
<script>
var idl = new IdlArray();
idl.add_idls(document.getElementById("idl").textContent);
idl.add_objects({Document: ["window.document"]});
idl.test();
</script>

Client-Server test

When you need server-side code as well

We have several examples:

  1. XMLHttpRequest
  2. Web Sockets

wptserve to the rescue!

wptserve

A server designed for writing tests for the web platform.

wptserve: built-in pipes

example.txt?pipe=slice(10,20)|status(403)

Running the test suite

Test suite results

Short summary: we have some ideas there but, in the meantime, you MAY do as follows:

Licenses For W3C Test Suites

W3C makes test suites available under two distinct licenses:

  1. a W3C Test Suite License for an Authoritative W3C Test Suite or when claims of performance with respect to a specification are required.
  2. a 3-clause BSD License for other applications.

This is builtin in w3c/web-platform-tests so no action needed on your part.

web-platform-tests, aka WPT

w3c-test.org

Strategies for testing a specification

For more information