Content

1. What do you test in a specification?
2. How to write a test?
3. W3C Test repositories
4. How to add a test suite into the W3C framework?
5. Licenses For W3C Test Suites
6. Strategies for testing a specification

What do you test in a specification?

1. Look for conformance requirements in the spec
2. Some statements are unambiguously requirements
3. Some statements are candidate requirements
4. Requirements are often stated within algorithms
5. A single requirement usually requires multiple test
6. A single test case may test a combination of requirements
7. In short, it’s complicated :-)

Look for conformance requirements in the spec

In the simplest case, a conformance requirement is any single statement that uses RFC 2119 language (MUST, MUST NOT, SHOULD, SHOULD NOT, MAY), that specifies required behavior for a particular conformance class defined elsewhere in the spec (e.g., a Web browser or other user agent). However, some conformance requirements may not use RFC 2119 language.

Here’s a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

Some statements are unambiguously requirements

Here again is a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

That paragraph contains three statements that use RFC 2119 “must” language and so are unambiguously requirements that need testing:

1. On getting, it must return its current value.
2. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value.
3. Other values must be ignored.

Some statements are candidate requirements

Here again is a paragraph from the HTML5 spec that contains several conformance requirements:

The dropEffect attribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When the DataTransfer object is created, the dropEffect attribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.

That paragraph contains one statement that may or may not actually be a requirement that needs testing:

When the DataTransfer object is created, the dropEffect attribute is set to a string value.

Although that statement does not use RFC 2119 language, it still states something that could possibly be tested. Determining whether or not it should be tested is something that requires reading other parts of the spec. For this particular example, no test is necessary for the statement above, because the actual requirement for setting the value of the dropEffect when the DataTransfer object is created is defined in another part of the spec.

Requirements are often stated within algorithms

Some algorithms in a spec may contain many individual statements or steps that don’t use each use RFC 2119 but that nonetheless state conformance requirements. Such algorithms are typically prefaced by a statement that does use RFC 2119 language.

Here’s an excerpt from an algorithm in the HTML5 spec:

When a user agent is required to fire a DND event named e at an element, using a particular drag data store, the user agent must run the following steps:

1. Let dataTransfer be a newly created DataTransfer object associated with the given drag data store.

2. Set the effectAllowed attribute to the drag data store's drag data store allowed effects state.

3. Set the dropEffect attribute to "none" if e is dragstart, drag, or dragleave…

Notice that the algorithm is prefaced by a statement saying, “the user agent must run the following steps”.

A single requirement usually needs multiple tests

There usually isn’t a one-to-one relationship between a conformance requirement in a spec and the number of test cases needed to test that requirement adequately. Instead, each requirement typically needs multiple test cases. For example, the following statement from the HTML5 spec would need at least four test cases—one for each of the possible values given.

On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value.

That requirement likely in fact would actually need even more test cases. Determining how to test it adequately would require reading other parts of the spec.

A test case may test a combination of statements

It’s always preferable for a test case to focus as narrowly as possible on isolating a test for one specific case (though that case will almost always have dependencies on other major features: the parsing algorithm, the entire DOM, etc.) But often it’s not that simple to narrow it down that specifically. For example, in the case of testing complex algorithms, a single test case may necessarily be testing a combination of requirements from different parts of the algorithm.

In short, it’s complicated :-)

For any particular spec, determining things like which statements in the spec require testing and how many test cases are needed to test them is something that requires a very close reading of the spec and very close familiarity with it.

Different spec editors take different approaches—and though the style and conventions of W3C specs for browser technologies have “normalized” and improved considerably over the years in terms of the degree of rigor and precision and lack of ambiguity with which the specs state requirements (for example, using RFC 2119 language consistently to clearly identify requirements), isolating exactly what to test and how to test it necessitates first spending a lot of time reading the spec.

Why follow Test the Web Forward?

1. They know how to test the Web
2. They want the Web to work, no exceptions
3. A growing number of browser vendors are running the TTWF tests
4. They have a lot of test infrastructure ready to use
5. Your lifespan is too short to reinvent it
6. They have happy and cool dinosaurs on their landing page

How to write a test?

Start with Documentation

1. Script Test
2. Reftests
3. Self-Describing Test
4. WebIDL Test
5. Client-Server Test

See Key Aspects of a Well Designed Test.

Script Test

testharness.js: When the result of your test could be determined by a script.

<script
  src="/resources/testharness.js">
</script>
<script
  src="/resources/testharnessreport.js">
</script>

testharness.js

<head>
<title>Document.title exists</title>
</head>
<body>
<div id='log'></div>
<script>
test(
 function() {
  assert_equals(document.title, "Document.title exists");
 },
 "Document.title exists"
);
</script>
</body>

See Demo

testharness.js: assertions

assert_true, assert_false, assert_equals, assert_not_equals, assert_approx_equals

assert_in_array, assert_array_equals, assert_regexp_equals

assert_own_property, assert_inherits, assert_idl_attribute, assert_readonly, assert_throws, assert_unreached, assert_any

See the full list of assertions with their documentation

asynchronous testing

Intended for:

• callback functions: requestAnimationFrame(callback), getCurrentPosition(successCallback), etc.
• event handlers: addEventListener("…", listener)
• Observers
• etc.

testharness.js: async test

var t = async_test("Load event fires");

window.onload = function(e) {
  t.step(
    function () {
      assert_equals(e.type, "load");
    }
  );
  t.done();
};

See also:

• Asynchronous Tests documentation
• EventWatcher: DOM events testing handled using Promises

testharness.js: promise test

promise_test(function() {
  return Promise.resolve("foo")
    .then(function(result) {
      assert_equals(result, "foo", "foo should return 'foo'");
    });
}, "Simple example");

See also Promise Tests documentation

testharness.js: metadata

In the <head> element:

<title>title exists</title>
<link rel="author"
 title="John Doe" href="mailto:john.doe@example.com"/>

testharness.js

testharness.js contains additional functionalities:

• setup: non-trivial test setup that may fail
• Timeouts: last resort if the test can't be reliable in some other way
• cleanup: avoid persistence beyond the test itself
• Single Page Tests: asynchronous testing with exactly one test
• standalone workers tests: for dedicated workers, shared workers and service workers

testharness.js: Further reading

• Start with testharness.js tutorial
• Read the documentation
• See tests to avoid

Reftest

When the result of your test is visual.

<bdo dir="rtl">HEBREW</bdo>

<span>WERBEH</span>

reftest: metadata

As for testharness.js, put it in the <head> element:

<title>rtl value on dir attribute on bdo element</title>
<link rel="help"
 href="http://www.w3.org/TR/html5/global-attributes.html#attr-dir-rtl"/>
<link rel="help"
 href="http://www.w3.org/TR/html5/the-bdo-element.html#the-bdo-element"/>
<meta name="assert"
 content="The dir attribute on bdo element could invert the text direction"/>
<link rel="author"
 title="John Doe" href="mailto:john.doe@example.com"/>

Reftest: further readings

• Writing Reftests

Manual test :(

When everything else wouldn't work

WebIDL test

Test your Web IDL interfaces!

<script
  src="/resources/testharness.js">
</script>
<script
  src="/resources/testharnessreport.js">
</script>
<script src="/resources/WebIDLParser.js"></script>
<script src="/resources/idlharness.js"></script>

WebIDL test

<pre id='idl'>
partial interface Document {
  readonly attribute boolean hidden;
  readonly attribute DOMString visibilityState;
};</pre>
<script>
var idl = new IdlArray();
idl.add_idls(document.getElementById("idl").textContent);
idl.add_objects({Document: ["window.document"]});
idl.test();
</script>

Client-Server test

When you need server-side code as well

We have several examples:

1. XMLHttpRequest
2. Web Sockets
3. …

wptserve to the rescue!

wptserve

A server designed for writing tests for the web platform.

• stash: store cross-request state
• pipes: alter parts of the server response
• handlers: functions to control the response depending on the request
  • ResponseWriter: directly write to the output stream

wptserve: built-in pipes

• sub: substitute variables from the server environment, eg {{domains[www]}}
• status: set the HTTP status of the response
• headers: add or replace http headers in the response
• slice: send only part of a response body
• trickle: send the body of a response in chunks with delays

example.txt?pipe=slice(10,20)|status(403)

Running the test suite

• test runner: From Inside a Browser
• wptrunner: By Automating the Browser

Test suite results

Short summary: we have some ideas there but, in the meantime, you MAY do as follows:

• Use w3c/test-results
• Follow the instructions

Licenses For W3C Test Suites

W3C makes test suites available under two distinct licenses:

1. a W3C Test Suite License for an Authoritative W3C Test Suite or when claims of performance with respect to a specification are required.
2. a 3-clause BSD License for other applications.

This is builtin in w3c/web-platform-tests so no action needed on your part.

web-platform-tests, aka WPT

• It's the GitHub repository for tests
• Contains repository dependencies (wptserve, wpt-tools, testharness.js, etc.)
• It has a lint tool for catching common mistakes
• wpt-stability-bot is there to prevent unstable tests
• OWNERS files are there to notify interested individuals of pull requests
• Pull requests get automatically labelled
• The reviewer can be anyone other than the original test author

w3c-test.org

• master branch is automatically synced to http://w3c-test.org/
• Pull requests that have been checked are automatically mirrored to http://w3c-test.org/submissions/
• If needed, ask on irc to replicate your pull request to w3c-test.org
• w3c-test.org is unstable and unreliable by design, so don't rely on it to run tests

Strategies for testing a specification

• Don't wait until the last minute
• Ideally, no modification to the spec can be made without a test
• Find a leader who can understand w3c/web-platform-tests and how to submit tests

For more information

• Test the Web Forward documentation
• Use public-test-infra@w3.org
• Try irc.w3.org on #testing