In the simplest case, a conformance requirement is any single statement that uses RFC 2119 language (MUST, MUST NOT, SHOULD, SHOULD NOT, MAY), that specifies required behavior for a particular conformance class defined elsewhere in the spec (e.g., a Web browser or other user agent). However, some conformance requirements may not use RFC 2119 language.
Here’s a paragraph from the HTML5 spec that contains several conformance requirements:
The
dropEffectattribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When theDataTransferobject is created, thedropEffectattribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.
Here again is a paragraph from the HTML5 spec that contains several conformance requirements:
The
dropEffectattribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When theDataTransferobject is created, thedropEffectattribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.
That paragraph contains three statements that use RFC 2119 “must” language and so are unambiguously requirements that need testing:
none”,
“copy”,
“link”, or
“move”, then the attribute’s current value
must be set to the new value.
Here again is a paragraph from the HTML5 spec that contains several conformance requirements:
The
dropEffectattribute controls the drag-and-drop feedback that the user is given during a drag-and-drop operation. When theDataTransferobject is created, thedropEffectattribute is set to a string value. On getting, it must return its current value. On setting, if the new value is one of “none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value. Other values must be ignored.
That paragraph contains one statement that may or may not actually be a requirement that needs testing:
When the
DataTransferobject is created, thedropEffectattribute is set to a string value.
Although that statement does not use RFC 2119 language, it still
states something that could possibly be tested. Determining whether or
not it should be tested is something that requires reading other
parts of the spec. For this particular example, no test is necessary for
the statement above, because the actual requirement for setting the value
of the dropEffect when the DataTransfer object
is created is defined in another part of the spec.
Some algorithms in a spec may contain many individual statements or steps that don’t use each use RFC 2119 but that nonetheless state conformance requirements. Such algorithms are typically prefaced by a statement that does use RFC 2119 language.
Here’s an excerpt from an algorithm in the HTML5 spec:
When a user agent is required to fire a DND event named e at an element, using a particular drag data store, the user agent must run the following steps:
Let dataTransfer be a newly created
DataTransferobject associated with the given drag data store.Set the
effectAllowedattribute to the drag data store's drag data store allowed effects state.Set the
dropEffectattribute to "none" if e isdragstart,drag, ordragleave…
Notice that the algorithm is prefaced by a statement saying, “the user agent must run the following steps”.
There usually isn’t a one-to-one relationship between a conformance requirement in a spec and the number of test cases needed to test that requirement adequately. Instead, each requirement typically needs multiple test cases. For example, the following statement from the HTML5 spec would need at least four test cases—one for each of the possible values given.
On setting, if the new value is one of “
none”, “copy”, “link”, or “move”, then the attribute’s current value must be set to the new value.
That requirement likely in fact would actually need even more test cases. Determining how to test it adequately would require reading other parts of the spec.
It’s always preferable for a test case to focus as narrowly as possible on isolating a test for one specific case (though that case will almost always have dependencies on other major features: the parsing algorithm, the entire DOM, etc.) But often it’s not that simple to narrow it down that specifically. For example, in the case of testing complex algorithms, a single test case may necessarily be testing a combination of requirements from different parts of the algorithm.
For any particular spec, determining things like which statements in the spec require testing and how many test cases are needed to test them is something that requires a very close reading of the spec and very close familiarity with it.
Different spec editors take different approaches—and though the style and conventions of W3C specs for browser technologies have “normalized” and improved considerably over the years in terms of the degree of rigor and precision and lack of ambiguity with which the specs state requirements (for example, using RFC 2119 language consistently to clearly identify requirements), isolating exactly what to test and how to test it necessitates first spending a lot of time reading the spec.
When the result of your test could be determined by a script.
<script src="/resources/testharness.js"> </script> <script src="/resources/testharnessreport.js"> </script>
<head>
<title>Document.title exists</title>
</head>
<body>
<div id='log'></div>
<script>
test(
function() {
assert_equals(document.title, "Document.title exists");
},
"Document.title exists"
);
</script>
</body>
See Demo
assert_true,
assert_false,
assert_equals,
assert_not_equals,
assert_approx_equals
assert_in_array,
assert_array_equals,
assert_regexp_equals
assert_own_property,
assert_inherits,
assert_idl_attribute,
assert_readonly,
assert_throws,
assert_unreached,
assert_any
Intended for:
requestAnimationFrame(callback), getCurrentPosition(successCallback), etc.addEventListener("…", listener)
var t = async_test("Load event fires");
window.onload = function(e) {
t.step(
function () {
assert_equals(e.type, "load");
}
);
t.done();
};
See Demo @@find a better example...
In the <head> element:
<title>title exists</title> <link rel="help" href="http://www.w3.org/TR/html5/dom.html#document.title"/> <meta name="assert" content="Document.title exists"/> <link rel="author" title="John Doe" href="mailto:john.doe@example.com"/>
<script>
test(
function() {
var myE = document.getElementById("MyE");
assert_true(!!MyE.firstElementChild);
},
"Element.firstElementChild exists",
{
help: 'http://www.w3.org/TR/domcore/#dom-element-firstelementchild',
assert: ['The firstElementChild property must exist'],
author: 'John Doe <john.doe@example.com>'
}
);
</script>
When the result of your test is visual.
| bdo-000.html | bdo-000-ref.html |
|---|---|
| HEBREW | WERBEH |
<bdo dir="rtl">HEBREW</bdo> |
<span>WERBEH</span> |
As for testharness.js, put it in the <head> element:
<title>rtl value on dir attribute on bdo element</title> <link rel="help" href="http://www.w3.org/TR/html5/global-attributes.html#attr-dir-rtl"/> <link rel="help" href="http://www.w3.org/TR/html5/the-bdo-element.html#the-bdo-element"/> <meta name="assert" content="The dir attribute on bdo element could invert the text direction"/> <link rel="author" title="John Doe" href="mailto:john.doe@example.com"/>
When everything else wouldn't work.
Iconify this window and restore it.
If the square is still red, please indicate that the test has failed.
See also Demo
Test your WebIDL interfaces!
<script src="/resources/testharness.js"> </script> <script src="/resources/testharnessreport.js"> </script> <script src="/resources/WebIDLParser.js"></script> <script src="/resources/idlharness.js"></script>
<pre id='idl'>
partial interface Document {
readonly attribute boolean hidden;
readonly attribute DOMString visibilityState;
};</pre>
<script>
var idl = new IdlArray();
idl.add_idls(document.getElementById("idl").textContent);
idl.add_objects({Document: ["window.document"]});
idl.test();
</script>
When you need server-side code as well
So far, we have two examples of those:
Use PHP to simulate server responses:
We currently have two Web Socket servers running on w3c-test.org:
If you're testing audio or video content, we have content already:
Checkout media.js for ways to make your test codec agnostic.
We currently have two shared areas for repositories for storing test cases:
If you don’t have admin access to create new repos in either of those places, contact Mike Smith <mike@w3.org> with the details and he’ll set up the repo for you.
The repos are all ACL-controlled, so submit (push) new test cases into a particular repo, you either need to added to the list of authorized contributors, or you need to be a member of a working group all of whose members are authorized to push to the repo.
If you need push access to a particular repo, contact Mike Smith <mike@w3.org>.
The general protocol for adding new test cases is to place your tests
in a subdirectory that has either your company’s name or your personal
name, under the top-level submission directory in the repo.
We have a shared system at http://w3c-test.org/framework/ for running test suites
Documentation on adding a new test suite to the framework is available at http://w3c-test.org/framework/docs/maintainer/.
There are four basic steps to adding a new test suite:
Once you’ve completed those steps, anyone will be able to come in and run the actual test cases in their own browsers.
W3C makes test suites available under two distinct licenses:
@@Guidelines for spec authors?