Accessibility Conformance Testing Framework

Editor’s Draft,

This version:
https://w3c.github.io/wcag-act/act-framework.html
Previous Versions:
https://w3c.github.io/wcag-act/archive_act-format/2017-02-20.html
Editors:
Wilco Fiers (Deque Systems)
Maureen Kraft (IBM Corp.)

CC0 To the extent possible under law, the editors have waived all copyright and related or neighboring rights to this work. In addition, as of 23 February 2017, the editors have made this specification available under the Open Web Foundation Agreement Version 1.0, which is available at http://www.openwebfoundation.org/legal/the-owf-1-0-agreements/owfa-1-0. Parts of this work may be from another specification document. If so, those parts are instead covered by the license of that specification document.

Abstract

The Accessibility Conformance Testing Framework 1.0 (ACT Framework 1.0) is a specification designed to harmonize how accessibility rules are described for automated test tools, and how test procedures are written for quality assurance testing. By writing these tests in a defined format, organizations are better able to document and share their method of testing. A defined format for writing tests is the first step in enabling a harmonized approach for accessibility conformance testing.

1. Introduction

There are currently many products available which aid their users in testing for conformance of accessibility standards such as WCAG 2.0. As the web develops and grows in both size and complexity, these tools are essential for managing the accessibility of resources available on the web. The volume of information and services organizations provide on the web make it often impractical to manually test for accessibility.

Accessibility experts often disagree on how accessibility requirements should be tested. These disagreements on how a requirement should be tested, lead to conflicting results of accessibility tests. This is true for both manual accessibility tests as well as for accessibility testing done through automated test tools (ATTs).

Describing how to test certain accessibility requirements is the answer to this. By describing the test procedures, the results of an accessibility test become reproducible, and the test method becomes transparent. The Accessibility Conformance Testing Framework (ACT Framework) defines the requirements of these test descriptions, known as Accessibility Conformance Testing Rules (ACT Rules).

2. Scope

The ACT Framework is created to develop rules for the conformance of web technologies, including those used in digital publishing. This includes technologies such as HTML, CSS, WAI-ARIA, SVG and more. The ACT Framework is designed to be technology agnostic, meaning it has no specific technology in mind. This also means that the ACT Framework could conceivably be used for other technologies. Whether or not this is the case depends on the specific technology.

Accessibility requirements such as Web Content Accessibility Guidelines, which is specifically designed for web content, can be tested using ACT Rules. Other accessibility requirements that would also be applicable to web technologies should also be testable with ACT Rules. Because some of those accessibility requirements may be applicable to technologies other than web technologies, the ACT Framework may not be suitable for every part of this requirement.

For example, the User Agent Accessibility Guidelines 2.0 is applicable to web-based user agents, for which ACT Rules could be developed, but other technologies can also be used to develop User Agents, which are not web-based.

3. ACT Rule Structure

3.1. Rule Outline

A rule MUST provide the following items written in plain language:

3.2. Rule Description

Each ACT Rule MUST have a description that:

3.3. Accessibility Requirements

Explain the accessibility requirement being tested such as the WCAG SC and / or the technique the rule maps to; For example WCAG 2.0 Technique H67.

3.4. Limitations, Assumptions or Exceptions

List any limitations, assumptions or any exceptions for the test, the test environment, technologies being used or the subject being tested; For example: A rule for 1.4.1: Use of Color has to make an assumption with CSS-properties are used to make a link visually evident. Typically something like background, border, color, font, or text-decoration.

3.5. Accessibility Support

Editor’s Note - The ACT Taskforce acknowledges that this approach does not capture all nuances of accessibility support. We are looking for feedback on how to balance the need for organizations to solve issues with buggy or unsupported features in assistive technologies, with keeping the development cycle lean, and delivering future-proof products.

Determining if a web page is accessible, depends partly on the assistive technologies that are used to access it. This is known as [Accessibility Supported in WCAG 2.0][accsupp]. When testing a web page for accessibility, it is important to know which assistive technologies are to be used for this page.

Because of this it is important that users of ACT Rules can determine which accessibility features are relied upon in a rule. With this information, user can determine which rules do not provide results that are in line with the accessibility support baseline set for this particular test.

Developers of Accessibility Test Tools should use this information to develop a default ruleset that is sensible for their user base. Additionally, they could allow users of ATT to customize which rules are run, allowing fine grained control of the accessibility test.

A list of `Relied upon accessibility features` is an optional part of ACT Rule. Not every accessibility requirement involves assistive technologies, and so certain rules will have no use for this feature. For HTML, XML and similar markup documents, the accessibility features should be described using a three part CSS selector.

Example 1: Requires support for the `alt` attribute on an `img` element, existing within a link.

  a[ref] img[alt]

Example 2: Requires support for role=menuitem on any type of element, existing within an element marked up as a menu

  *[role=menu] *[role=menuitem]

4. ACT Input Data

4.1. Test Subjects

Web pages, including publications and applications go through many different stages before it is rendered to the end user. For example, PHP may be used to put various pieces of content into a template. That template is then sent as an HTML text document to a web browser, which in term parses it and turns it into a DOM tree, before rendering it to the screen. At each of these stages, accessibility tests could be run. It is therefore important to specify the *test subject type* that an ACT Rule expects.

The following test subject types are common in accessibility testing:

Other types of test subjects may be possible. In those cases the ACT Rule *must* include a detailed description of the test subject type, or a reference to the that description.

4.1.1. HTTP Response Testing

Testing the files as they are served to the web browser (or other user agent) has its limitations. The files may be manipulated in different ways through presentation and scripting. Although this is an excellent place for parser testing.

4.1.2. DOM Tree Testing

After the web browser (or other user agent) has parsed the files, and executed scripts to get it into a specific state (be it the initial state or any other), tests can be run against the DOM Tree. The DOM Tree can be tested for things like correct parent-child relationships, use of required attributes or properties and more.

4.1.3. Rendered Page Testing

Testing the browser is the next level up from DOM Tree Testing. In addition to building the DOM Tree, the browser styles elements in the DOM tree and positions them. This enables a rule to determine if an element is visible, which is critical for many tests. Additionally, testing things like the color contrast becomes possible at this level.

4.1.4. Template Testing

A template is document that has open fields that are filled with pieces of content or other templates. E.g. an HTML template contains a head with a variable title, a predefined navigation, and a variable content area.

Example: `` tags with a variable `src`, *must not* have a static `alt` value.

4.1.5. Script Testing

A composition is a class or component that extends other native elements or other compositions, to build a higher level component. E.g. a login form component, consists of a form, a few fields, and a label.

Example: Component properties starting with `aria-` *must* exist in the list of WAI-ARIA attributes.

5. ACT Test Procedure

5.1. Selector

A selector is a pattern that is used as a condition to filter input data to be evaluated against the test procedure. For example, finding all nodes in the DOM tree, or finding tags that are incorrectly closed in an HTML document.

Selector syntax depends on the test subject type. When the input data is an HTML document or set of elements, the selector must be a CSS selector. When a formal selector syntax is not available for the test subject type, the selector may be described unambiguously in plain English.

5.2. Test Cases

Each rule contains one or more test cases written in plain language. Each test case MUST provide the following:

Test case results are combined with other test case results to provide a cumulative outcome, pass or fail, of the rule. When a rule has multiple test cases, the results of all test cases are combined to give a single result for each selected item. The results are cumulative, meaning that as long as one test case passes, the rule has passed.

Example: A rule for HTML img elements has two test cases, one that checks if there is a text alternative, and another that checks if the image is marked as decorative. If either one of these passed, the rule has passed. Only if both fail, does that element fail the rule.

6. ACT Output Data

With ACT Rules, it becomes important that data coming from different sources can be compared. Only by having a shared vocabulary, can accessibility data that is produced by different auditors be compared and where necessary aggregated. Every ACT Rule MUST therefore require that the output is expressed in a format that has all of the features described in the ACT Data Format.

6.1. ACT Data Format

Rules are tested in two steps. Firstly, the selector is applied to the web page or other test subject. This results in a list of *Selected item* (elements, tags or other components) to test. Following this, each *selected item* will be taken through the test procedure of the rule. This will give the *outcome* for each *selected item*. For contextual information, the output data must also include *test subject* and the *rule identifier*.

This will mean that every time a rule is executed on a page, it will return a set with zero or more results, each of which has at least the following properties:

6.1.1. Example: (using EARL and JSON-LD)

{
  "@context": "<see appendix 1>",
  "@type": "Assertion",
  "subject": "https://example.org/",
  "test": "auto-wcag:rules/SC1-1-1-css-image.html",
  "outcome": "Failed",
  "pointer": "html > body > h1:first-child"
}

6.1.2. Selected Item {#selected-item}

When representing the selected item in the output data, it is often impractical or impossible to serialize the selected item as a whole. Instead of this, a pointer can be used to indicate where the selected item exists within the web page or other test subject. There are a variety of pointer methods available, such as those defined in [Pointer Methods in RDF 1.0](https://www.w3.org/TR/Pointers-in-RDF/).

The pointer method used in the output data of an ACT Rule MUST include the pointer method used [Implementation Test Cases](#quality-implement).

6.1.3. Outcome {#outcome}

The procedure of a rule MUST always return one of the following outcomes:

6.1.4. Ensure Comparable Results

In addition to the Selected Item and the Outcome, ACT Rules MUST require contextual information be provided. This must include what web page, file or other test subject the rule was applied to, as well as an identifier of the rule itself.

6.1.5. Test Subject

When a single URL can be used to reference the web page, or other test subject, this URL MUST be used. In scenarios where more complex actions are required to obtain the test subject (in the state that it is to be tested), it is left to ATT developers to determine which method is best used to express the test subject.

6.2. Rule Aggregation

Editor note: In this section we describe how the data that is returned from a rule can be combined to give a higher level view of the conformance to accessibility requirements. Rules provide very low level information, this is valuable for people working at that level, but managing accessibility of products as a whole requires a higher level understanding of the accessibility.

7. Rule Quality Assurance

7.1. Managing Exceptions

Editor note: This section will describe how a rule author should document scenarios where a rule might cause incorrect results. Such exceptions exist in nearly every rule and must be managed actively. Some exceptions can be mitigated by adjusting the rule, but others may be unavoidable. In both cases documenting such exceptions is valuable in interpreting the results of a rule.

7.2. Implementation Validation

Implementation tests are tests for accessibility test tools. These tests enable the developers and users of ATTs to validate the implementation of an ACT Rule. Each rule must have implementation tests for the selector, as well as for each test case in the rule.

An implementation test consists of two parts: a piece of input data and an expected result. When applying the test, the piece of input data –for instance an HTML code snippet– is evaluated by following the rule’s test procedure. The result is then compared to the expected result of the test. The expected result consists of a list of [pointers](#selected-item) and the expected [outcome](#outcome) (pass, fail, inapplicable) of the evaluation.

7.3. Accuracy Benchmarking

Editor note: This section describes how to measure the rate of incorrect results to correct results of a rule. Measuring this accuracy, not just on test data, but on pages and components that the rules would actually be applied to, is important to give users of the test results confidence in the accuracy of this data.

7.4. Update Management

7.4.1. Version Numbers

Each ACT Rule must have it’s own version number. The version number has to follow the semantic versioning schema. Using the X.Y.Z schema in the following way:

X / Major updates:

The major version number must be increased if the change can lead to new failure results.

Y / Minor updates:

The minor version number must be increased if the test logic has been updated, which could lead to a a different result.

Z / Patch updates:

The patch version number must be increased if the change does not affect the result of a rule. This includes editorial changes, new issues on the issues list, an updated description, etc.

7.4.2. Change List

All major and minor changes to an ACT Rule must be recorded in a change log, that is part of the updated rule. The change log must at least include the changes since the last minor update, as well as a reference to the previous version.

7.4.3. Issues List

An ACT Rule may include an issues list. This list must be used to record cases in which the ACT Rule might return a failure where it should have returned a pass or vice versa. There may be several reasons why this might occur, including:

The issues list serves two purposes. For users of ACT Rules, they give insight into why an inaccurate result might have occurred, as well as provide confidence in the result of that rule. For the designer of the rule, the issues list is also useful to plan future updates to the rule. A new version of the rule might see issues resolved and the item moved to the change log.

Appendix 1: JSON-LD Context

{
  "@vocab": "http://www.w3.org/ns/earl#",
  "earl": "http://www.w3.org/ns/earl#",
  "ptr": "http://www.w3.org/2009/pointers#",
  "auto-wcag": "https://auto-wcag.github.io/auto-wcag/",
  "Assertion": "earl:Assertion",
  "CSSSelector": "ptr:CSSSelectorPointer",
  "subject": { "@type": "earl:TestSubject" },
  "pointer": { "@type": "ptr:CSSSelectorPointer" },
  "test": { "@type": "@id"},
  "outcome": { "@type": "@id" },
  "Failed": "earl:Failed",
  "Passed": "earl:Passed",
  "CantTell": "earl:CantTell",
  "Inapplicable": "earl:Inapplicable",
  "Untested": "earl:Untested"
}

[accsupp]: https://www.w3.org/TR/WCAG20/#accessibility-supporteddef

Conformance

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

References

Normative References

[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://tools.ietf.org/html/rfc2119