This document outlines the testing strategies, reporting, and specifies the W3C Candidate Recommendation (CR) exit criteria for the EPUB 3.3 specifications. The three Recommendation-track specifications that are subject to testing and PR review are:
At the core, an EPUB 3.3 publication consists of a number of [=publication resources=] that are in XHTML, SVG, CSS, or various media formats. Beyond these publication resources the EPUB publication includes some EPUB-specific files (e.g, a [=package document=]) whose formats are defined by the EPUB 3.3 standard family. Accordingly, an EPUB 3.3 reading system has to:
See also the the EPUB 3 Overview [[epub-overview-33]] for more details on the structure of EPUB 3.3 publications.
As a consequence of this particular structure, the validity and conformance of an EPUB 3.3 publication, as well as of a reading system implementation, includes the requirement of valid and conformant publication resources, and the conformant rendering thereof. Comprehensive testing would therefore require to include the union of all HTML, SVG, or CSS tests both on the content side as well as the implementation side as well as testing the unique, EPUB 3.3 specific features. The standard checking tools that are widely used by the publishing community, like EPUBCheck or ACE, indeed include externally developed HTML, CSS, or accessibility checkers, which would check the validity or conformance of the publication resources. Also, today’s reading systems do not develop an HTML renderer themselves; they, rather, rely on external tools (typically a “WebView” implementation) that can be assumed to conform to the relevant W3C specifications.
This means that these additional tests would be quite unnecessary for the purpose of testing the EPUB 3.3 specification (which is the main purpose of a W3C Candidate Recommendation). As a consequence, the testing strategy of EPUB 3.3 concentrates on the unique EPUB 3.3 features only, and considers the publication resources, as well as the reading systems, to conform to the relevant Open Web Platform specifications.
EPUB 3.3 consists of three recommendation-track documents:
Due to the specificities of these documents the strategy, exit criteria, and reporting are different. These are detailed below.
Implementers of the EPUB 3.3 Reading Systems specification are provided with a test suite that follows the examples of the Web Platform test model, but adapted to the specificities of EPUB. The test suite provides separate tests for all must statements, as well as for many should and for some may statements, that appear in the EPUB 3.3 Reading Systems specification.
The test suite documentation describes the individual tests, linking them to the relevant section(s) of the specifications. The editors’ draft (as opposed to the official, published version) of the specification has a pull-down menu attached to each must statement providing the references to the relevant tests.
The EPUB 3.3 Reading System document must have at least two passing, independent implementations for each test linked to a must statement in the specification.
The results of testing are available in a separate implementation report.
The normative features of the EPUB 3.3 specification can be divided, roughly, into the following categories:
Features that have a direct effect on the reading system behavior, and whose behavioral details are mostly specified by EPUB 3.3 Reading Systems. Examples are the list of supported media types, semantics of some internationalization features, or the specificities of URLs in the case of packaged contents. These features have their counterpart in the EPUB 3.3 Reading Systems document and they share the relevant tests. In other words, some of the aforementioned tests, and their implementations, are also relevant for the core EPUB 3.3 document.
Structural constraints on a the EPUB-specific files of the publication; examples include the structure of the container format (OCF) or of the package document.
Extra restrictions concerning content documents such as the restrictions on XHTML, the format of an XHTML navigation document, or the requirements on XML conformance.
Vocabulary items which serve as metadata for the publication, and added to the EPUB 3.3 package document. Although only few of those vocabulary items are required for a conformant EPUB specification, their formats, value constraints, etc., are normatively specified in the standard and these must be followed for a publication to be valid if they are used.
The criteria are different for each category of the normative features:
Features in category 1 share the testing methodology, exit criteria, as well as the tests and implementation results, with the criteria of EPUB 3.3 Reading Systems. This includes the mechanism whereby the tests are referenced from the editors’ draft via pull-down menus.
Testing the features in categories 2 and 3 concentrates on whether these restrictions are feasible in practice, i.e., whether they are enforceable when checking validity.
The latest version of EPUBCheck, namely EPUBCheck 5, validates EPUB 3.3. Earlier versions of EPUBCheck are extensively used throughout the industry for several years now, and act as “gatekeepers” in the publication workflow for the major EPUB platforms; it is to be expected that all these platforms will upgrade their workflow for EPUB 3.3.
EPUBCheck has its own test suite; the exit criteria for testing categories 2 and 3 features is therefore formulated by requiring that each relevant must statement in EPUB 3.3 is covered by at least one EPUBCheck test.
The results of testing is shown in a separate document, providing a short description, and reference, for each relevant EPUBCheck test. The test descriptions are grouped using the structure of EPUB 3.3, and these groups are referenced, via pull-down menus, from the section headers of the editors’ draft of the specification.
Testing the features in category 4 concentrates on vocabulary term usage, i.e., whether real-world publishers use those terms. Each term must have at least two independent publishers using those terms in production.
The results of testing are available in the separate EPUB 3.3 Metadata Usage Report.
The goals of the EPUB Accessibility 1.1 document is to define accessibility conformance of EPUB publications beyond the accessibility requirements defined by WCAG for publication resources. The normative features of this specification can be divided, roughly, into the following categories:
Additional structural constraints on a publication; examples include the requirement on page lists or reading order. Through those constraints the document defines accessibility conformance requirements on top of the conformance levels defined by WCAG.
Vocabulary items which serve as metadata for reporting; for example, conformance levels, or disclosing the accessibility features related to this publication.
Testing the features in category 1 tests whether these restrictions are feasible and used in practice. This is achieved by the requirement that at least 5 publishers will produce at least 1 EPUB publication each that conform to EPUB Accessibility 1.1 - WCAG 2.0 Level AA.
Note that these conformance levels rely on each publication resource to conform to WCAG 2.0 AA, when applicable; there are only a few EPUB specific features like page numbering. At least two independent publishers should be fulfilling that conformance level.
The separate EPUB Accessibility 1.1 Usage Report shows the conformance table.
Testing the features in category 2 concentrates on vocabulary term usage, i.e., that real-world publishers use those terms. Each term must have at least two independent publishers using those terms in production.
The separate EPUB Accessibility 1.1 Metadata Usage Report shows the implementation table.