This white paper introduces EPUB+WEB, a vision for the future of digital publishing that is based on a fully native representation of documents within the Open Web Platform. EPUB+WEB achieves full convergence between online and offline/portable document publishing: publishers and users won't need to choose one or the other, but can switch between them dynamically, at will.
The name “EPUB+WEB” is used colloquially in this paper. The eventual effort may choose to use a completely different name that is more evocative of the larger community we want to engage; e.g. “Portable Document”, “Portable Multimedia Document”, “Offline Web Document”, “Multipart Electronic Publication“,… In what follows, for the sake of simplicity, we will use the term “EPUB+WEB”.
This is a living document which at the time of writing reflects the authors’ opinions only. Comments and suggestions are welcomed, either via the authors’ email addresses (see above) or via the github issue tracker (with a preference for the latter).
Our vision for EPUB+WEB is that portable documents become fully native citizens of the Open Web Platform. In this vision, the current format- and workflow-level separation between offline/portable (EPUB) and online (Web) document publishing is diminished to zero. These are merely two dynamic manifestations of the same publication: content authored with online use as the primary mode can easily be saved by the user for offline reading in portable document form. Content authored primarily for use as a portable document can be put online, without any need for refactoring the content. Publishers can choose to utilize either or both of these publishing modes, and users can choose either or both of these consumption modes. Essential features flow seamlessly between online and offline modes; examples include cross-references, user annotations, access to online databases, as well as licensing and rights management.
A portable document is a collection of content items (e.g. pages, chapters, modules, articles) structured as a single, self-contained logical unit. Individual items can consist of text, images, graphics, possibly interactive mathematical or chemical formulae, as well as audio and video. These documents by definition have a default, linear “reading order”, however the user may choose to skip around in the content just as with a book on paper; alternatively, interactive aspects of the content may alter the reading order on behalf of the user.
Several portable document formats exist. The only vendor independent and HTML based format is EPUB [[EPUB3]], which emphasizes a dynamic determination of content presentation and a closer alignment with the Open Web Platform. EPUB can represent reflowable contents as well as sequences of final-form fixed pages; depending on the publication type, an EPUB document may default to one of those. EPUB is built on Web Standards, and the individual items that make up an EPUB publication are identical to types of content on a Web site: [[html5]], [[svg]], [[css21]], [[ECMAScript]], [[JPEG]] and [[png]] images, etc.
EPUB can be viewed as simply defining a specialization of Web content that assures that a collection of content items has the needed properties of completeness and logical structure, and does so in a standard way that other processing tools and services can reliably create, manipulate, and present such collections. This completeness constraint is key for bridging the current gap between an online and offline/portable view of the same content (see section on usage patterns below).
The differences between the distinguishing characteristics of web documents and portable document can be viewed as situational and gradual rather than as representative of bright-line distinctions. Just as most of the content features of Web pages and portable documents implemented via EPUB are held in common, portable documents and documents on the Web share a desire for occasional or on-demand use of their distinguishing characteristics. Authors of HTML Web pages using CSS for styling may want pagination features. Users of Web sites want the occasional ability to download a part of a Web site for offline use with links intact to both the saved offline content and remaining content that has not been downloaded. Reliable navigation of a Web site would increase usability and accessibility.
EPUB can be considered to be at a tipping point. EPUB has been broadly adopted globally for trade ebooks, and is starting to gain adoption among textbook publishers as well as corporate marketing departments. However, EPUB has largely been seen as an “offline” format up until now. Various browser extensions supporting EPUB exist (Readium in Chrome, EPUB in Firefox, et al.). Other solutions exist for delivering EPUB files in browsers (Readium-Cloud, EPUB.js, Safari Books Online, et al.). Browser- and cloud-based solutions require relatively complex server and/or client software. In many cases browser- and cloud-based solutions depend on a proprietary transformation of the packaged EPUB files into formats more suitable to network delivery. A focused effort to make EPUB a first-class Open web Platform citizen will result in significant reduction in the complexity of deploying EPUB content into browsers for both online and offline consumption. Further, this focused effort will increase the momentum of EPUB and associated web adoption across communities who are looking for an open, non-proprietary, next-generation portable document format.
The broader Web Platform can also be considered to be at a tipping point. Mobile platform web site use is diminishing in favor of native applications. Hybrid applications that use web content alongside native application technology, and web-technology-based system applications are growing. The specific means of delivering hybrid and web-technology-based system applications is currently proprietary to specific applications frameworks and/or browser platforms. The point of EPUB+WEB is to increase problem solving momentum in package, metadata, and offline support applicable to both portable documents and installed applications. Open and native solutions to replace proprietary packaging, metadata, and offline support are intended to ensure the broadest possible general adoption of the Open Web Platform.
In many respect, the convergence is already happening. In a number of areas (e.g., educational publications, travel books, etc.) publishers already exploit the advanced possibilities of EPUB3 to produce highly interactive documents whose features are very close to what one is used to on the Web. A good example is Cay Horstmann’s “Big Java, Late Objects” [[BigJava]] that combines the feel of a traditional book with interactive learning materials that makes it reminiscent of similar, Web based tutorials (see the video on a companion page for the book). And the converse is also true: tutorial and introductory articles have appeared on the Web that have the quality of traditional publications that one was used to see in a scientific magazine, but combined with the interactive possibilities of the Web (Mike Bostock’s article on visualizing algorithms or Bret Victor’s article on visualization are just two of several possible examples).
The convergence of EPUB and the Open Web Platform provides a common set of solutions and opportunities to various stakeholders:
Book publishers are investing in the development of technical expertise in web technologies. While gaining understanding of technical topics is important to new and future publishing workflows, the lack of communication between the trade publishers and web application developer communities is resulting in unnecessary duplication and investments in effort.
Collaboration between the web content development and publishing communities will result in major benefits to publishers. Adopting a universal and interoperable format means publishers can concentrate on engaging content authors in the production of high quality content. The web content development community can be relied on to deal with sophisticated technical issues (e.g., CSS, SVG). Potential future web content formats (e.g., 3D rendering) and various interactive web programs (e.g., visualization tools like D3) will naturally flow into the publishing realm through EPUB+WEB, hence increasing publishers' opportunities to sell new content products across the board.
Realizing new opportunities is a reality for publishers traditionally considered to be on the leading edge of technological advances in working with content. These publishers include STM and educational publishing houses, as well as scholarly and journal publishing organizations (see the section on scholarly publishing in this document).
A converged platform will support more tools and services and a much larger population of trained practitioners compared to the current state of working in parallel universes.
Scholarly journal publishers also provide articles for download these days. The most popular distribution format for journal articles continues to be [[PDF]] as a direct reflection of the scholarly community which highly prioritizes linear text and preservation of print typography. Indeed, the original goal for scholarly publisher to make files available online was to enable readers to download and print content directly, instead of borrowing a paper copy of a journal issue and photocopying relevant articles.
But things are changing. First of all, Web-only publications become part of the mainstream (e.g., the multidisciplinary PLOS ONE or the new PeerJ CompSci journals) with the main content being published with traditional Web technologies like HTML and CSS. And there is much more. Scholarly communication increasingly use additional media such as video, audio, animated graphics, or very large images, and the trend is to consider these as integral parts of the scientific output. (Mike Bostock’s recent article on visualizing algorithms or the “live” presentation of data in a paper published by F1000 Research are good examples for the new possibilities.) Furthermore, publishing the scientific data sources, like the results of a sociological survey or measurement output of biochemical experiments in XML or CSV formats, alongside the “main” publication, is also coming to the fore, with some journals and institutions actually requiring a public access to those. Gaining access to all these various media and contents both online and offline is important for scholars, whether the goal is to read the publication on the Web, or to download the papers for various reasons: reading the article offline, inclusion of the paper into bibliographic management systems like Mendeley or Zotero, or peer-reviewing submissions. Any offline format for scholarly purposes should be adapted to these needs.
EPUB offers a solution to many of the problems (see, e.g., [[Sigarchian]]), and those will be further enhanced by EPUB+WEB. Having an essentially identical online and offline versions of the same content, including the usage of various media, leads to similar reading experiences whether online or offline. User annotations, formal reviews, etc, performed by the scholar on a small, mobile device while being offline can be automatically synchronized with the online version as soon as there is Internet access. Being based on a general archival format, EPUB+WEB provides an easy way to consistently include video, audio, interactive scripts, any kind of data, and can naturally contain active links to the scientific data published elsewhere on the Web in case the data is too large to be distributed offline. These and other possibilities provided by EPUB+WEB may contribute to fundamentally change the way scholarly publishing works.
A special form of document production is related to technical and/or user documentation of complex products as well as complex administrative documents. Such documents are often akin to STM or scholarly publications edited by traditional trade or scholarly publishers but, often, the sheer quantity and complexity of production, as well as confidentiality requirements, mean that the production are done in-house. In many respects major corporations such as IBM, Intel, Renault, or Boeing, or institutions like the European Commission, the FAO, or the UNESCO have become specialized publishers themselves.
The quantity of documentation makes it infeasible to produce these documents in print (or print-only); instead, publishing them on the public Web or an Intranet and/or providing them through specialized mobile devices is the viable alternative. The production of these documents has similar challenges to scholarly publications like accessibility issues, portability of annotations, or the possible inclusion of complex media.
Just as for scientific publications, EPUB+WEB will provide new possibilities for these types of documents. Documentation in EPUB+WEB can be used offline in, for example, a cockpit, while being easily updated through the Web when possible. Inclusion of interactive animation, explanations, etc., become easy thanks to the possibilities provided by the Open Web Platform, whether online or offline.
Reading system developers will also benefit. It is already true today that, due to the large scale use of the Open Web Platform technologies in EPUB3, reading systems often rely on existing Web browser “cores”. This means that the development of these reading systems already benefit from a level of synergy insofar as they can rely on software developments done elsewhere. Making EPUB+WEB “native” to browsers will mean that an even larger percentage of the necessary software will be available as part of the “core” and developers can concentrate on book-specific issues such as specialized user interfaces or connection to online bookstores.
But the main advantage of EPUB+WEB for reading systems is a vastly larger user base. Whilst, today, reading systems are mainly used to read traditional novels, the introduction of EPUB+WEB will open up new possibilities for, e.g., scholarly and educational use, journals and magazines, governmental usage, etc.
The synergy between the traditional publishing community and the Web site designers may help in greatly improving the quality of overall Web page design. Indeed, the publishing community has significant experience on issues like ergonomy, complex layout design, paged layout, or user interface problems when consuming, for example, long, elaborate, and mostly linear content. Publishers also have an experience in a proper editorial and curatory workflow in producing content, which can be easily transposed from traditional publishing to Web site production.
Another aspect of Web page design is its adaption to various environments easily. Creating documents on the Web that could be uncompromisingly displayed both on traditional screens and on mobile devices with varying screen sizes is already a growing trend today; with EPUB+WEB, users will be able to create digitally native documents easily, whether the document is viewed online or offline.
Generation of an offline version of a Web page (mainly in terms of very long and complex content) is an area where browsers will benefit from EPUB+WEB. Such a facility is important: when roaming charging are often high, or when internet access may be of a low quality or not available at all, users need the possibility to create, in an ad-hoc and easy manner, an offline version of the Web page they are reading. Several browsers offer such facilities already, albeit in mutually incompatible formats. Making EPUB+WEB native to a browser means to standardize an archive format that can be used through a suitable user interface by anyone using a browser. Also, some of the facilities required by reading systems are also extremely useful for “traditional” Web content; annotation facilities are an obvious example. A joint development will therefore provide a welcome addition to the core browser facilities.
It must be emphasized, however, that EPUB+WEB is not meant to create an offline version of any Web page; the emphasis is on Web documents and not to, so to say, duplicate the Web. For example, it is not the goal of EPUB+WEB to store the page of a Web-based email client. The exact boundaries and limitations will have to be properly specified alongside the work on archival formats.
Note that, technically, the inclusion of EPUB+WEB capabilities in browsers is a matter of enhancement rather than addition of completely new capabilities: because EPUB+WEB documents are based on the core Web Technologies, the “extras” to make them a native feature of the Web is limited to a smaller set of tasks like handling packages, dealing with features like reading order, dynamic pagination, and displaying tables of contents. An important goal of EPUB+WEB will be to further streamline these tasks, all while making sure it is feasible to include EPUB+WEB content handling even in mobile environments, where the computing and memory limits are more demanding.
Users will benefit, arguably the most, from a convergence of efforts between EPUB+WEB documents and other uses of Web technologies. Users will have the choice among different reading systems for the same content, ranging from specialized devices to traditional Web Browsers. Beyond the overall qualities of the reading environment the choice can also be made based on the content and usage: whereas a specialized device would work well for reading a novel on the beach, a Web browser or a high-end tablet may be preferred to consume highly interactive educational content in a class room. Publishers do not have to make this decision: users can do that. The same content can also smoothly migrate from one device or system to another, possibly carrying notes, annotations, but also the possibility to fill interactive Web forms offline (and “pushing” the results to its destination when on line again). Features for people with disabilities will also be provided consistently, whether the content is a portable document or a Web page.
This section lists some of the work areas that EPUB+WEB should engage in. The list is not exhaustive and there are only hints at the technical solutions; one of the main goals of the work ahead will be to clarify the requirements and technical details. It must be emphasized that the solutions to these problems may not come from either IDPF or W3C, but possibly from other, external organizations (document identification is a typical example).
The latest evolution of browser technologies around Web Workers[[web-workers]] and Service Workers[[service-workers]] may fundamentally change the way browsers operate in terms of offline/online. Service Workers will provide a flexible and programmable way to efficiently implement local caching of Web resources. Caching is implemented as a programmable network proxy, meaning that the browser’s rendering engine becomes oblivious on whether a resource originates from the local cache or diretly from the network.
This evolution makes it particularly timely to bring online and offline/portable documents under the same abstraction and provides with a possible realization strategy for EPUB+WEB. Indeed, in this approach, EPUB+WEB can be considered, conceptually, as a general abstraction, as a Web Publication, identified with a URI, and representing a collection of Web resources (i.e., HTML, CSS, SVG, etc., files). When the publication is requested via its URI the server may return either a high level description of the publication (essentially listing the resources that are logically bundled together by the publication) or return a packaged version thereof in some packaging format (e.g., if those resource happen to be physically stored in a file). The client, more exactly the rendering part thereof (let us refer to it as the “reading system”) is oblivious to this difference and only deals with the logical references (URI-s) of the constituent resources. Essentially, the reading system would behave as if all resources were simply Web resources. Reading the content will also remain possible if the client is off-line.
This strategy is made available through a specialized Service Worker[[service-workers]] in the client. When accessing the Web Publication for the first time, this “EPUB+WEB” Worker would either directly cache the resources, if retrieved through traditional HTTP requests, or unpack the incoming package and store the resulting resources in the same cache (the package may also be originating from the local file system). All subsequent network accesses, originating from the reading system, will be proxied through the “EPUB+WEB” Worker, accessing the content in local cache. The access to the cache will be done transparently to the reading system.
Of course many of the details have to be worked, taking also into accoount that the Web Workers[[web-workers]] and Service Workers[[service-workers]] specifications are not finalized yet.
Regardless of the details or the EPUB+WEB architecture, an archive format is necessary for the storage of a publication as one file (e.g., for distribution). A variety of formats for offline/archival storage of collections of digital resources exist today [[OCF]], [[ODF]], [[OOXML]], but none of them is universally recognized and supported across ecosystems. Depending on the general architecture, EPUB+WEB may continue using the current EPUB packaging format (based on [[OCF]])) or it may need to be based on the definition of an archive format that is generic and native to the Open Web Platform.
W3C’s Web Application Working Group has published a First Working Draft for a Streamable Package Format for the Web[[web-packaging]] to encompass the needs of various applications (like installing Web Applications or downloading data for local processing). It is may become advantageous for EPUB+WEB to adopt this format, thereby being compatible with what Web Browsers may implement anyway. While this general packaging format could hypothetically be compatible with the ZIP+XML manifest format used by EPUB (and also by the Open Document Format [[ODF]]) the broader requirements of installable applications and other types of content, and efficient incremental transmission over networks, may well imply a different and incompatible packaging format.
The IETF has published an informational draft on a top-level media type for archives. Although that draft does not specify a specific archive format it shows the overall interest in packaging on the Web, in line with the concerns of EPUB+WEB.
Portable document formats usually include information pertaining to the overall publication structure, such as the logical reading order(s) of the set of resources that comprise the publication (e.g. the “spine” and associated constructs in EPUB), as well as predictable user-facing meta-structures, such as one or several tables of contents or “guides”. EPUB already incorporates definitions of how to express these data based on W3C Standards such as XML and HTML. For EPUB+WEB’s requirements it may be imperative to further optimize these data structures in a way that is even more native to the Open Web Platform and more easily supported in authoring tools, browsers, and Reading Systems. If the vision based on Service Workers, as outlined above, becomes the approach chosen for EPUB+WEB, that overall structural information may be what a client would return on a request for an EPUB+WEB instance, as an abstraction for a collection of Web resources.
The Manifest for web application format [[web-manifest]] is one example of a technology that could potentially be used to achieve this goal.
Whilst these information objects are important for larger and/or more complex publications, it is unnecessary for a number of use cases for EPUB+WEB. A typical example is the archival of a document consisting of a single Web page. EPUB+WEB should therefore include the definition of a set of “defaults”, i.e., it should not require the presence of, say, a spine if the publication contains one single HTML file. Such defaults are not currently present in EPUB.
On the web, HTTP URIs serve as the fundamental method of identifying a resource, or a fragment thereof. Among the various portable document formats available today, there is no equivalent ubiquitous method for identifying a publication that by definition does not have an HTTP address. Within the scholarly publishing industry, for example, initiatives such as DOI and CROSSREF have addressed this problem by providing explicit URN resolver services, but these services are not used by traditional “trade” publishing that rely more on ISBN related services. Also, for a universally applicable portable document format, unconditionally relying on distinct resolver services is suboptimal for a number of use cases, primarily as these may not be free of charge, and may require registration process that is not be applicable to the use case at hand.
A typical, and extremely valuable use case is in academic and scholarly publishing. There are currently several methods for citing online works (disputed among style guides), but there is no standard method for citations to ebooks. Even if a reflowable ebook is by a scholar, she must refer to PDF, paper copy, or HTML version to cite it in her bibliography. EPUB+WEB should enable stable citations.
Today’s EPUB supports a mechanism [[CFI]] for fine-grained references into a publication, but it is not defined in a manner that natively handles transitions between online and offline/portable states.
EPUB+WEB needs to define a way to utilize URI schemes for identifying documents and/or fragments thereof such that the addressing scheme does not break, nor needs to be changed, when a document transits from online to offline/portable states, or vice versa. A number of questions arise and need answers, e.g.:
Beyond the identification of the document as a whole, there is a further need for fragment identification, i.e., to identify “anchors” within the document. EPUB+WEB needs to include fragment identification schemes that are agnostic to the online/offline state, and that can address fragments of various kinds (e.g. resources within archives, elements with or without IDs, text ranges, time positions, etc.) and for various media types.
If the structure for EPUB+WEB is based on Service Workers, as outlined above, the usual fragment identifiers, as used on the Web, may be adopted and used without any further modifications.
The W3C Annotation Working Group has a joint deliverable with the W3C Web Application Working Group called “Robust Anchoring”. This deliverable will provide a general framework for anchoring; and, although defined within the framework of annotations, the specification can also be used for other fragment identification use cases. Similarly, the W3C Media Fragments specification [[media-frags]] may prove useful to address some of the use cases. Finally, the Streamable Package Format draft, mentioned above, also includes a fragment identification mechanism. Would that package format be adopted for EPUB+WEB, that fragment identification may also come to the fore as an important mechanism to consider.
Throughout the digital publishing industry, highly specialized metadata vocabularies and serialization forms thereof are being used. Within book trade publishing as an example, ONIX [[ONIX]] has attained a dominant status as a metadata package that typically travels (in XML form) independently of the publication, and contains not only bibliographic metadata, but also trade information such as pricing. Scholarly publishing, on the other hand, uses various derivatives of the ubiquitous BibTeX vocabulary.
While not contradicting the obvious use cases for out-of-line metadata records as used by publishers, retailers and libraries, EPUB+WEB must define a syntax for basic in-line metadata records that is agnostic to the online and offline modes. This means that the syntax must seamlessly support discovery and harvesting by both generic Web search engines, as well as dedicated bibliographic/archival/retailer systems. While it is expected that EPUB+WEB will define a minimal set of required metadata (cf. the section on identities and fragments above), development and adoption of further vocabularies in EPUB+WEB will most likely be deemed as out of scope; in other words: domain-specific metadata requirements are up to the domains themselves to define via a profiling mechanism, or similar yet-to-be-defined means.
The adoption of HTML as the vehicle for expressing publication-level metadata (i.e., using RDFa [[html-rdfa]] and/or Microdata [[microdata]] for metadata like authors or title) would have the added benefits of better I18N support than XML or JSON formats.
As outlined in [[dpub-latinreq]] or [[jlreq]], the Open Web Platform in general, and CSS in particular, is still lacking solutions for meeting all of the publishers’ expectations on satisfactory typography and layout for digital publications. While improved presentation fidelity will be of paramount importance to the overall success and adoption rate of EPUB+WEB, it is clear that many of these issues are going to be addressed on a case-by-case basis by the CSS WG over a longer period of time. STM publishing, for example, where the faithful representation and rendering of, e.g., mathematical or chemical formulae is of a paramount importance, has particularly severe requirements that must be fulfilled by the Open Web Platform technologies. Similarly, dynamic pagination of reflowable content is not natively supported by browsers today, and as a result EPUB Reading System developers are forced to implement pagination using various ad-hoc approaches, all coming with a significant penalty in terms of development costs, performance and stability.
It is anticipated that native support for pagination (in CSS and/or in the DOM) is going to be put forward by stakeholders as a critical component of EPUB+WEB; thus the finalization of EPUB+WEB may be contingent on the availability of a native pagination model for Web content. Today’s EPUB does not define a particular pagination model for reflowable content (although work has been done on this, leading to the experimental EPUB Adaptive Layout specification [[PGT]] that has informed subsequent related work in CSS WG).
Note that the “Houdini” Task Force, recently started jointly by the W3C CSS WG and the W3C TAG, may open new avenues to handle pagination.
The security model of the Web, based primarily on the same-origin policy and the concept of “site”, does not apply to portable documents, as the notion of “origin” is based on HTTP properties that are invalidated/non-existent when a document transitions from its online state to the portable state. EPUB+WEB must incorporate a state agnostic security model that defines rules for both the online and portable states.
When reading long-form (and sometimes mission-critical) publications, personalization — i.e. the ability for users to adapt the presentation to suit their needs — is of a paramount importance. While technologies such as CSS Media Queries have come a long way in terms of adapting content to devices, this is not the same thing as adapting to a user. Presentation control features are often available in EPUB Reading Systems, for example the possibility to dynamically change font size or background/foreground color schemes, but implementations are brittle and limited due to the lack of an underlying framework that explicitly supports user adaptation.
EPUB+WEB needs to incorporate an explicit framework for achieving advanced and predictable user-triggered presentation control. (Note that from this perspective, accessibility can be seen just a radical case of personalization.)
Different domains of digital publishing have vastly different expectations and/or requirements on the nature of the content and their presentation. In the digital comics domain for example, the default presentation form is, traditionally at least, pre-paginated, fixed-form, and image-based media, possibly with a set of omnipresent (i.e., cross-publisher) user interaction patterns that are expected to be enabled. On the other hand, for trade publishing the default form is fully reflowable content, where user interaction patterns are defined entirely by the user agent. In educational publishing, the ability to control structure, to include rich domain-specific structural semantics and extensive specialized metadata, are at the basis for enhanced reading system behaviors, as well as predictable content discovery and repurposing.
To allow for the predictability of content within those domains that need it, EPUB+WEB needs to incorporate a notion of “profiles” that content can be authored and validated against, and that user agent implementations can use to trigger enhanced behaviors, if any. To allow for agile feature-set extensions and innovation, EPUB+WEB profiles also needs to embrace the notion of “feature addons” that can be included by a publisher without risking to invalidate the integrity and functionality of the basic publication.
This White Paper outlines a vision for the convergence between the Open Web Platform and portable documents while also significantly advancing and expanding the existing EPUB ecosystem. The realization of this vision would require a strong cooperation between the traditional publishing and Web communities, ideally based on a close collaboration between IDPF and the W3C and potentially other relevant organizations. While it is envisaged that most of the work could be done in one or more dedicated Working Groups (within IDPF and/or W3C), it must be emphasized that many of the features will affect and will be affected by work done elsewhere, within or outside these organizations. The starting point will be to explore and plan for the detailed technical challenges to gain a better insight into the work ahead; this exploration should be done together with the various interested communities.