Last updated Wed May 05 2021
There are hundreds of thousands of existing titles of comicbooks, graphic novels and other publications which storytelling techniques rely heavily on more or less complex page layout. Deconstructing such page designs to fit on smaller screens or adding accessibility features for such publications is very hard, and many times not feasible as entire pages are often rendered as single bitmap images.
SyncMedia lets an author describe a narrative journey through an existing publication using standard fragment selectors such as Media Fragments, PDF selectors and EPUB CFIs. Each step in the SyncMedia timeline directs the SyncMedia Player to a specific area on the page using X and Y coordinates as well as a prefered zoom level. A clip mask can also be applied after the translation has been completed in order to isolate the page content for clarity.
SyncMedia further allows the author to add spoken narration to each step and by doing so add accessibility features such as adding alternate descriptions of the overall context, as specific dialog present in speech bubbles etc.
A very common use case for synchronization of different types of media is in book subscription services. Users of these services often expect a "seamless" transition between the text rendition and audio renditions of the same book. This is a feature that is hard to develop and maintain since there are no standardized ways to express these synchronization points.
SyncMedia offers a standardized way to add granular to relationships between equivalent content in different renditions of the same title. It uses existing selector formats such as EPUB CFIs and Media Fragments to link to resources and positions within the various formats.
In addition to acting as a mapping between media formats its explicit linking between locations within the different formats also allow for parallell playback, analogous to the way that EPUB Media Overlays work.
Olga, a content creator, wants to include background music in her narrated publication. She chooses SyncMedia because it allows her to isolate the background music on its own sync:track and also allows her to use different music for different chapters.
Sanjay starts reading a SyncMedia-enhanced publication that contains background music. He wants to mute the background music because he has an Auditory Processing Disorder that makes the background music very distracting. His SyncMedia-aware player gives him independent control over each sync:track and allows him to mute the background music track but leave the narration track unaffected.
Casey starts reading a SyncMedia-enhanced comic book that includes sound effects in parallel with the narration. They would prefer to hear less of the sound effects and more of the narration. By using a SyncMedia-aware player, they are able to independently adjust the volume levels of the different types of audio, because the author of the comic book has used the sync:track feature of SyncMedia to place the sound effects audio on a different track than the narration audio.
Francesca, who has low-vision, prefers to read with very large text and simultaneous audio narration. She purchases the SyncMedia-enhanced version of a book for these reasons. When she reads, they can enlarge the HTML-based text, which reflows naturally on whatever size of screen she's using; and with the addition of SyncMedia for narration, the book has audio segments associated with each sentence. Her SyncMedia-aware player not only gives her high-level navigation for moving between publication chapters, but also lets her move by sentence. The experience is seamless because the text and audio remain synchronized throughout.
https://w3c.github.io/coga/content-usable/#objective-8-support-adaptation-and-personalization-0
Olivia has been enjoying a SyncMedia-enhanced publication but gets interrupted and has to stop reading. When she resumes later, her SyncMedia-aware player automatically picks up where she left off.
Juan is listening to a narrated book while also reading the text on a screen. They read slowly and want to reduce the rate of the narration to match. Their SyncMedia-aware player lets them slow down the presentation to 50% and maintains the fidelity of the audio (e.g. pitch) as much as possible. The rate modification does not affect the synchronization aspect of the presentation: the text segments are still aligned with the audio segments, even at a different playback speed.