WebVTT

The synchronization data is in a WebVTT file; specifically, a file using metadata content.

HTML

The text to be synchronized with timed media playback is in an HTML document, consumed in a web browser.

Timed media

This document does not specify which format(s) of timed media are allowed. The media must be able to be rendered via an HTMLMediaElement.

CSS

The highlight style is specified via the ::highlight pseudo-element.

Javascript

A small amount of Javascript is used to pair TextTrackCue events with Custom Highlights.

Selectors and states

Identify text to be highlighted with a selector.

The WebVTT file

The synchronization data must follow WebVTT's file structure, which consists of a series of cues, following the WEBVTT declaration at the top.

Each cue has an identifier (a number, in the example) and contains timing information for the media segment. It also includes a JSON-formatted custom cue payload with these properties:

WEBVTT

1
00:00:00.000 --> 00:00:03.187
{"selector":{"type":"FragmentSelector","value":"dtb1"},"group":"phrase"}

2
00:00:03.187 --> 00:00:07.184
{"selector":{"type":"FragmentSelector","value":"dtb2"},"group":"phrase"}

3
00:00:07.184 --> 00:00:10.945
{"selector":{"type":"FragmentSelector","value":"dtb3"},"group":"phrase"}

Because of how WebVTT files are parsed, it is important to not have a blank line anywhere in the cue payload.
"Note that you cannot provide blank lines inside a metadata block, because the blank line signifies the end of the WebVTT cue,".

Associating the WebVTT file with the HTMLMediaElement

The synchronization data of WebVTT is associated with an HTMLMediaElement by being referenced as a metadata track on that media object.

<audio controls autoplay src="chapter.mp3">
 <track default src="highlight.vtt" kind="metadata">
</audio>

Styling the highlight

Style the highlight in CSS using the ::highlight pseudo-element:

::highlight {
    background-color: yellow;
}

Text highlights shows how to make the cue into a CSS Highlight and therefore available to style in this way.

Cue events

As the HTMLMediaElement plays, associated TextTrackCues will trigger events when their timestamp is reached. For example, this is how to listen to the enter event, for when a cue starts:

let track = Array.from(document.querySelector('audio').textTracks)[0];
Array.from(track.cues).map(cue => {
    cue.onenter = e => {
        let cuePayload = JSON.parse(cue.text);
        doHighlighting(cuePayload.selector);
    };
});

Text highlights

This is how to programatically create a Custom Highlight from a selector.

Create a StaticRange from a selector.

{
    "selector":{
        "type": "CssSelector", 
        "value": "nth-child(1 of .stanza) > :nth-child(1 of .line)",
        "refinedBy": {
            "type": "TextPositionSelector",
            "start": 0,
            "end": 4
        }
    }
}

function createRange(selector) {
    let node = document.querySelector(selector.value);
    return new StaticRange({
        startContainer: node.firstChild,
        startOffset: selector.refinedBy.start,
        endContainer: node.firstChild,
        endOffset: selector.refinedBy.end + 1
    });
}

Then create a Custom Highlight using that range.

function doHighlighting(selector) {
    let range = createRange(selector);
    let highlight = new Highlight(range);
    CSS.highlights.set("sync", highlight); // "sync" is chosen arbitrarily here
}

Style the highlight using the ::highlight pseudo-element.

::highlight(sync) {
    background-color: yellow;
}

WEBVTT

1
00:00:07.800 --> 00:00:08.200
{"group":"word", "selector": {"type": "CssSelector", "value": ":nth-child("1" of .stanza) > :nth-child(1 of .line)", refinedBy: {"type": "TextPositionSelector", "start": 0, "end": 3}}}

2
00:00:08.200 --> 00:00:08.800
{"group":"word", "selector": {"type": "CssSelector", "value": ":nth-child("1" of .stanza) > :nth-child(1 of .line)", refinedBy: {"type": "TextPositionSelector", "start": 5, "end": 8}}}

3
00:00:08.800 --> 00:00:09.200
{"group":"word", "selector": {"type": "CssSelector", "value": ":nth-child("1" of .stanza) > :nth-child(1 of .line)", refinedBy: {"type": "TextPositionSelector", "start": 10 "end": ,0"}}}

4
00:00:07.800 --> 00:00:12.600
{"group":"line", "selector":":nth-child(1 of .stanza) > :nth-child(1 of .line)"}

5
00:00:12.600 --> 00:00:16.200
{"group":"line", "selector":":nth-child(1 of .stanza) > :nth-child(2 of .line)"}

6
00:00:17.200 --> 00:00:21.000
{"group":"line", "selector":":nth-child(1 of .stanza) > :nth-child(3 of .line)"}

7
00:00:07.800 --> 00:00:31.100
{"group":"stanza", "selector":":nth-child(1 of .stanza)"}

let range = createRange(cuePayload.selector)
let highlight = new Highlight(range);
CSS.highlights.set(cuePayload.group, highlight); 

::highlight(stanza) {
    background-color: lightyellow;
}

::highlight(line) {
    text-decoration: none;
    background-color: yellow;
}
::highlight(word) {
    text-decoration: underline;
    text-decoration-style:dotted;
    text-decoration-thickness: 2px;
}

Content navigation