Content Generation

Mezzanine File content is normally delivered to the service provider as a file with near lossless compression. Specifically, for High Definition content at 1080p with 24, 30, 50, or 60 frames-per-second, the bitrate of the original content is typically 25Mbps to 150Mbps for 1080P HD and 150Mbps - 300 Mbps for 4K UHD.

ABR streaming content is generated by transcoding the Mezzanine File against an transcoding profile. Firstly, the mezzanine file is transcoded into a set of versions, each with its own average bitrate. Secondly, each version is packaged into segments of a specified duration.

The first process is defined by an transcoding profile. A profile describes the set of constraints to be used when video is being prepared for consumption by a range of video applications. The description includes the different bitrates to be generated during the transcoding process that will allow for the same content to be consumed on a wide variety of devices on different networks from cellular to LAN.

The second process is performed by a packager which segments the different bitrates. Next, the output is packaged into a transport format such as transport streams (.ts) or fragmented MP4s (.m4s). Lastly, they are optionally encrypted with a DRM that is suitable for the environment where the content is going to be played out. The packager is also responsible for the generation of a manifest file, typically a DASH (.mpd), HLS (.m3u8) or possibly Smooth (.ism) or HDS (.f4v), that specifies the location of the media and its format.

Content Delivery

After the content has been generated the resulting segments of video and corresponding manifest files are pushed to an origin server. However, the assets are rarely delivered directly from the origin.

At this stage, the control in the chain switches to the client's web video application. The content provider supplies the client with the URL of a manifest file located on a CDN rather than the origin. The manifest URL is typically passed to a video player for playback. The player makes a HTTP GET request for the manifest from CDN edge. If the CDN edge does not currently have the manifest available, the CDN requests a copy of the file from the origin and the file is cached at the edge for later use. The CDN edge then returns the the manifest to the requesting player. (NOTE: There are different levels of caching; popular VOD content is kept closer to the edge in the CDN network in this way it can be delivered to customers faster than an edge server that isn't tuned for high volume delivery.)

Content Playback

As mentioned in the Content Generation section, the player uses a tag within the manifest to determine the playout type. In HLS, if there is a type with the value of VOD, the player will not reload the manifest. This has important consequences if there are changes in availability after the session as commenced. In DASH, the difference between a live and VOD playlist are more subtle. At this point the behavior differs between players found on the different devices depending on the transport formats. However, broadly, it behaves in the following way:

1. While the player's ABR algorithm determines which rendition to use during playback, for instance based on the currently measured network throughput or the current buffer, the algorithm might not have enough information at playback start to use such metrics. The initial decision depends on the player's ABR algorithm but is often based on prior knowledge. This can for instance be previous measurements of network throughput, a simple default value that is used if not enough data is available to estimate the current network throughput, or a custom algorithm to determine the initial bandwidth estimation.
2. Once the player has this information it can then compare this with the metadata from the manifest that describes the different qualities that the content provider is supplying. It picks the quality level with an average bitrate that is as close to the calculated throughput but within its bounds to avoid a situation where a consistent experience is interrupted as the player requires more data than the current network bandwidth can supply, where the player’s buffer is emptying faster than it is being filled.
3. It then requests a segment from a location on the edge server that typically relative to the location of the manifest.
4. The downloaded segment is then added to the video decoder pipeline for playback. For player's that are based on Media Source Extensions [[MEDIA-SOURCE]] the segment is appended to the corresponding media source buffer. Note that the segment is appended as is even if the content is encrypted. The underlying DRM system is usually integrated with the decoder pipeline and will decrypt the samples prior to decoding.
5. The media renderer pulls the video data from the buffer and passes it to the video surface where it is rendered.
6. If the available bandwidth remains constant the player will continue to request segments from the same bitrate stream, pulling down chunks and filling its video buffer. In the event a change in network availability the player will make a decision about the need to either drop to a lower rendition or request a higher bitrate. In case of HLS, switching the rendition will trigger a download of a new child manifest prior to the download of the associated segment. In case of MPEG-DASH the player can start downloading the associated segment directly.

Content Generation

For VOD the source is typically static file-based rather than a feed. The encoding profiles are also subtly different. A greater priority can be placed on high quality as the latency, time to live, is not a requirement. There are important differences in the manifests created. In HLS there is a tag that tells the player whether the playlist is describing on demand material: #EXT-X-PLAYLIST-TYPE:VOD. As we will see shortly this is used by the player. There are also client side restrictions where certain profiles are blocked due to rights restrictions and network consumption capped bitrates on movies and entertainment while being allowed on sports content. Fragments size will also effect playout as a player's ABR can be more responsive if the chunk is smaller, e.g. 2 seconds rather than 10 seconds.

Content Delivery

The CDN configuration and topology for delivering VOD content is also different to live. There are different levels of caching; popular VOD content which is kept closer to the edge in the CDN network in this way it can be delivered to customers faster than an edge server that isn't tuned for high volume delivery. Older and less popular content is retained in mid-tier caching while the long tail content is relegated to a lower tier.

Content Playback

As mentioned in the content generation section the player uses a tag within the manifest to determine the playout type. In HLS if there is a type with the value of VOD, then the player will not reload the manifest. This has important consequences if there are changes in availability after the session as commenced. In DASH the difference between a live and VOD playlist are more subtle but one of the primary identifier for liven content in MPEG-DASH is the profiles attribute of the MPD element which could for example reference the urn:mpeg:dash:profile:isoff-live:2011 profile to indicate live content.

There are other differences in playout as well. Unlike live, a VOD asset has a predefined duration, information around duration and playback position can be used to update the UI to provide feedback to the user on the proportion of the asset watched.

Additionally, the UX requirements are different between VOD and live. Unlike live content, which lends itself to being browsed within an EPG (electronic program guide), VOD content is typically navigated using tiles or poster galleries. There is also a trick play bar to view the current playback position and seek to other points within the content.

VOD Pre-caching

In the previous sections, we outlined the use cases associated with video streaming. In this section, we give some examples of use-cases that are specific to on-demand streaming and are mainly related to strategies employed on the clients to improve performance in some way.

Pre-caching is a strategy used in on-demand streaming and is important to ensure the best user experience. By buffering the beginning segments of a video, you can make playback as close to instantaneous as possible. This is important because buffering (the state of the video application when the player has insufficient content within its framebuffer to continuously play content), has a direct relationship to engagement and retention. For every second of buffering within a session a certain amount of users abandon a video stream. Web video application developers will use points within an application's UX to pre-cache content. For example, when entering a mezzanine/synopsis page, the application might preemptively cache the content by filling the player’s video buffer or, alternatively, storing the chunks locally.. Pre-caching allows the video to commence playing without buffering and provides the user with a more responsive initial playback experience. This technique is used by Netflix, Sky and the BBC in the case of on-demand content being watched in an in-home context. This technique is not used for cellular sessions where the user’s mobile data would be consumed on content that they do not watch.

Byte range requests in context of web video application

An HTTP range request allows a client to request a portion of a larger file. In the VOD scenario where a user wants to access a specific location, a range request allows the player access content at a specific location without downloading the entire transport chunk.

Both the player and the server need to support range requests. The player needs to be able to playback a source buffer and the server must be configured to serve ranges. The exchange begins when a client makes an HTTP HEAD request. If range requests are supported, the server will then respond with a header that includes Accept-Ranges: bytes and the client can issue subsequent requests for partial content. The returned bytes are added to an ArrayBuffer and then appended to the SourceBuffer which in turn is used as the SRC parameter of the of the video HTML5 tag/element.

HDCP security requirement for HDMI

Some content has limitations placed on its distribution by the owners. There are different ways of protecting the owner’s rights. The most common is DRM (digital rights management) which prevents the content being watched on a client device if the user cannot ‘unlock’ it with a key. However, once the content is unlocked the service provider should make every effort to prevent the user from re-distributing this content.

A technical solution to prevent interception as a signal travels from a device to a television or projector is termed High Bandwidth Content Protection (HDCP). If implemented by a device manufacturer, HDCP takes the form of a key exchange between devices. If the exchange fails, the signal is not transmitted to the display and the device is responsible for notifying the end user of the error.

Watermarking

Support for real-time watermarking is becoming an important consideration for distributors of high-value content. A service provider’s right to distribute content is linked to their ability to protect it with studios and sports channels insisting on the capacity for a service provider to detect a breach of copyright on a stream by stream basis. A service provider can include a vendor's SDK in the client that can add a watermark at run-time. Normally, the watermark is invisible to the consumer and takes the form of a digital signal, this is referred to as forensic water marking. Part of the service provided by the watermarking vendor is to monitor the black market for re-distributed material. Illicit streams or files are intercepted and screened for the digital fingerprint inserted on the client. If a suspect stream is found the vendor directs the service provider(s) to the source of the misappropriated stream.

While watermarking is not an issue that developers will often be faced with, the processing requirements can dictate the choice of platform for the content distribution. The watermarking process might require processing power and low-level functionallity through ring-fenced resources that platforms such as Android, iOS or Roku provide. The options for watermarking in the browser are limited to overlays and this limitation is then reflected in a distributor’s choice of platform for their web video application.

Content Generation

In contrast to VOD content generation, the typical input is a live feed of data. Usually there is also a higher priority on low-latency and time to live, which is in turn reflected in smaller segments.

A big difference between VOD and Live content generation can be found in the differences between manifests for the two content types. Besides a general profile that tells the player if the manifest represents live content, there are a few other, important properties that define how a player will be able to playback the live content, when updates will be fetched, and how close to the live edge playback starts. These properties are pre-defined and expressed in the manifest during content preparation, but play an important role in content playback and view experience.

Content Delivery

The content delivered through a CDN is generally very similar to the VOD playback use-case. There might be slight differences when it comes to caching and distribution of segments, and you might observe higher load on the origin when playback is configured to be very close to the live edge.

Content Playback

Even though playback and playback parameters between Live and VOD playback are very similar, there are critical differences. Maybe the biggest difference is the playback start position. While VOD playback session usually start at the beginning on the stream, a Live playout usually starts close to the live edge of the stream, with only a few segments between the playback start position and the end of the current manifest.

The other main difference is that the live content changes over time and the player needs to be aware of these changes. In both DASH and HLS, this results in regular manifest updates. Depending on the format, it is possible to define the interval for the manifest updates.

What happens during a typical playback session is:

1. Player loads the manifest for the first time
2. Playback starts at the live-edge defined in the manifest
3. Player re-loads the manifest in regular, pre-defined, intervals.
4. The new manifest is used to update playback state information and the segment list

This is a typical loop that the player goes through on each manifest update cycle.

Potential Issues

One of the most common cases that an application needs to be prepared for is the player falling behind the live window. Assume for example that you have a buffer window of 10 seconds on your live stream. Even if the user interface does not allow explicit seeking, buffering events can still stall the playout and push the playback position towards the border of the live window and even beyond it. Depending on the player implementation the actual error might be different, but implementation should be aware of the scenario and be prepared to recover from it.

Another common problem that can occur both in Live and VoD playback sessions are missing segment data or timestamp alignment issues. It might happen that a segment for a given quality is not available on the server. At the same time, misalignments in segment timestaps might occur. Although both of these issues migh be critical, how they are handled depends on the player implementation. A player can try to jump over missing segments or alignment gaps. A player can also try to work around missing segments by downloading the segment from a different rendition and allow a temporary quality switch. At the same time, a player implementation might also decide to treat such issues as fatal errors and abort playback. Applications can try to manually restart playback sessions in such scenarios.

Device Type

In the context of a mobile client the broad device type is already known - you can’t install an Android client on an iOS device. However, operating systems evolve and are extended. Within the application layer you will still need to determine the level of support for feature X and branch your code accordingly.

If the application is hosted or the same application is deployed via different app stores then the clients runtime could be more ambiguous. The classic approach is to use the navigator.userAgent. While this returns a string that does not explicitly state the device by name a regular expression match can be used to look for patterns that can confirm the device family. As an example the name ‘Tizen’ can be found in the user agent strings for Samsung and ‘netcast’ for LG devices. In the Chrome browser the strings will be different on Windows, Mac, Android and webview.

Another method similar to feature detection is to look for available APIs, many of these are unique to a device, for example if(tizenGetUser){ then do X }.

Besides the mentioned feature switches, device type identification can be crucial to determine which content should be loaded on a client device. A typical scenario for DRM protected content is to have different content types for different platforms based on the platforms playback capabilities. One might for instance package HLS and MPEG-DASH variants of the same content but using different encryption and DRM schemes. In that case the client application needs to decide which content to load based on the device type and its capabilities.

Unique device Identifier

After you have determined what type of device you are on you may need to identify that device uniquely. Most device manufacturers provide an API so once you have discovered what type of device you are on you then know what API’s will be available to you. Each manufacturer provides a string constructed in a different way, some use the serial number of the device itself while others use lower level unique identifiers taken from the hardware. In each case the application layer should attempt to namespace this in some way to avoid an unlikely, but theoretically possible, clash with another user in any server-side database of users and devices.
In a classic PC/browser, rather that mobile or STB (set-top-box), environment there are technical issues with using unique identifiers as a user can access these stored locally and either delete them or reuse them in another context, this could allow them to watch content on more devices than their user account allows. For this reason some vendors prefer to use DRM solutions that both create and encrypt unique identifiers in a way that obscures them from the user.

As mentioned above one typical use case where a unique device identifier is needed is to restrict the number of devices that a user can use. This could refer to overall devices or the number of devices a user can use in parallel. In both cases a unique device identifier will be required.

Client-side Ad Insertion

Client-side ad insertions typically involve using a script available in the client runtime, JavaScript in the case of the web, to insert an advertisement during the user’s session. CSAI can be used for any type of content, including on-demand and live.

For Live Streaming use cases, in-video-stream metadata (for example, using ID3 in HLS) can be used to enable all viewers to see an ad at the same time, synced with the content at a specific time offset. Another approach is to use web sockets to signal a player via a pushed event to play an advertisement.

Additionally, for live TV (broadcast TV), many broadcasters use SCTE-104/35 in their workflows to signal ads. When preparing the content to be distributed through Internet these SCTE-104/35 signals are usually "converted" to ID3/EMSG messages for HLS/DASH respectively. This keeps the media workflow for live ads insertion easy (SCTE to ID3/EMSG conversion is supported by most of encoders/media servers used by live content generators), is easy to consume from media players (most of players support parsing ID3/EMSG metadata), ensures synchronization with the content and avoids the need of using alternative communication channels like websocket that add complexity to the solution.

As an example workflow: an ad serving vendor typically provides a client-side script that a publisher can embed in their application. These scripts createa bridge between the advertising demand source and the publisher's video player. Close to the player’s initiation the client library makes its API available. The web application listens to events associated with playback, for example the video elements media event ‘playing’. The web application then calls the DOM pause method on the video element and then calls the play method provided by the client-side ad library, passing it the ‘id’ of the video asset. This is then returned to the vendor, possibly along with other identifiers that can be used to target the audience with a specific ad. At this stage an auction is performed with business logic at the ad vendor determining which provider supplies the ad (this is a complex topic and outside the scope of this document). The vendor responds with a VAST (Video Ad Serving Template) payload that includes the URI of the ad content appropriate for the playback environment. In some cases, there is no ad, if this is the case the user is presented with the content they originally requested and control is passed back to the web video application.

If there is an ad targeted against the content then the library performs DOM manipulation and injects a new video element into the document this is typically accompanied by a further script that provides the vendor with insights based on the current session. The ad plays. The ad object will conform to VPAID (Video Player Ad Serving Definition) and present a standardized interface to the player for possible interaction, it will issue a standard set of events which the web application can listen to. In response to an ‘adEnded’ event the local library will tear down the injected DOM elements and in turn issue an event that the web application can use to trigger a return to playing the original content.

Server Side Ad Insertion

Especially in live playback environments, because of the live nature of the playout, ad-insertions are usually not handle client side. One of the reasons is that the live feed continues independent of the client side ad insertion. In that case the player can easily fall behind the live window. Server side ad insertions allow the player to play a continuous feed, independent of any ad insertions, which are "injected" on the server side, directly into the playout feed.

One of the requirements of the underlying streaming format such as DASH or HLS is that the encoding does not change during the playout of a single rendition. To still be able to insert advertisements, HLS and DASH propose different strategies.

HLS allows the packager to add a "discontinuity" tag. This is expressed with a #EXT-X-DISCONTINUITY entry before any potential format change. See Section 4.3.2.3. in [[HLS]] for more detail. This tag then usually appears before the ad starts and again before the main content continues.

In contrast, DASH uses multiple "periods" to split the content into main and advertisement sections. See Section 3.1.31 in [[MPEGDASH]] for more details. Each switch between periods is a trigger for the player to potentially reset the decoder.

Where the player usually receives dedicated events in CSAI such as 'adStarted' or 'adEnded', Server-side insertions require a different setup to trigger such events. Typically, in-band notification are used to trigger time based events. For example, SCTE-35 markers are a common format to insert time based meta-data into the feed. These markers are read and interpreted by the player and can be used to trigger events or carry additional meta-data such as callback URLs.

The Packaged App

The Packaged App is a type of web app that distributed in its entirety to a user through a single download. Generally, we see packaged apps within App Markets. Packaged Apps have these characters:

• Downloaded in entirety through a store or market
• Generally offline by default, as all the code is living on the users device
• Retrieve data from the Internet via services similar to native apps. generally called via AJAX

Some Markets, such as Window Store and Amazon App Store enable you to submit packaged apps directly to the market without the need of any external packaging tools. These markets also expose additional APIs for packaged apps that aren’t available to apps via the browser. Other markets, such as iOS App Store and Android Play require a third party utility such as Cordova or Crosswalk to enable packaged web apps.

The Hosted App

Hosted Apps are web apps submitted to app markets that point to web content that remains on the webserver as appose to being packaged and submitted to the store. Hosted app’s have these Characteristics:

• Have a thin layer of native code, usually containing a webview and / or an app manifest that is submitted to the market
• Can run dynamic web content from web server (thank asp.net PHP, etc).
• By default have no functionality when device is offline (just like web)
• Hosted web apps are updated on the web server instead of pushing packages to the app store.

Progressive Web Apps

Progressive web apps or PWAs has become the de facto app format for the web. Just like hosted web apps, Progressive web apps load their content from the server when the app is opened. However, Progressive web apps also can store this web app locally through a new technology called service workers, which allows Progressive web apps to bridge across both the benefits of package web apps and hosted web apps. Some platforms run PWAs in the browser while others (like Android and Windows 10) run PWAs in a stand along app container.

PWAs are build around a common universal web app manifest that is being standardized by the W3C. The web manifest contains meta data about your application such as presentation preference, start URL, and even ratings and categories for store listing.

          {
            "lang": "en",
            "dir": "ltr",
            "name": "Donate App",
            "description": "This app helps you donate to worthy causes.",
            "short_name": "Donate",
            "icons": [{
              "src": "icon/lowres.webp",
              "sizes": "64x64",
              "type": "image/webp"
            },{
              "src": "icon/lowres.png",
              "sizes": "64x64"
            }, {
              "src": "icon/hd_hi",
              "sizes": "128x128"
            }],
            "scope": "/racer/",
            "start_url": "/racer/start.html",
            "display": "fullscreen",
            "orientation": "landscape",
            "theme_color": "aliceblue",
            "background_color": "red",
            "serviceworker": {
              "src": "sw.js",
              "scope": "/racer/",
              "use_cache": false
            },
            "screenshots": [{
              "src": "screenshots/in-game-1x.jpg",
              "sizes": "640x480",
              "type": "image/jpeg"
            },{
              "src": "screenshots/in-game-2x.jpg",
              "sizes": "1280x920",
              "type": "image/jpeg"
            }]
          }

        

You can learn more about building Progressive Web Apps here.

Mobile App container

For a web app to be distributed through the store on platforms like iOS and Android, an app container needs to be used. Popular app containers such as Cordova and Crosswalk are used by web developers to deliver a web app through a store. This approach often adds over head to the application as the web app is rendered in a webview or the app may contain the entire browser runtime within each app container.

Chrome Embedded Framework and Electron

In desktop environments, we see the use of Electron and Chromium Embedded Framework (CEF) to deliver web apps as desktop applications. CEF and Electron uses the open source version of chrome as a runtime environment for the application. Much like we see one mobile platforms, this approach has additional overhead as instead of using the native rendering engine of the platform, chromium is delivered inside each application to render the web app. This gives the web app additional privileges to access native API’s, but at the same time raises many security concerns.