Coastguard/Mountain Rescue

A missing person is reported to the rescue services, who deploy a drone to search inaccessible areas of coastline or moorland for their target. The drone relays back a live video stream from its camera and geolocation data from its GPS receiver to a remote human operator who is piloting it.

As the search continues, the operator spots a target on the video feed and can instantly call up an electronic map, synchronised to the video, which has been automatically following the drone’s position and plotting its ground track. The display gives the operator immediate context for the video, and allows them to override the automatic map control and zoom in to pinpoint the target’s precise location from the features visible in the video and on the map/satellite view. They mark the location and then zoom out to assess the surrounding terrain and advise the recovery team of the best approach to the target. For example, the terrain may dictate very different approach routes if either the person has twisted their ankle at the top of a cliff, or has fallen and is lying at the bottom of the same cliff, though the co-ordinates are almost identical in both cases.

The operator has been able to make important decisions quickly, which may be life critical, and deploy recovery resources effectively.

• Rapid decision making;
• Effective resource deployment.

Area Survey

A survey drone is equipped with a camera which records an image of the ground directly below it. The pilot is a remote human operator, tasked with surveying a defined area from particular height in order to capture the required data.

As the survey progresses, geometric shapes are automatically added to the map to represent areas which have been included. Once the drone has finished its sweep, the operator can quickly confirm whether the required area has been completely covered. If any areas have been missed, the pilot can use the map to navigate and make additional passes to fill the gaps, before returning to base.

Adding map track (VMT) files to the survey archive provides a geospatial index to the videos, allowing a particular geographic location to be found more rapidly by virtue of their small file size. Online video archives can be indexed more quickly using this web-friendly format.

The operator has been easily able to verify the quality of their own work and correct any errors, saving time and additional effort in redeployment. Video footage has been indexed by geolocation rapidly and in a search-engine-friendly format.

• Autonomous quality assurance;
• Cost saving;
• Rapid archive indexing for search engines.

Outdoor Trails

An outdoor sportsperson, e.g. snowboarder or cyclist, is equipped with a helmet camera and/or mobile phone to record video footage and GPS data. They set off to find new challenges and practice their skills, e.g. off-piste or on mountain trails, and discover new routes and areas that they would like to explore in future, chatting to the camera as they go. Afterwards, they upload the video to share their experience with the online community, so others can quickly identify locations of particularly interesting sections of the featured trail. Using the synchronised map view in their browser, community members can easily see where they need to go in order to try it for themselves.

The operator has been able to fully engage in their sporting activity, without making any written notes, while simultaneously recording the details needed to guide others to the same locations. Their changing location over time can also be used to calculate speed and distance information, which can be displayed alongside the footage.

• Non-invasive capture;
• Information sharing;
• Speed and distance calculation.

TV Sports Coverage

A TV production company is covering a sports event that takes place over a large area, e.g. rallying, road cycling or sailing, using a number of mobile video devices including competitor cams, e.g. dash cams or helmet cams, and drones to provide shots of inaccessible areas, e.g. remote terrain or over water.

Feeds from all the cameras are streamed to the production control room, where their geolocation data are combined on a map showing the locations of every competitor and camera, each labelled for easy identification. The live map enables the director to quickly choose the best shot and anticipate where and when to deploy their drone cameras to catch competitors at critical locations on the course as the competition develops in real time.

Multiple operators can function concurrently, both autonomously and under central direction. Mobile assets can be monitored and deployed from an operations centre to provide optimum coverage of the developing live event.

• Multiple mobile video devices;
• Real time asset management.

Proxy Explorer

Important details of a remote area have been captured on video. It is not possible to revisit the location for safety reasons or because it has physically changed in the intervening time. Footage can be retrospectively geotagged against a concurrent map to allow the viewer to better interpret and identify features seen in the footage. Explanatory annotations can be added to the video-map track to help future viewers' understanding and aggregate the collective analysis.

Multiple operators can contribute their observations to provide a group analysis, iteratively adding new details and discarding out-of-date information. Experts can offer insight about filmed locations, which would otherwise be inaccessible to them.

• Remote analysis of inaccessible locations;
• Knowledge aggregation and sharing for archive footage.

Treasure Hunt

A TV production company designs a new game show which involves competitors searching for targets across a wide area, with an operations centre remotely monitoring their progress and providing updates. Competitors are equipped with body-worn video or helmet cameras to relay footage of their view.

Geolocation context allow central operators to better understand the participants' actions and to remotely direct them more efficiently. Competitors' positions can displayed to the TV audience on annotated 2D- or 3D-maps for clearer presentation.

• Speed and distance calculation;
• Knowledge aggregation and sharing for real-time footage.

Swarm Monitoring

A swarm of drones is deployed to perform a task, and their operations are monitored centrally. Geolocation details of the swarm are automatically collated and broadcast to the drone pilots, showing the locations of all the drones and each is circled with a suitable safety zone to warn their operators in case two units find themselves flying in close proximity.

Pilots are safely able to operate either autonomously or under the direction of central control. Extra zonal information can be added to the operators' maps to show the outer perimeter of their operating area and warn of fixed aerial hazards, e.g. a radio mast, or transient hazards, e.g. a helicopter.

• Static and dynamic hazard indication;
• Central swarm monitoring;
• Autonomous swarm monitoring.

Crisis Response

Disaster strikes, e.g. hurricane, tsunami, and emergency response teams are deployed to the affected area. However, it is difficult to verify which problems people are facing, what resources would help them and exactly where these events are occurring. Maps are unreliable as the infrastructure has been damaged, though people on the ground have the relevant knowledge if it could be reliably recorded and shared.

Anyone with a basic smartphone could video events with reliable geospatial data, as GPS receivers can operate without the need for a mobile phone signal by using satellite data, to accurately document the problems they face. Even if the cell network is not operational, this information can be physically delivered to crisis coordinators to notify them of the issues that need to be addressed, including an accurate location in a common format. Crisis events can be reliably recorded, knowledge can be shared and aggregated, and relief resources can be accurately targeted and deployed to the correct locations.

• Reliable dispersed information gathering and sharing;
• Accurate resource deployment.

Fastest Lap

An amateur racer attempts to set a fastest lap time in a vehicle equipped with a dashcam. The film can be reviewed with details of location and vehicle telemetry to analyse its performance, and that of the driver, to identify where improvements can be made. The ideal racing line can be determined and measured against the actual line taken, and optimum braking points can be established by suggesting and testing corrections.

Driver's commentary can be used to identify performance issues along with vehicle audio response, e.g. engine tone and tyres, for comparison with telemetry data to rapidly iterate testing using community analysis tools. Results can be shared online in a common format to display metrics and to demonstrate veracity.

• Data synchronisation with video and audio;
• Motion analysis using community tools.

Police Evidence

A web-based police system is established to allow dashcam video evidence of driving offences to be submitted digitally by members of the public who have witnessed them. Detectives are able to identify the time and vehicles involved directly from the uploaded footage, and accurately determine the location at which the incident occurred from the digital metadata included.

Its ability to accept open format data also makes the system available to cyclists and pedestrians who can record video with location on their helmet cameras and smartphones respectively, providing wider access to the service beyond the dashcam community. Metadata, e.g. location, from different video manufacturers is often recorded in mutually-incompatible formats, but video-map track support enables synchronised location (and other) data to be extracted from recordings using manufacturers' or community tools, without affecting source video integrity, and submitted to the police system in a common format, significantly reducing development costs.

Officers have been able to identify incident locations quickly and accurately, without sacrificing evidence integrity. The online service has been made available to a wider audience of drivers, cyclists and pedestrians, without incurring additional development costs.

• Accurate location with evidence integrity preserved;
• Development costs reduced;
• Service extended to a wider audience.

Area Monitoring

An area of interest is monitored operationally by a collection of different mobile video devices, e.g. drones, body-worn video, helicopter, etc. Video footage, possibly in different formats, is added to an archive with location (and other) metadata in a common format which forms a time-location index suitable for rapid parsing by a web crawler. Users can submit online queries to search by location and return a time-ordered sequence of video frame stills captured within a radial distance of the chosen location.

Video archives can be quickly indexed using a common metadata format regardless of video encoding, e.g. MPEG, WebM, OGG, and video files are only accessed in case of a positive search result, which reduces bandwidth in comparison to embedded metadata. Linked files also allow different security permissions to be applied to the crawling and querying processes, so an AI algorithm can be authorised to read metadata without being able to access image content if there are security concerns over data privacy, e.g. illicit facial recognition.

• Homogenized video metadata from disparate sources;
• Reduced search bandwidth;
• Structured security support;
• Web search engine compatible.

Vehicle Collision

Dashcam footage is searched to automatically identify vehicle collisions from impact acceleration profiles recorded in video metadata. Dashcam manufacturers typically embed metadata in an unpublished format and provide a proprietary video player to allow users to display it. Exporting embedded metadata to a linked file in a web-friendly format enables searchable video archive data to be shared quickly and easily, without affecting evidence integrity, and to be accessed through a common web interface.

Vehicles can be automatically monitored using a low-cost dashcam and web-based tools to ensure that collisions are accurately recorded by drivers and that commercial vehicles remain safe and undamaged. Interoperability means that users are not limited to a particular dashcam brand and can share evidence with insurers and the police in a common format without damaging its integrity.

• Accurate vehicle collision detection;
• Common format for data sharing;
• Web search support;
• Evidence integrity preserved.

Golden Tutorial

Augmented reality (AR) software is used to control assets or view content in situ at a particular location. For example, nearby street lights can be switched off or on by a service engineer for maintenance purposes, or an architect can see how their structural design integrates with the surrounding landscape at its proposed location before any building work has started.

Video footage can be recorded with location, camera orientation and other metadata so AR overlays be generated on demand. Such recordings can be used to demonstrate how AR content is displayed and controlled in order to educate users with a 'golden tutorial', to provide 'proof of action' as evidence of work done for auditing purposes, or to create example data for AR software testing and debugging.

• Accurate AR video and data recording;
• Improved AR software development.

Virtual Guide

A user triggers an audio track which provides guidance about the local area or instruction for a known object, e.g. Web of Things (WoT) device at that location. The audio timeline is synchronised with events that can display AR content, control WoT devices and display points of interest on a map which provide guidance with real world context by highlighting places or objects of interest and showing possible actions.

Users can be guided by a virtual assistant through an area of interest or sequence of actions augmented with AR/VR and WoT devices to visualise events and by an annotated map or model to provide additional geospatial context. Greater insight is given to the user by showing detailed views of the location on a map or internal structure of the identified object using a virtual model.

• Contextual guidance provided in situ;
• Concurrent operation with AR/VR video;
• Integration with Web of Things.

Current Solutions

Material Exchange Format (MXF) was developed by the Society of Motion Picture and Television Engineers (SMPTE) to synchronise metadata, including geolocation, with audio and video streams using a register of key-length-value (KLV) triples. The breadth of its scope has resulted in interoperability issues, as different vendors implement different parts of the standard, and has produced implementations from high-profile companies which are mutually incompatible. KLVs can also be embedded within MPEG files, though this does not address the synchronisation issue for other web video formats such as WebM.

Video camera manufacturers have taken various approaches, resulting in a number of non-standard solutions including embedding geolocation data within the MPEG metadata stream in disparate formats, e.g. Go-Pro Metadata Format (GPMF), or recording a separate geolocation file in a proprietary format alongside the associated video file. From a hardware perspective, a few high-end cameras provide geotagging out of the box and all require an add-on device to support this feature.

Geospatial data are not currently accessible in the video Document Object Model (DOM) in HTML nor via video playback APIs in smartphones, e.g. Android, though their host devices are typically equipped with both a video camera and Global Navigation Satellite System (GNSS) receiver capable of capturing the required information.

In sharp contrast, still photos have a well-established geotagging standard called Exif, which was published by the Japan Electronic Industries Development Association (JEIDA) in 1995 and defines a metadata tag to embed geolocation data within TIFF and JPEG images. This is widely supported by manufacturers of photographic equipment and software worldwide, including low-end smartphones, making this feature cheap and accessible to the public.

Growing Requirements

Historically, there has been no requirement for a comparable video standard, but the urgency for such a standard is growing fast due to the emerging markets for 'mobile video devices,' e.g. drones, dashcams, body-worn video and helmet cameras, as well as the rise in high-quality video and geolocation support in the global smartphone market.

Accessible Standard Opportunity

Using current W3C recommendations, it is possible for a competent programmer to synchronise video-geolocation 'metadata' with a <video> element using a <track> child element. However, this is a non-trivial development task which requires an understanding of the video DOM events and Javascript file handling, making it inaccessible to the vast majority of web users. In addition, video map tracks are a clearly identified metadata subclass, which could be isolated in a similar way to video text tracks.

Establishing a standard file format would allow interoperability and information sharing between the public, the emergency services, police and other mobile video device users, e.g. drone pilots, giving cheaper and easier access to this important source of information. If web browsers supported video geotagging natively using this file format, it would also become accessible to most web users. Current low-end smartphones already provide suitable hardware to concurrently capture video and geolocation streams, which would make this technology easily accessible to the general public, and encourage the user and developer communities to grow rapidly.

Map Cues

Map cues display their payload between a start time and end time. The end cue time may be omitted to signify the end time of the video, which may be unknown in the case of streamed video.

WEBVMT

MEDIA
url:TowerBridge.mp4
mimetype:video/mp4

MAP
lat:51.506,lng:-0.076
rad:250

00:00:02.000 --> 00:00:05.000
{ "moveto":
  { "lat": 51.504362, "lng": -0.076153 }
}
{ "lineto":
  { "lat": 51.506646, "lng": -0.074651 }
}
          

WEBVMT

MEDIA
url:../movies/TowerOfLondon.webm
mimetype:video/webm

MAP
lat:51.162,lng:-0.143
rad:20000

00:00:03.000 -->
{ "panto":
  { "lat": 51.508, "lng": -0.077, "end": "00:00:05.000" }
}

00:00:06.000 -->
{ "zoom":
  { "rad": 250 }
}
          

Comments

Comments are blocks that are preceded by a blank line, start with the word NOTE (followed by a space or newline), and end at the first blank line.

WEBVMT

NOTE Associated video

MEDIA
url:/home/myuser/movies/TowerLandmarks.ogg
mimetype:video/ogg

NOTE Map config

MAP
lat:51.506,lng:-0.076
rad:500

NOTE Tower Bridge

00:00:01.000 --> 00:00:05.000
{ "moveto":
  { "lat": 51.504362, "lng": -0.076153 }
}
{ "lineto":
  { "lat": 51.506646, "lng": -0.074651 }
}

NOTE City Hall

00:00:02.000 -->
{ "circle":
  { "lat": 51.504789, "lng": -0.078642, "rad": 20 }
}

NOTE Tower Of London
This line is also part of the comment

00:00:03.000 --> 00:00:04.000
{ "polygon":
  { "points":
    [ { "lat": 51.507193, "lng": -0.074844 },
      { "lat": 51.508756, "lng": -0.074716 },
      { "lat": 51.509036, "lng": -0.075638 },
      { "lat": 51.508929, "lng": -0.077162 },
      { "lat": 51.507727, "lng": -0.077848 },
      { "lat": 51.507220, "lng": -0.075767 }
    ]
  }
}
          

Styling

Display style is controlled by CSS, which may be embedded in HTML or within the VMT file.

<!doctype html>
<html>
  <head>
    <title>WebVMT Style Example</title>
    <style>
      video::cue-map {
        stroke-color: red;
        stroke-opacity: 0.9;
      }
    </style>
  </head>
  <body>
    <video controls width="640" height="360">
      <source src="video.mp4" type="video/mp4">
      <track src="maptrack.vmt" kind="metadata" for="vmt-map" tileurl="https://api2.ordnancesurvey.co.uk/mapping_api/v1/service/zxy/EPSG%3A3857/Outdoor%203857/\{z}/{x}/{y}.png?key=VALID_OS_KEY">
      Your browser does not support the video tag.
    </video>
    <div id="vmt-map" style="height: 360px; width:640px;"></div>
  </body>
</html>
          

WEBVMT

MEDIA
url:http://example.com/movies/Greenwich.mp4
mimetype:video/mp4

MAP
lat:51.478,lng:-0.001
rad:50

STYLE
::cue-map {
  stroke-color: red;
}

NOTE Comments are allowed between style blocks

STYLE
::cue-map {
  stroke-opacity: 0.9;
}

NOTE Prime Meridian marker

00:00:00.000 -->
{ "moveto":
  { "lat":51.477901, "lng": -0.001466 }
}
{ "lineto":
  { "lat":51.477946, "lng": -0.001466 }
}

NOTE Style blocks may not appear after the first cue
          

Animation

Map annotations may be animated using an animate command, in a similar way to the <animate> element in SVG.

Paths have additional properties to other annotations including a unique identifier, and an animation which can be controlled separately for distance calculation purposes.

WEBVMT

NOTE Associated video

MEDIA
url:LondonBrighton.mp4
mimetype:video/mp4

NOTE Map config

MAP
lat:51.1618,lng:-0.1428
rad:20000

NOTE London overview

00:00:01.000 -->
{ "panto":
  { "lat": 51.4952, "lng": -0.1441 }
}

00:00:02.000 -->
{ "zoom":
  { "rad": 10000 }
}

NOTE From London Victoria...

00:00:03.000 -->
{ "panto":
  { "lat": 50.830553, "lng": -0.141706, "end": "00:00:25.000" }
}
{ "moveto":
  { "lat": 51.494477, "lng": -0.144753, "path": "cam1" }
}
{ "lineto":
  { "lat": 51.155958, "lng": -0.16089, "path": "cam1", "end": "00:00:10.000" }
}

NOTE ...via Gatwick Airport...

00:00:10.000 -->
{ "lineto":
  { "lat": 50.830553, "lng": -0.141706, "path": "cam1", "end": "00:00:25.000" }
}

NOTE ...to Brighton (at 00:00:25.000)

00:00:27.000 -->
{ "zoom":
  { "rad": 20000 }
}
          

WEBVMT

NOTE Associated video

MEDIA
url:SafeDrone.mp4
mimetype:video/mp4

NOTE Map config

MAP
lat:51.0130,lng:-0.0015
rad:1000

NOTE Drone starts at (51.0130, -0.0015)

00:00:05.000 -->
{ "panto":
  { "lat": 51.0070, "lng": -0.0020, "end": "00:00:25.000" }
}
{ "moveto":
  { "lat": 51.0130, "lng": -0.0015, "path": "drone1" }
}
{ "lineto":
  { "lat": 51.0090, "lng": -0.0017, "path": "drone1",
    "end": "00:00:10.000" }
}

NOTE Safety zone

00:00:05.000 --> 00:00:10.000
{ "circle":
  { "lat": 51.0130, "lng": -0.0015, "rad": 10,
    "animate":
    [ { "name": "lat", "to": 51.0090, "end": "00:00:10.000" },
      { "name": "lng", "to": -0.0017, "end": "00:00:10.000" }
    ]
  }
}

NOTE Drone arrives at (51.0090, -0.0017)

00:00:10.000 -->
{ "lineto":
  { "lat": 51.0070, "lng": -0.0020, "path": "drone1", "end": "00:00:25.000" }
}
{ "circle":
  { "lat": 51.0090, "lng": -0.0017, "rad": 10,
    "animate":
    [ { "name": "lat", "to": 51.0070, "end": "00:00:25.000" },
      { "name": "lng", "to": -0.0020, "end": "00:00:25.000" }
    ]
  }
}

NOTE Drone ends at (51.0070, -0.0020)
          

YouTube Integration

Embedded YouTube content can be displayed using an <iframe> element, specifying the unique 10-character content identifier for the posted video, using the official YouTube IFrame API with the Javascript API enabled.

<!doctype html>
<html>
  <head>
    <title>WebVMT YouTube Example</title>
  </head>
  <body>
    <!-- Video display -->
    <iframe src="http://www.youtube.com/embed/YOUTUBE_VIDEO_ID?enablejsapi=1" width="640" height="360" frameborder="0">
      <track src="maptrack.vmt" kind="metadata" for="vmt-map" tileurl="mapbox://styles/mapbox/streets-v9">
    </iframe>
    <!-- Map display -->
    <div id="vmt-map" style="height: 360px; width:640px;"></div>
  </body>
</html>
          

WEBVMT

NOTE Associated YouTube video

MEDIA
url:http://www.youtube.com/embed/YOUTUBE_VIDEO_ID
mimetype:video/mp4
          

WebVMT Cues

A WebVMT cue is a text track cue with an optional end time and that additionally consists of the following:

A cue text

The raw text of the cue which is interpreted as time-aligned metadata, and rules for its interpretation.

A WebVMT cue without an end time indicates that the cue end time is unbounded, for example during live streaming when the time of the next data sample is unknown or when the duration of the media is unknown.

WebVMT Map

A WebVMT map is the map viewport and provides a rendering area for WebVMT cues.

A WebVMT map consists of:

A map center latitude

The latitude of the location at the center of the map.

A map center longitude

The longitude of the location at the center of the map.

A map zoom radius

The radius in metres of the minimum area visible from the center of the map.

A map object

The control interface object for the map.

The precise format of the map object control interface is implementation dependent, for example the OpenLayers API.

WebVMT Media

A WebVMT media is the linked media data with which WebVMT cues are synchronised, for example audio or video.

A WebVMT media enables a web crawler to rapidly search media metadata by providing sufficient information to construct a time-metadata index of the linked media file without opening it. Search engine data throughput is reduced as only matching media files selected by the user need be read, and non-matching media files are not accessed at all. Care should be taken to maintain WebVMT media details correctly, for example when a media file is renamed.

A WebVMT media consists of:

A media URL

The URL of the linked media file.

A null media URL indicates that no linked media file exists.

A media MIME type

The MIME type of the linked media file.

A null media MIME type indicates that no linked media file exists.

A media start time

The global time and date at which the linked media file begins.

The media start time allows multiple WebVMT files to be aggregated. A null media start time indicates that no start time is associated, for example in the case of an animation.

A media path

The path identifier which uniquely identifies the moving object capturing the linked media file.

A null media path indicates that no moving object is associated, for example when no linked media file exists.

WebVMT Command Structures

A WebVMT command is an instruction to display WebVMT metadata content.

A WebVMT command consists of one of the following components:

• A WebVMT map control command;
• A WebVMT shape annotation command;
• A WebVMT path.

WebVMT commands are executed in order from first to last in the WebVMT file.

Planned Features

This section lists potential features which have been identified during the development process, but have not yet matured to a full design specification.

Features which appear in this section warrant further investigation, but are not guaranteed to appear in the final specification.

Planned Interfaces

This section lists interfaces which have been identified during the development process, but have not yet matured to a full design specification.

              [Exposed=Window,
              Constructor(double startTime, double endTime, DOMString text),
              Constructor(double startTime, DOMString text)]
              interface VMTCue : DataCue {
                attribute unrestricted double endTime;
              };
            

              [Exposed=Window,
              Constructor]
              interface VMTMap {
                attribute double centerLat;
                attribute double centerLng;
                attribute double zoomRad;
                attribute object getMap;
              };