Well-deployed technologies
In games, the mouse may be used as a controller, e.g. to control the orientation of the character in a first person perspective 3D application. Games may use Pointer lock to avoid interactions between mouse events and the DOM, and to remove the cursor from view.
An increasing share of mobile devices relies on touch-based interactions. While the traditional interactions recognized in the Web platform (keyboard, mouse input) can still be applied in this context, a more specific handling of touch-based input is a critical aspect of creating well-adapted user experiences, which Touch Events in the DOM (Document Object Model) enable.
The Media Capture and Streams API gives access to cameras and microphones that games may use for various purposes, including as an input controller (e.g. user makes specific sounds to control game actions), an interaction mechanism (e.g. users have to put a specific object/barcode in front of the camera), contextual information (e.g. in Augmented Reality scenarios to render the user's surrounding in the background), or simply for cross-player communication purposes.
The Geolocation API provides a common interface for locating the device, independently of the underlying technology (GPS, Wi-Fi networks identification, triangulation in cellular networks, etc.).
The DeviceOrientation Event Specification defines several DOM events that provide information about the physical orientation and motion of a hosting device. Most browsers support this specification, although various interoperability issues have arised. The work on the specification itself has been discontinued when the Geolocation Working Group was closed. The Devices and Sensors Working Group is now chartered to reduce inconsistencies across implementations and finalize the specification accordingly, and develop the more powerful Orientation Sensor specification in parallel.
The Vibration API lets game developers take advantage of haptic feedback to create new forms of interactions.
The User Agent Accessibility Guidelines (UAAG) 2.0 note defines principles and guidelines for user agents to design an accessible user agent interface and communicate with assistive technologies. The supporting document UAAG 2.0 Reference explains the intent and best practices of UAAG 2.0 success criteria, and lists numerous examples for each of them.
Following the Web Content Accessibility Guidelines (WCAG) 2.1 will make content accessible to a wider range of people with disabilities. The 2.1 revision adds new success criteria and guidelines to version 2.0, including new criteria related to user input that have a specific resonance in games, such as the Pointer Gestures and Target Size criteria.
Web content developers may benefit from authoring tools that follow the Authoring Tool Accessibility Guidelines (ATAG) 2.0 standard, which provides guidelines for designing Web content authoring tools that are both more accessible to authors with disabilities and that help design content that conforms to WCAG.
The Accessible Rich Internet Applications (WAI-ARIA) 1.1 standard provides an ontology of roles, states, and properties that define the semantics of user interface elements and that can be used to improve the accessibility and interoperability of Web content and applications. The Core Accessibility API Mappings 1.1 standard describes how user agents should expose these semantics to accessibility APIs.
Technologies in progress
The Gamepad specification defines a low-level interface that exposes gamepad devices attached to the browsing device.
The WebXR Device API exposes VR/AR-specific input, including tracked controller state and hand gesture.
The Pointer Events Working Group has made good progress on an alternative approach to handle user input, Pointer Events, that allows to handle mouse, touch and pen events under a single model. It provides a complementary and more unified approach to the currently more widely deployed Touch Events.
In particular, the CSS property touch-action that lets filter gesture events on elements is gaining traction beyond implementations of Pointer Events.
The early proposal for an Input Device capabilities API would provide information about whether a given “mouse” event comes from a touch-capable device.
The Generic Sensor API defines a framework for exposing sensor data to the Web platform in a consistent way. In particular, the specification defines a blueprint for writing specifications of concrete sensors along with an abstract Sensor interface that can be extended to accommodate different sensor types.
A number of sensor APIs are being built on top of the Generic Sensor API. The Proximity Sensor specification defines an API to monitor the presence of nearby objects without physical contact. The Ambient Light Sensor specification defines an API to monitor the ambient light level or illuminance of the device's environment..
The detection of motion is made possible by a combination of low-level and high-level motion sensor specifications, also built on top of the Generic Sensor API:
- the Accelerometer to obtain information about acceleration applied to the device's local three primary axes;
- the Gyroscope to monitor the rate of rotation around the device's local three primary axes;
- the Magnetometer to measure magnetic field around the device's local three primary axes;
- the Orientation Sensor to monitor the device's physical orientation in relation to a stationary 3D Cartesian coordinate system.
The Motion Sensors Explainer document is an introduction to low-level and high-level motion sensors, their relationship, inner workings and common use-cases.
The Geolocation Sensor is an API for obtaining geolocation reading from the hosting device. The feature set of the Geolocation Sensor is similar to that of the Geolocation API, but it is surfaced through the Generic Sensor API, improves security and privacy, and is extensible.
As depth-sensing cameras come to the market, the Media Capture Depth Stream Extensions allows capturing of the additional dimension in information they produce.
Game players enjoy recording and sharing their performance on social networks. The Media Capture from DOM elements API lets games turn content rendered in the application into a media stream for recording or transmission. More generally, content produced by any other application (or the whole display) can also be turned into such a stream via the Screen Capture API.
Any media content captured, generated or received as a stream can be recorded via the Media Recorder API.
The Mediastream Image Capture specification caters for more advanced usage of cameras that offer still-pictures settings (e.g. to control zoom or white-balance).
Whether players are speaking commands to the game or interacting with them through non-haptic mechanisms, they risk seeing the screens turned off automatically by their devices screensaver. The Screen Wake Lock API lets developers signal the needs to keep the screen up in these circumstances.
Exploratory work
Games may want the user to interact with physical objects to improve the experience. The Web Near-Field Communications (NFC) API enables wireless communication between two devices at close proximity. Similarly, the Web Bluetooth specification describes an API to discover and communicate with devices over the Bluetooth Low Energy (BLE) mode.
Detecting specific objects from a video stream is hard and CPU-intensive. Beyond traditional video processing, modern GPUs often provide advanced vision processing capabilities (e.g. face and objects recognition) that would have direct applicability e.g. in augmented reality applications. The Shape Detection API is exploring this space.
Mobile devices are also in many cases well-suited to be used through voice-interactions; the Speech API Community Group developed a JavaScript API to enable interactions with a Web page through spoken commands. Speech synthesis is well supported across browsers. Support for speech recognition is still underway.
The Input Method Editor (IME) API provides Web applications with scripted access to an IME (input-method editor) associated with a hosting user agent. Editorial support is required for this specification to move forward.
Features not covered by ongoing work
- Gesture events
- As mentioned above, touch-based interaction is common on mobile devices and available to Web applications through Touch events. Gesture-based interaction, which includes pinching, rotating and swiping, is also a common interaction paradigm on mobile devices. Web developers may derive gesture events from touch events to some extent, but may have to develop multiple versions for different browsers. Native support for these interactions would reduce fragmentation and improve performance. Early discussions to define Gesture events have started in the Merging of Web and Mobile APP Community Group.
Discontinued features
- Intent-based events
- As the Web reaches new devices, and as devices gain new user interactions mechanisms, it seems useful to allow Web developers to react to a more abstract set of user interactions: instead of having to work in terms of “click”, “key press”, or “touch event”, being able to react to an “undo” command, or a “next page” command independently of how the user instructed it to the device. The IndieUI Events specification was an attempt to address this need. The work has been discontinued for now, due to lack of support from would-be implementers.