This document provides definitions and best practices related to the specification of manifest files and similar document formats on the Web.

Some specifications on the Web deal with defining sets of files or resources to be consumed together. A common design pattern is to provide a manifest or configuration file that defines which resources are available and how various resources should be used or to provide various kinds of metadata about a collection of resources. This document provides definitions and best practices related to the specification of manifest files and similar document formats on the Web.

Introduction

Some specifications on the Web deal with defining sets of files or resources to be consumed together. A common design pattern is to provide a manifest or configuration file that defines which resources are available and how various resources should be used or to provide various kinds of metadata about a collection of resources.

Some examples of specifications like this include Webapp [[APPMANIFEST]], Miniapp [[MINIAPP-MANIFEST]], and ePub [[EPUB-33]].

Manifest file formats usually contain syntactic or user-supplied values that are not localizable. But most manifest file formats also contain at least some localizable content fields. A specification that defines a manifest file format that includes localizable content values needs to provide a means for users to localize the manifest into different languages. This document covers the pros and cons of common designs, along with some best practices related to building and managing manifests.

These (and many other) best practices, along with links to supporting materials, can also be found in the Internationalization Best Practices for Spec Developers [[INTERNATIONAL-SPECS]]. In addition to the best practices found here, additional best practices relating to language metadata on the Web can be found in [[STRING-META]]. Core terminology can also be found in the Internationalization Glossary [[I18N-GLOSSARY]].

Use Cases

There are different ways that manifest files might be used on the Web and these affect the choice of manifest localization strategy.

Packaged Manifest. Some manifests are packaged with other files to form an application or document. Since the manifest is generally consumed as part of the larger set of content and the content is downloaded as a single unit, manifests of this type can use more modular localization strategies, such as using multiple files with names or paths based on the locale.

Shorthand Description Manifest. Some manifests are meant to be used as a shorthand description of an application, document, or other resource. Manifests of this type are often downloaded by the client, sometimes as a precursor to retrieving the actual resource or its component parts. Since retrieving multiple files has latency, size, and storage implications, this type of manifest usually seeks to use a single file or other less-modular approaches.

Managing Language Preferences

Another consideration is the language negotiation strategy employed by the specification. If a manifest can be generated on demand (rather than being cached at a well-known URL), then the localization might be applied on a per-request basis. This points more toward a shorthand manifest style. In other cases, where some implementations might be able to negotiate while others cannot, it might be important for the specification to permit different strategies to be used.

Bear in mind that modern operating environments support quite extensive sets of available locales and that application owners often wish to satisfy diverse audiences with a single localized application or document. While examples in specifications necessarily are constrained to perhaps three or four locales, actual applications with 50 or more specific locales are not uncommon.

Manifests need to match a user's language preference to the available localized resources. The user's locale preference can include more than just a primary language subtag. It might include the script (in tags such as zh-Hans, meaning "Chinese as written in the Simplified Han script") or region (in tags such as fr-CA, meaning "French as used in Canada")—or both. Some systems support Unicode Locale Identifiers, an extension to [[BCP47]] that includes tailoring such as calendars, number shaping, collation, and more. For more information, see [[LTLI]].

Matching Language Tags and Locale Fallback

Matching localized resources to a user's locale preference usually involves "locale fallback", using a mechanism such as [[BCP47]]'s "lookup" matching scheme or [[CLDR]]'s "best match" scheme. These involve comparing the language tags on each resource seeking the longest tag that completely matches the user's preference. There is also, usually, a default locale to use for cases in which none of the localized resources match the user's preference.

Matching of language tags to available resources (such as found in [[[#path-vs-file]]]) or to items in some form of language map (see [[[#language-maps]]]) can require an implementation to examine all of the available values, as the first potential match might not be the best overall match.

For example, consider the following language map:

"localized-resource": {
	"en":     {"value": "This is English"},
	"en-GB":  {"value": "This is UK English"},
	"en-US":  {"value": "This is US English"}
}
			  

If the user's locale preference were en-GB or en-US, the best match is included in the list. Obviously, the entire list must be read to find the best match, since the entry for en could be considered a match for either value.

If the user's locale preference were en-AE (English as used in the United Arab Emirates), the best match in the list is en, although some implementations of best matching might apply out-of-band information to potentially match a longer tag.

If the user's locale preference were fr-FR (French as used in France), none of the values match and some form of defaulting would be needed to select the value to use.

Finally, the user's locale preference could include additional subtags, such as those used to tailor locale settings or such as a language variant. An example of a tailored locale might be en-US-u-hc-h23 (US English but with the "hour cycle" set to 24 hour time). An example of a language variant might be en-GB-oxendict (UK English, but using spelling conventions of the Oxford English Dictionary).

Locale fallback is implemented such that the user's preference is matched against the available list of values, and, if no exact match is found, removing the last subtag and repeating. This is described as "lookup" in the language tag matching part of [[BCP47]] (specifically [[RFC4647]]). Thus, the tag en-GB-oxendict would be compared against the above languge map as follows:

en-GB-oxendict  // no match
en-GB           // matches en-GB
en              // if en-GB were not present, matches en
(default)       // no match, use defaults
			  

Indicating Language and Direction

Natural language values stored in a manifest, like any document format or data structure, require language and direction metadata in order to be used by consumers successfully. For detailed information about the requirements, see [[STRING-META]].

There are four common serializations for string values in a manifest, and these are described in this section. Specifications are intended to use these together to form a complete localization solution.

Non-Linguistic Fields

For non-linguistic fields (that is, strings that contain data that is not human language) language or direction metadata SHOULD NOT be associated with the value. Note that this includes application-internal data values [[INTERNATIONAL-SPECS]].

If a consumer is required to assign a language tag to the data, the value zxx (Non-Linguistic) SHOULD be used. If a consumer is required to assign a base direction to the data, the value auto SHOULD be used.

Single-Language Localizable Text Field

For localizable text fields that appear in a single language, use a data structure to represent the value. The recommended representation is an object with three fields. The field value contains the actual string. The field lang contains a valid [[BCP47]] language tag. The field dir contains the string's base direction (one of the values ltr, rtl, or auto).

See The Localizable WebIDL Dictionary in [[STRING-META]].

Document-Level Defaults

Single-language localizable text fields are at their best if there are only a few natural language strings in a given document. When a document such as a manifest contains many natural language strings, this becomes inefficient. To reduce the complexity of encoding these strings, one workaround is to establish a document-level default for language and base direction. These are separate values, as language does not imply direction. There should still be the ability to override either value on any given string value by using the single-language localizable text field representation found above.

For specifications that can make use of the [[JSON-LD]] @context mechanism, use the @language and @direction fields to supply the document level defaults.

If your specification defines its own document level defaults, provide two optional fields:

Language Maps

The world is not monolingual. Having documents that contain only a single language would mean providing many iterations of the document, one for each language, in order to localize the manifest. This also might require language negotiation when requesting the manifest.

One way to address this is to allow multilingual values for each localizable text field inside the manifest document.

Language selection is not merely the exact matching of language tag string values to the user's preferred locale. The usual object representation of a localizable text field requires that the object be deserialized in order to discover the language tag associated with the value. This can be inefficient when there are many values in a given manifest file. In these cases, the best practice is to use a language map to organize localizable text values. Such a map exposes the language tag for the purposes of selection, but still uses an object representation on the value side of the map, since both language and direction might need to be overridden for a given string value.

Common Approaches

There are several common approaches to creating localized manifests. Each offers certain advantages (and suffers from certain drawbacks):

Unitary Manifest Files

A unitary manifest approach provides for localization by defining localizable fields using an array of language values.

The advantages of a unitary manifest include:

  1. The manifest is a single file, allowing for ease of downloading.
  2. It is easier to ensure that the correct version of the resources in every language are being used.
  3. Language defaulting may be easier to manage.

There are also disadvantages to this approach:

  1. If the manifest is rendered into many languages, the file's size may grow disproportionately.
  2. Localization processes are generally optimized to create files that are copies of the original, only with the language-bearing materials replaced. Creating, managing, and synchronizing localization in a highly multilingual file may be more complex than using separate files per language. Generating the manifest via scripts or tooling is one approach to managing this complexity.

Unitary manifests work best with smaller source files with a limited number of localizable content fields and/or a limited number of languages. Most unitary manifests define some type of “language indexing” scheme (such as found in [[JSON-LD]]) to perform the actual selection.

Language or Locale-specific Manifest Files

Locale-specific manifests create a separate file for each supported language or locale. Usually the specification will choose either a path or filename based organizational scheme for allocating and finding the manifest files.

The advantages of locale-specific manifests include:

  1. Managing the creation or addition of locales is generally easier. Localization processes are generally optimized for “creating a localized copy” of a given source file.

The disadvantages include:

  1. Even simple locale fallback chains can require the client to search for or retrieve multiple files.
  2. Because the files are separate, versioning of the content must be rigorously controlled.
  3. It may be necessary to test many more variations to ensure proper operation.

Hybrid Approaches

A common compromise is to have a central manifest file that contains most or all of the non-localizable content, coupled with separate files that contains the localizable content. In some cases, the central manifest file contains defaults for the localizable content. The central manifest might also contain a directory or list of available locales (or the URL of language-specific manifests) so that clients only need to try to retrieve localized files that are actually available.

Path versus File Name Localization

Some manifests opt to store localized variants of the manifest using modification of the path. Others prefer to modify the file name.

Path-based storage has the advantage that multiple files can be localized together. For example, the localized manifest can include a localized icon, logo, stylesheet, README, and terms-of-service file together with the manifest. New languages can be added by adding a new directory.

Path-based systems have the disadvantage that the filenames are generally “all the same” and the contents of a given file is not obvious from the filename—only by its location in the application hierarchy.

Filename-based storage has the advantage that the contents of the file is obvious from the filename and managing files can sometimes be easier in this regard.

Other Considerations

Note that manifest localization does not remove the need for the string content in a manifest to provide language and direction metadata for each separate data value. The manifest file can define a file-wide default for language and direction, but specific files should be able to override either value. See [[STRING-META]] for more information.

Some manifests allow for "sparse population" of values in the localized file. For example, the en-US (English, United States) file might only contain one or two US-specific values, relying on the en (English) file for most of its content. Specifications need to be explicit about whether sparse population is allowed and how it works to avoid misconfiguration errors or missing translations.