W3C Data Platform

Introduction

This page is part of W3C project to use a data-centric approach to redesigning its site and services. If you have comments, please write to site comments. Note that this is a public archive of messages.

For particulars about the design of the API, and the implementation of a prototype for search, see data platform design.

People

Antonio, Denis, Jean-Gui, Ted, Ian, Daniel Davis

Scope

At the current time:

• Access to data about W3C specifications, groups, and events.
  • Out of scope for the first version: data about mail
• Still in scope but lower priority: individuals, organizations.
• JSON representation of data

Timeline

Depends on:

• Whether we can leverage some modules/extensions
• Changes to CG site based on beta

Rough estimate

Timeline:

• August, September 2014: Research into what we can use
• September 2014: Work starts in earnest
• December 2014: Crisp definition of APIs; templates for lists and metadata pages available
• February 2015: APIs available publicly

Requirements

Project Requirements

• The APIs must be developed in public.
• The APIs must be publicly documented.

Design Requirements

• Client-side functionality in key index pages:
  • For specifications: Filtering by group, title, tag
  • For groups: Filtering by title, tag
  • For events: filtering by title, tag
• APIs return JSON
  • To be formatted on the client side
  • In the future there may be APIs for getting HTML-formatted information (e.g., for inclusion on vertical pages)
• APIs must be able to take into account the user's credentials.
  • For example, for a given API we might provide Member-only additional information if the user is logged in with an appropriate account.
• Filter/sort states on the client should be URI addressable
  • We will need general purpose query strings
  • We may also want some friendlier URIs for common queries.
• There should be an "all" view of each kind of index (specs, groups, etc.) that does not rely on JavaScript.

Questions

• How will we identify individual specifications? Related specifications (e.g., short name)?
• How will we identify individual groups? Activities? Domains?

Server Requirements

• Our data servers must be extremely high performance (e.g., with strong caching in front).
  • Some symfony, some between that and public (like varnish)
• We need to consider unwanted bot traffic as we will have a larger number of URIs advertised.

Questions

• Our current framework sits on top of several databases that have some inconsistencies in field name conventions and other things which if exposed directly could confuse or frustrate developers. There are extensions to our framework that could greatly facilitate in creating this API but not with the best forward facing mapping. How much does this need to be cleaned up to make it more intuitive?
• W3C logic on publications workflow, group formation etc would be challenging to permit modification through the Web API (eg POST of revised or new JSON) even with authentication. What would be desirable to allow writes to? Individuals' information, with the exception of organization affiliation as that requires verification, is a candidate.
• Should the API URI scheme leverage our current (not visible to public at present) convention for groups, users and specs or establish a new scheme? It would be nice to use the existing for discoverability and consistency but a new URI space could be beneficial for caching, scalability, acl and other considerations.
• Do we want to be our own CDN?

Privacy requirements

• Will address when we get to individuals and members.

Use Cases

Specifications

• A developer wants to write a new app based on our data and requires all the data about all specifications.
  • Includes all Technical Reports of any status and all registered CG and BG reports.
• TR page views
  • Each view will be a list of Limited_Specification_Data restricted to WG/IG publications.
  • Each view will allow:
    • Sorting by date (ascending/descending), title (ascending/descending)
    • Filtering by group, title, tag
  • Views
    • Fifty most recently published specifications, and the ability to request the next 50.
    • All publications within a date range (e.g., for AC meeting fact sheet)
    • The most recent draft of every technical report.
• Group wishes to create custom display of information about its publications
  • Specifications published by a list of 1 or more identified groups, all data.
  • Specifications published by a list of 1 or more identified groups, Limited_Specification_Data
• Per specification
  • All data for an identified specification; this will be used to construct the "spec view" for each spec in the TR page.
  • Title query
  • Limited_Specification_Data for all specifications whose title at least partially matches (case-insensitive) a string.
• Tag query
  • Limited_Specification_Data for all specifications whose tags correspond exactly to an input list of tags. We will use this in the "major topic" views (e.g., show me specs of particular interest to the automotive industry).

Notes

• We are only planning limited query capabilities. For custom queries, use a series of individual queries.

Groups

• A developer wants to write a new app based on our data and requires all the data about all groups.
  • Includes WGs, IGs, CGs, BGs, Coordination Groups, TAG, and AB
• List of Groups page
  • A list of Limited_Specification_Data
  • Allow:
    • Sorting by title (ascending/descending)
    • Filtering by domain, title, tag
• Per group
  • All data for an identified group; this will be used to construct the "group view" for each group in the list
• Title query
  • Limited_Specification_Data for all groups whose title at least partially matches (case-insensitive) a string.
• Tag query
  • Limited_Specification_Data for all groups whose tags correspond exactly to an input list of tags. We will use this in the "major topic" views (e.g., show me groups of particular interest to the automotive industry).

Individuals

• Group participation
• Roles (e.g., editor of which spec)

Organizations

• @@

Events

• A developer wants to write a new app based on our data and requires all the data about all events.
  • Includes meetings and other events, publications, spec/charter review start and end dates, election deadlines, registration deadlines.
• W3C calendar
  • A list of Limited_Event_Data
• Title query
  • Limited_Event_Data for all events whose title at least partially matches (case-insensitive) a string.
• Tag query
  • Limited_Event_Data for all events whose tags correspond exactly to an input list of tags. We will use this in the "major topic" views (e.g., show me events of particular interest to the automotive industry).

APIs

Specifications

Groups

Events

Data

Specifications

• @IJ likely to rewrite this as a table with different profiles, including "all" and "limited"

Limited Specification Data

• Title
• Link to latest WD
• Publication date
• Status
• Review end date (if applicable)
• Identifier(s) of responsible group(s)
• Specification tags
• URI to get full data

Full Public Specification Data

Groups

Limited Group Data

• Title
• URI of group home
• Group type
• Identifier of Domain
• Group tags
• URI to get full data

Full Public Group Data

Additional Member Data

Events

Limited Event Data

• Title
• URI of event announcement (e.g., home page or news item)
• Event start/end dates
• Event location (may be empty)
• Event type
• Event tags
• URI to get full data

Full Public Event Data

Additional Member Data

Notes on Performance

• One idea is to create cached copies of data since it does not change that frequently. This can be server-side included (or accessed by URL).

Notes on Pages without Javascript

• When JS is turned off, we want to provide access to all the data, meaning we serve an HTML file. One technique for this is to use something like phantom.js to format the all.html page, which gets updated on a regular basis. This would allow us to reuse the same templates for "view some" as for "view all." PLH likes this approach but wants to be sure that this technique will be available in a decade.

Feeds/Notifications

• Twitter?
• Atom?

Daniel Davis URI notes

/collection/item

/collection/item/sub-collection

/collection/item?filter=value

• Consistency and predictability (which should make it future-proof)
• Uses existing URL strings for specs, groups, etc.

Examples

/groups - data for all groups
/groups/dpub - data for DigiPub group
/domains/ink - data for INK domain

/domains/ink/groups - data for all groups within INK domain

/domains/ink/groups?type=ig - data for all IGs within the INK domain
/specs/discovery-api - data for the Network Service Discovery API
/groups/dap/specs?status=wd - data for all WD specs in DAP WG

Discussion

• 9 September 2014 minutes

Resources

• Tantek/Elika work on spec styles