Credibility Signals

1.1 Purpose

This document is intended to support an ecosystem of interoperable credibility tools.  These software tools, which may be components of familiar existing systems, will gather, process, and use relevant data to help people more accurately decide what information they can trust online and protect themselves from being misled. We expect that an open data-sharing architecture will facilitate efficient research and development, as well as an overall system which is more visibly trustworthy.

The document has three primary audiences:

1. Software developers and computer science researchers wanting to build systems which work with credibility data.  For them, the document aims to be a precise technical specification, stating what they need for their software to interoperate with any other software which conforms to this specification.
2. People who work in journalism and want to review and contribute to this technology sphere, to help make sure it is beneficial and practical.
3. Non-computer-science researchers, interested in helping develop and improve the science behind this work.

In general, we intend for this document to be:

• Welcoming for implementers of systems using credibility data
• Easy for non-tech folks to understand the proposed signals & contribute
• Practical to maintain by the editors
• Practical to contribute to, for a wide audience
• A source of accurate guidance about signal quality and adoption

1.2 Credibility Data

The document builds on concepts and terminology explained in Technological Approaches to Improving Credibility Assessment on the Web.  Our basic model is that an entity (human and/or machine) is attempting to make a credibility assessment — to predict whether something will mislead them or others — by carefully examining many different observable features of that thing and things connected with it, as well as information provided by various related or trusted sources.

To simplify and unify this complex situation, with its many different roles, we model the situation as a set of observers, each using imperfect instruments to learn about the situation and then recording their observations using simple declarative statements agreed upon in advance. Because those statements are inputs to a credibility assessment process, we call them credibility signals.  (The term credibility indicators is sometimes also used.)

This document, then, is a guide to these signals.  It states what each observer might say and exactly how to say it, along with other relevant information to help people choose among the possible signals and understand what it means when they are used.

Because this is a new and constantly-changing field, we do not simply state which signals should be used.  Instead, we list possible signals that one might reasonably consider using, along with information we expect to be helpful in making the decision.

1.3 Example

[explain]

Assessing credibility of https://news.example/article-1

   Looking at title

      I consider it to be clickbait

      It's clickbait because it's a cliffhanger

   Looking at article

      It cites scientific research

   Looking at provider

      Established in 1974

      Owned domain since 2006

1.4 Factors in Selecting Signals

When building systems which use credibility signals and trying to decide which signals to use, there are different factors to weigh.  This section is aspirational; we hope this document will in time provide guidance on all these factors.  

1.5 Publishing Credibility Data

TBD, basically follow schema.org technique using json-ld.

1.6 Consuming Credibility Data

TBD, point to some tools and the relevant specs.  Basically JSON-LD.

1.7 Organization of this document

Section 1 (“Introduction”) provides instructions for how to use and help maintain this document, along with general background information.

The rest of this document, after the introduction, is a list of signals and information about them, as discussed in the introduction.  The signals are organized into related groups, in hierarchical sections.  The lowest level of the hierarchy is the signals themselves; all the higher levels provide grouping of the signals, to help people understand them.

One important level of the hierarchy identifies the subject type of the signal.  This is the conceptual entity being examined, considered, or inspected, when one makes the observation being recorded in the signal data.  This could be imagined in different ways: when you are observing a claim made in the 3rd paragraph of an article published in some newspaper, are you observing the claim, the paragraph, the article, the newspaper, or even the author of the article?  In general, we aim for the smallest granularity that makes sense, which in this case would probably be the claim.

At times, it may not be obvious to which subject type a signal belongs, or it could sensibly belong with several different ones.  In this case, it might be moved around in the document as people come to understand it better.  When it’s not clear, there should be links from the places a signal could reasonably be to the place it actually is.

this may require discussion, and might remain open for debate.  When a signal or group of signals makes sense in two places, consider linking it from the places it isn’t, to help people find it.

In many cases, a signal could be seen as a set of similar signals which are not strictly identical. This can be handled by adding additional signal headings with the finer distinction, when necessary. In this case, template statements might appear under more than one signal.

Note that sections may be moved and renumbered.  Do not rely on section numbers remaining the same.  For linking to a part of the document, consider using the gdocs h.xxxxxx fragment ids, provided by the Table of Contents; those should remain stable.  Also, whenever changing a heading, especially a signal heading, if someone might be referring to it by name, please move there old text into a paragraph starting “Also called:”.  

1.8 Template Statements

The most important thing about a signal definition is to be clear what observation the signal data is recording. If the signal heading is “Article length”, does that mean length in words or bytes or characters or some other metric? Does it include the title? For each signal, we want an easy way to communicate its definition that is short but clear, while being as detailed as necessary.

The technique we use here is to express the semantics of the signal using plain and simple sentences in natural language which convey the same knowledge as the signal data. If you imagine people using credibility software exchanging these statements (perhaps in text messages or on Twitter), you should get the right semantics. You can assume metadata, like who sent it and when it was sent is available, so the statements include terms like “I” and “now”.

For machine-to-machine data Interoperability, these template sentences and the signal heading are turned into a data schema, after which the JSON-LD/schema.org/sematic web/linked data technology stack can be used.

The statements we use are templates because they abstract over a variety of similar sentences which differ in specific limited ways.  For example, these statements:

1. I have examined the article at https://example.com/alice and find it highly credible
2. I have examined the article at https://example.com/brian and find it highly credible
3. I have examined the article at https://example.com/casey and find it highly credible

are all the same, except in the URL.   We convey this using a template statement, which has a variable portion in square brackets, like:

• I have examined the article at [subject] and find it highly credible

The bracketed template expression “[subject]” is is required in every template, to indicate what entity is being observed.  Additional bracket expressions can be used when there are other elements of the statement to make variable.  In particular, [string] (for text in quotes) and [number].

(For now, try to just use those three.  Software and documentation is being developed to allow more features. If you find this too restrictive, go ahead and write something else inside the square brackets and we'll deal with it later, but include a question mark so it's clear you knew you were making it up.)

An example needing multiple variables:

1. https://example.com/alice took 4.75 seconds to load, just now.
2. https://example.com/brian took 5.9 seconds to load, just now.

could be matched by:

• [subject] took [number] seconds to load, just now.

1.9 Instructions for editing this document

As an experiment, this document is currently set so everyone can edit it, like Wikipedia. It is the Google docs version that is editable. We suggest you change the “Editing Mode” to “Suggesting” (using the pencil icon in the upper-right) until you are quite familiar with this document. You may also comment using the usual Google Docs commenting features.

If you make or suggest any edits to this document, you are agreeing to the W3C Community Contributor License Agreement which has significant copyright and patent implications.

The subsections below give some advice for how to make edits which are helpful.

1.10 Contributors

Folks who add content to this document are encouraged to add themselves in this section, potentially with some affiliation & credential information.  This also allows the “By” column to stay short, as people can use short forms of names (eg only first or last name, if unique in this doc).

• Entries marked as by “Credibility Coalition” are prior work by members of the Credibility Coalition. At the time, individual authorship information was not maintained. Moving forward, specific authorship detail is welcome.
• Entries marked as by “Public Editor” are prior work by members of Public Editor. At the time, individual authorship information was not maintained. Moving forward, specific authorship detail is welcome.
• (add yourself here...)

2.1 Claim Review

The “ClaimReview” model developed at schema.org grows out of the tradition of independent, external fact-checking, as in PolitiFact.  With this model, a fact-checker reviews a claim, typically made by a public figure, and then publishes a review of that claim, a “claim review”. Within schema.org, this parallels other reviews, like restaurant reviews.

[ Can we fit claimreview neatly into this observer/signal model?  It’s a bit of a stretch.  TBD. ]

7.1 Originality

7.2 Rhetoric

Issue: Are these really about the article, or at they about the text? Would the observations be the same if the same text appeared in a different article?  Unclear.  Leave in Article for now.

7.3 Logic/Reasoning

7.4 Outbound References

8.1 Quality

These are general quality signals, not containing many details about why something might be high or low quality.

8.2 Misleading about content

8.3 Misleading about the world

8.4 Nonmisleading consumer manipulation

9.1 Layout

Issue: Should this be a Heading1 like Title?  Probably no, because the statements naturally get phrased with the subject of the statements being a web page. Most people wouldn't conceptualize the page layout as its own entity.

9.2 Typefaces

9.3 Metadata in transmission headers

9.4 Metadata in page head

9.5 Metadata inline in body

10.1 Advertisements