Manifest format

The tests are available in the folder structure starting at tests. Each folder contains one or more manifest files that describe the tests in that folder.

The manifest file is defined in RDF and is compatible with the manifest used by other W3c initiatives like RDF 1.1 or SPARQL

<> a mf:Manifest ;
     rdfs:label "...Manifest label"
     rdfs:comment "...Manifest comment" ;
     dc:creator "...creator..."
     mf:entries
     (
     ....list of ManifestEntries
     ) .

Each manifest entry is a single test which can be one of the following types:

• Validate RDF data

Validate RDF data

These tests verify that the validation of a given data graph against a given shapes graph produces a specified validation report. Both the data graph and the shapes graph may be identical.

Example:

<entry1> a sht:Validate ;
   mf:name "Validate simple RDF data with a simple shapes graph" ;
   mf:action [
        sht:shapesGraph <example-shapes.ttl> ; 
        sht:dataGraph <example-data.ttl> ; 
   ] ;
   mf:result [
        a sh:ValidationReport ;
        sh:conforms true ;
   ] ;
   mf:status sht:proposed .

To perform the test, a validation engine must validate the data graph specified by sht:dataGraph against the shapes graph specified by sht:shapesGraph. The values of both must be relative URLs pointing at a Turtle file in the same directory as the test file itself. It is also valid that either of these graphs point at the manifest graph itself, using the relative URL <>.

Implementations may report two levels of compliance:

• partial compliance means that an implementation is able to perform conformance checking only, i.e. produces the same boolean value for sh:conforms as required by the expected report.
• full compliance means that an implementation can produce a complete validation report that can be compared with the expected validation report as described in the remainder of this section.

For full compliance: Let R be the blank node that is the value of mf:result of the sht:Validate test case in the manifest graph. Let expected be the graph that consists of all triples from the manifest graph that have R as subject plus all triples that have all values of sh:result of R as their subject, as well as any triples needed to correctly represent the sh:resultPath.

Let actual be the results graph produced by the validation, i.e. the validation report. The test framework may post-process this results graph to ensure the following preconditions. Before comparing the results graph with the expected results, the actual validation report must not include any "nested" results, via sh:details. The actual results also must not depend on subclass relationships, so that instances of subclasses of sh:ValidationReport and sh:ValidationResult must have these as their (only) rdf:type values. Also, all instances of sh:ValidationResult and sh:ValidationReport must be blank nodes, i.e. if the engine produces URIs then these need to be replaced with blank nodes. Also, the blank node structures representing property paths via sh:resultPath must not be shared among multiple results, i.e. they may need to be cloned prior to comparison. Blank node structures representing property paths via sh:resultPath must not reuse the same path blank node (as values of sh:inversePath etc) in multiple places - these need to be normalized before the comparison, i.e. share blank node structures need to be cloned into new blank nodes. Furthermore, only the following predicates are used by triples in the expected graph and therefore any other triples need to be removed from the actual graph prior to comparison:

• rdf:type sh:ValidationReport
• sh:result
• sh:conforms
• rdf:type sh:ValidationResult
• sh:focusNode
• sh:resultPath (including depending blank node structures)
• sh:resultSeverity
• sh:sourceConstraint
• sh:sourceConstraintComponent
• sh:sourceShape
• sh:value

An exception is sh:resultMessage: As a general rule, all triples with sh:resultMessage as subject need to be removed from the actual graph, except those with object ?object for which the expected graph contains a triple ?any sh:resultMessage ?object. This makes sure that certain tests can verify that sh:resultMessage has been correctly populated from the sh:message triples in the shapes graph.

To pass full compliance, the actual validation report must be isomorphic to expected. If the test case has sht:Failure as its value for mf:result then the test is passed if the validation reported a failure. Alternatively, implementation may still report pass if they have manually checked that their validation reports satisfy the requirements specified by the SHACL specification. The latter also applies to scenarios that are well-formed SHACL validation results according to [[!shacl]] while the test framework has stricter expectations. For example, not all implementation are required to produce sh:sourceShape triples.

In order to improve precision of comparing test results, blank nodes should be avoided as values of sh:sourceShape and sh:sourceConstraint.

Test Creation, Approval and Modification Process

Tests are maintained by members of the Working Group in the GitHub repository. Anyone, including non-members, can submit or suggest test cases via GitHub pull requests. Submissions by other means may be rejected without response. Test cases that have not been accepted by the Working Group yet must have mf:status sht:proposed.

At the chair's discretion, individual tests or groups of tests are put to the Working Group, e.g. in the weekly telecon. The Working Group has complete discretion to approve or reject tests. Approved tests must have mf:status sht:approved, those that have been rejected may get marked as mf:status sht:rejected or deleted from the main branch of the repository. All approved tests must be reachable from the root manifest.mf file.

Editors may modify test cases, but SHOULD go through the WG process for non-trivial changes.

In addition to validation tests, we encourage users to submit examples of ill-formed shapes graphs. These may be used in the future to collect syntax tests.

Submitting Implementation Reports

Implementations listed here need to be submitted as a Turtle file in the format similar to the TopBraid SHACL API EARL Report. In a nutshell, we use [[EARL10-Schema]] to represent individual test results and metadata about the test subject. To indicate full compliance, use earl:passed as value for earl:outcome. To indicate partial compliance (i.e. can report true and false), use sht:partial.

The preferred way of submitting test reports is via a pull request into the reports folder on GitHub. Alternatively, send your Turtle file to the public comments mailing list.

Corese SHACL

dotNetRDF

Netage SHACL Engine

pySHACL

RDFUnit

shaclex

TopBraid SHACL API

Validate RDF Data Tests