This document defines the format of the tests in the SHACL test suite, the process to use them to evaluate SHACL implementations, and lists test results from implementations that have submitted their test results.

Namespace prefixes

In this document we will employ the following namespace prefixes:

sh
http://www.w3.org/ns/shacl#
sht
http://www.w3.org/ns/shacl-test#
mf
http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#
earl
http://www.w3.org/ns/earl#

Test Cases Format and Process

Manifest format

The tests are available in the folder structure starting at tests. Each folder contains one or more manifest files that describe the tests in that folder.

The manifest file is defined in RDF and is compatible with the manifest used by other W3c initiatives like RDF 1.1 or SPARQL

<> a mf:Manifest ;
     rdfs:label "...Manifest label"
     rdfs:comment "...Manifest comment" ;
     dc:creator "...creator..."
     mf:entries
     (
     ....list of ManifestEntries
     ) .

Each manifest entry is a single test which can be one of the following types:

Validate RDF data

These tests verify that the validation of a given data graph against a given shapes graph produces a specified validation report. Both the data graph and the shapes graph may be identical.

Example:

<entry1> a sht:Validate ;
   mf:name "Validate simple RDF data with a simple shapes graph" ;
   mf:action [
        sht:shapesGraph <example-shapes.ttl> ; 
        sht:dataGraph <example-data.ttl> ; 
   ] ;
   mf:result [
        a sh:ValidationReport ;
        sh:conforms true ;
   ] ;
   mf:status sht:proposed .

To perform the test, a validation engine must validate the data graph specified by sht:dataGraph against the shapes graph specified by sht:shapesGraph. The values of both must be relative URLs pointing at a Turtle file in the same directory as the test file itself. It is also valid that either of these graphs point at the manifest graph itself, using the relative URL <>.

Implementations may report two levels of compliance:

For full compliance: Let R be the blank node that is the value of mf:result of the sht:Validate test case in the manifest graph. Let expected be the graph that consists of all triples from the manifest graph that have R as subject plus all triples that have all values of sh:result of R as their subject, as well as any triples needed to correctly represent the sh:resultPath.

Let actual be the results graph produced by the validation, i.e. the validation report. The test framework may post-process this results graph to ensure the following preconditions. Before comparing the results graph with the expected results, the actual validation report must not include any "nested" results, via sh:details. The actual results also must not depend on subclass relationships, so that instances of subclasses of sh:ValidationReport and sh:ValidationResult must have these as their (only) rdf:type values. Also, all instances of sh:ValidationResult and sh:ValidationReport must be blank nodes, i.e. if the engine produces URIs then these need to be replaced with blank nodes. Also, the blank node structures representing property paths via sh:resultPath must not be shared among multiple results, i.e. they may need to be cloned prior to comparison. Blank node structures representing property paths via sh:resultPath must not reuse the same path blank node (as values of sh:inversePath etc) in multiple places - these need to be normalized before the comparison, i.e. share blank node structures need to be cloned into new blank nodes. Furthermore, only the following predicates are used by triples in the expected graph and therefore any other triples need to be removed from the actual graph prior to comparison:

An exception is sh:resultMessage: As a general rule, all triples with sh:resultMessage as subject need to be removed from the actual graph, except those with object ?object for which the expected graph contains a triple ?any sh:resultMessage ?object. This makes sure that certain tests can verify that sh:resultMessage has been correctly populated from the sh:message triples in the shapes graph.

To pass full compliance, the actual validation report must be isomorphic to expected. If the test case has sht:Failure as its value for mf:result then the test is passed if the validation reported a failure. Alternatively, implementation may still report pass if they have manually checked that their validation reports satisfy the requirements specified by the SHACL specification. The latter also applies to scenarios that are well-formed SHACL validation results according to [[!shacl]] while the test framework has stricter expectations. For example, not all implementation are required to produce sh:sourceShape triples.

In order to improve precision of comparing test results, blank nodes should be avoided as values of sh:sourceShape and sh:sourceConstraint.

Test Creation, Approval and Modification Process

Tests are maintained by members of the Working Group in the GitHub repository. Anyone, including non-members, can submit or suggest test cases via GitHub pull requests. Submissions by other means may be rejected without response. Test cases that have not been accepted by the Working Group yet must have mf:status sht:proposed.

At the chair's discretion, individual tests or groups of tests are put to the Working Group, e.g. in the weekly telecon. The Working Group has complete discretion to approve or reject tests. Approved tests must have mf:status sht:approved, those that have been rejected may get marked as mf:status sht:rejected or deleted from the main branch of the repository. All approved tests must be reachable from the root manifest.mf file.

Editors may modify test cases, but SHOULD go through the WG process for non-trivial changes.

In addition to validation tests, we encourage users to submit examples of ill-formed shapes graphs. These may be used in the future to collect syntax tests.

Submitting Implementation Reports

Implementations listed here need to be submitted as a Turtle file in the format similar to the TopBraid SHACL API EARL Report. In a nutshell, we use [[EARL10-Schema]] to represent individual test results and metadata about the test subject. To indicate full compliance, use earl:passed as value for earl:outcome. To indicate partial compliance (i.e. can report true and false), use sht:partial.

The preferred way of submitting test reports is via a pull request into the reports folder on GitHub. Alternatively, send your Turtle file to the public comments mailing list.

Implementation Reports

This section summarizes the outcomes of test reports for submitted SHACL implementations.

Corese SHACL

Homepage: http://wimmics.inria.fr/corese
Tests Updated: 2017-05-04

Netage SHACL Engine

Homepage: http://www.netage.nl
Tests Updated: 2017-05-23

RDFUnit

Homepage: http://aksw.org/projects/RDFUnit
Tests Updated: 2017-10-18

TopBraid SHACL API

Homepage: https://github.com/TopQuadrant/shacl
Tests Updated: 2017-10-24

Validate RDF Data Tests

FileStatusTest CaseCoreseNetageRDFUnitTopBraid
98 / 121 (81%)100 / 121 (83%)82 / 121 (68%)121 / 121 (100%)
core/complex/personexampleapprovedTest of personexamplepassedpassedpassedpassed
core/complex/shacl-shaclapprovedfrozen eat your own ( eat your own frozen dogfood )passedno datapassedpassed
core/misc/deactivated-001approvedTest of sh:deactivated 001passedpassedpassedpassed
core/misc/deactivated-002approvedTest of sh:deactivated 002passedpassedpassedpassed
core/misc/message-001approvedTest of custom sh:message 001passedpassedpassedpassed
core/misc/severity-001approvedTest of sh:severity 001passedpassedpassedpassed
core/misc/severity-002approvedTest of sh:severity 002passedpassedpassedpassed
core/node/and-001approvedTest of sh:and at node shape 001passedpassedpartialpassed
core/node/and-002approvedTest of sh:and at node shape 002passedpassedpartialpassed
core/node/class-001approvedTest of sh:class at node shape 001passedpassedpassedpassed
core/node/class-002approvedTest of sh:class at node shape 002passedpassedpassedpassed
core/node/class-003approvedTest of sh:class at node shape 003 multiple classes, overlapping target setspassedpassedpartialpassed
core/node/closed-001approvedTest of sh:closed at node shape 001passedpassedpassedpassed
core/node/closed-002approvedTest of sh:closed at node shape 002passedpassedpassedpassed
core/node/datatype-001approvedTest of sh:datatype at node shape 001passedpassedpassedpassed
core/node/datatype-002approvedTest of sh:datatype at node shape 002passedpassedpassedpassed
core/node/disjoint-001approvedTest of sh:disjoint at node shape 001passedpassedpassedpassed
core/node/equals-001approvedTest of sh:equals at node shape 001passedpassedpassedpassed
core/node/hasValue-001approvedTest of sh:hasValue at node shape 001passedpassedpassedpassed
core/node/in-001approvedTest of sh:in at node shape 001passedpassedpassedpassed
core/node/languageIn-001approvedTest of sh:languageIn at node shape 001passedpassedpassedpassed
core/node/maxExclusive-001approvedTest of sh:maxExclusive at node shape 001passedpassedpassedpassed
core/node/maxInclusive-001approvedTest of sh:maxInclusive at node shape 001passedpassedpassedpassed
core/node/maxLength-001approvedTest of sh:maxLength at node shape 001passedpassedpassedpassed
core/node/minExclusive-001approvedTest of sh:minExclusive at node shape 001passedpassedpassedpassed
core/node/minInclusive-001approvedTest of sh:minInclusive at node shape 001passedpassedpassedpassed
core/node/minInclusive-002approvedTest of sh:minInclusive at node shape 002 - dateTime with timezonepassedno datapassedpassed
core/node/minInclusive-003approvedTest of sh:minInclusive at node shape 003 - dateTime without timezonepassedno datapassedpassed
core/node/minLength-001approvedTest of sh:minLength at node shape 001passedpassedpassedpassed
core/node/node-001approvedTest of sh:node at node shape 001passedpassedpartialpassed
core/node/nodeKind-001approvedTest of sh:nodeKind at node shape 001passedpassedpassedpassed
core/node/not-001approvedTest of sh:not at node shape 001passedpassedfailedpassed
core/node/not-002approvedTest of sh:not at node shape 002passedpassedfailedpassed
core/node/or-001approvedTest of sh:or at node shape 001passedpassedfailedpassed
core/node/pattern-001approvedTest of sh:pattern at node shape 001passedpassedpassedpassed
core/node/pattern-002approvedTest of sh:pattern at node shape 002passedpassedpassedpassed
core/node/qualified-001approvedTest of qualified parameters allowed in node shapespassedno datapassedpassed
core/node/xone-001approvedTest of sh:xone at node shape 001passedpassedfailedpassed
core/node/xone-duplicateapprovedTest of validation report for shape xone-duplicate by property constraintspassedno datafailedpassed
core/path/path-alternative-001approvedTest of path sh:alternativePath 001passedfailedpassedpassed
core/path/path-complex-001approvedTest of path complex (rdf:type/rdfs:subClassOf*) 001passedfailedpassedpassed
core/path/path-complex-002approvedTest of complex path validation resultspassedno datapartialpassed
core/path/path-inverse-001approvedTest of path sh:inversePath 001passedfailedpassedpassed
core/path/path-oneOrMore-001approvedTest of path sh:oneOrMorePath 001passedfailedpassedpassed
core/path/path-sequence-001approvedTest of path sequence 001passedfailedpassedpassed
core/path/path-sequence-002approvedTest of path sequence 002passedfailedpassedpassed
core/path/path-sequence-duplicate-001approvedTest of path sequence with duplicate 001passedfailedpartialpassed
core/path/path-strange-001approvedTest of strange path 001 two valid paths togetherpassedno datapassedpassed
core/path/path-strange-002approvedTest of strange path 002 valid and invalid paths togetherpassedno datapassedpassed
core/path/path-unused-001approvedTest with unused ill-formed pathpassedno datapassedpassed
core/path/path-zeroOrMore-001approvedTest of path sh:zeroOrMorePath 001passedfailedpassedpassed
core/path/path-zeroOrOne-001approvedTest of path sh:zeroOrOnePath 001passedfailedpassedpassed
core/property/and-001approvedTest of sh:and at property shape 001passedpassedpartialpassed
core/property/class-001approvedTest of sh:class at property shape 001passedpassedpassedpassed
core/property/datatype-001approvedTest of sh:datatype at property shape 001passedpassedpassedpassed
core/property/datatype-002approvedTest of sh:datatype at property shape 002passedpassedpassedpassed
core/property/datatype-003approvedTest of sh:datatype at property shape 003passedpassedfailedpassed
core/property/datatype-ill-formedapprovedTest of validation report for ill-formed literalspassedpassedpassedpassed
core/property/disjoint-001approvedTest of sh:disjoint at property shape 001passedpassedpassedpassed
core/property/equals-001approvedTest of sh:equals at property shape 001passedpassedpassedpassed
core/property/hasValue-001approvedTest of sh:hasValue at property shape 001passedpassedpassedpassed
core/property/in-001approvedTest of sh:in at property shape 001passedpassedpassedpassed
core/property/languageIn-001approvedTest of sh:languageIn at property shape 001passedpassedpartialpassed
core/property/lessThan-001approvedTest of sh:lessThan at property shape 001passedpassedpassedpassed
core/property/lessThan-002approvedTest of sh:lessThan at property shape 002passedpassedpartialpassed
core/property/lessThanOrEquals-001approvedTest of sh:lessThanOrEquals at property shape 001passedpassedpartialpassed
core/property/maxCount-001approvedTest of sh:maxCount at property shape 001passedpassedpassedpassed
core/property/maxCount-002approvedTest of sh:maxCount at property shape 002passedpassedpassedpassed
core/property/maxExclusive-001approvedTest of sh:maxExclusive at property shape 001passedpassedpassedpassed
core/property/maxInclusive-001approvedTest of sh:maxInclusive at property shape 001passedpassedpassedpassed
core/property/maxLength-001approvedTest of sh:maxLength at property shape 001passedpassedpassedpassed
core/property/minCount-001approvedTest of sh:minCount at property shape 001passedpassedpassedpassed
core/property/minCount-002approvedTest of sh:minCount at property shape 001passedpassedpassedpassed
core/property/minExclusive-001approvedTest of sh:minExclusive at property shape 001passedpassedpassedpassed
core/property/minExclusive-002approvedTest of sh:minExclusive at property shape 002passedpassedpassedpassed
core/property/minLength-001approvedTest of sh:minLength at property shape 001passedpassedpassedpassed
core/property/node-001approvedTest of sh:node at property shape 001passedpassedpartialpassed
core/property/node-002approvedTest of sh:node at property shape 002passedpassedpartialpassed
core/property/nodeKind-001approvedTest of sh:nodeKind at property shape 001passedpassedpassedpassed
core/property/not-001approvedTest of sh:not at property shape 001passedpassedfailedpassed
core/property/or-001approvedTest of sh:or at property shape 001passedpassedfailedpassed
core/property/or-datatypes-001approvedTest of sh:or of sh:datatypes at property shape 001passedpassedfailedpassed
core/property/pattern-001approvedTest of sh:pattern at property shape 001passedpassedpassedpassed
core/property/pattern-002approvedTest of sh:pattern at property shape 002passedpassedpassedpassed
core/property/property-001approvedTest of sh:property at property shape 001passedpassedpartialpassed
core/property/qualifiedMinCountDisjoint-001approvedTest of sh:qualifiedMinCount with disjoint shapes at property shape 001passedfailedfailedpassed
core/property/qualifiedValueShape-001approvedTest of sh:qualifiedValueShape at property shape 001passedpassedfailedpassed
core/property/qualifiedValueShapesDisjoint-001approvedTest of sh:qualifiedValueShapesDisjoint at property shape 001passedfailedfailedpassed
core/property/uniqueLang-001approvedTest of sh:uniqueLang at property shape 001passedpassedpartialpassed
core/property/uniqueLang-002approvedTest uniqueLang with other boolean literal for truepassedpassedpassedpassed
core/targets/multipleTargets-001approvedTest of multiple targets 001passedpassedpassedpassed
core/targets/targetClass-001approvedTest of sh:targetClass 001passedpassedpassedpassed
core/targets/targetClassImplicit-001approvedTest of implicit sh:targetClass 001passedpassedpassedpassed
core/targets/targetNode-001approvedTest of sh:targetNode 001passedpassedpassedpassed
core/targets/targetObjectsOf-001approvedTest of sh:targetObjectsOf 001passedpassedpassedpassed
core/targets/targetSubjectsOf-001approvedTest of sh:targetSubjectsOf 001passedpassedpassedpassed
core/targets/targetSubjectsOf-002approvedTest of sh:targetSubjectsOf 002passedpassedpartialpassed
core/validation-reports/sharedapprovedTest of validation report for shape shared by property constraintspassedpassedpassedpassed
sparql/component/nodeValidator-001proposedTest of sh:nodeValidator 001no datano datano datapassed
sparql/component/optional-001approvedTest of sh:optional 001no datapassedpassedpassed
sparql/component/propertyValidator-select-001approvedTest of sh:propertyValidator with SELECT 001no datapassedpassedpassed
sparql/component/validator-001approvedTest of sh:validator 001no datapassedpassedpassed
sparql/node/prefixes-001approvedTest of sh:prefixes 001no datapassedpassedpassed
sparql/node/sparql-001approvedTest of sh:sparql at node shape 001no datapassedpassedpassed
sparql/node/sparql-002approvedTest of sh:sparql at node shape 002no datapassedpartialpassed
sparql/node/sparql-003approvedTest of sh:sparql at node shape 003no datapassedpartialpassed
sparql/pre-binding/pre-binding-001approvedTest of pre-binding in FILTERno datapassedpartialpassed
sparql/pre-binding/pre-binding-002approvedTest of pre-binding in UNIONno datapassedfailedpassed
sparql/pre-binding/pre-binding-003approvedTest of pre-binding in inner {...} blocksno datapassedfailedpassed
sparql/pre-binding/pre-binding-004approvedTest of pre-binding in BIND expressionsno datapassedpartialpassed
sparql/pre-binding/pre-binding-005approvedTest of pre-binding in BGP and FILTERno datapassedfailedpassed
sparql/pre-binding/pre-binding-006approvedTest of pre-binding in nested SELECTno datapassedfailedpassed
sparql/pre-binding/pre-binding-007approvedTest of pre-binding in nested SELECTno datapassedfailedpassed
sparql/pre-binding/shapesGraph-001approvedTest of $shapesGraph and $currentShapeno datapassedfailedpassed
sparql/pre-binding/unsupported-sparql-001approvedTest of unsupported MINUSno datapassedpassedpassed
sparql/pre-binding/unsupported-sparql-002approvedTest of unsupported VALUESno datapassedpassedpassed
sparql/pre-binding/unsupported-sparql-003approvedTest of unsupported SERVICEno datapassedpassedpassed
sparql/pre-binding/unsupported-sparql-004approvedTest of unsupported SELECTno datapassedfailedpassed
sparql/pre-binding/unsupported-sparql-005approvedTest of unsupported AS ?preboundno datapassedpassedpassed
sparql/pre-binding/unsupported-sparql-006approvedTest of ASK trying to reassign ?valueno datapassedpassedpassed
sparql/property/sparql-001approvedTest of sh:sparql at property shape 001no datapassedpassedpassed

Status: Waiting for updated test results (one proposed test only)