Sesame compliance with RDF Test Cases of August 1

The following is an overview of Sesame's[3] compliance with the
test cases as mentioned in [0], along with editorial comments on
the format and presentation of the test cases.
Parser Test cases:
------------------
We have run Sesame's RDF parser from Sesame 0.9.2 against the new
RDF test cases of 25 July 2003[1]. Of the 170 APPROVED parser test
cases, 167 pass and 3 fail.
The 3 failing test cases are:
* rdfms-seq-representation/test001.rdf
 This test fails because Sesame's RDF parser generates
 (_:nodeId, <rdf:type>, <rdf:List>) triples for every bNode that
 is a list. This is conform the specification of 23 January 2003,
 but this has been changed in the editor's draft. The parser will
 be adapted to the new specs once they are officially released.
* rdf-charmod-literals-error001.rdf and
 rdf-charmod-literals-error002.rdf
 This is a known issue in Sesame's RDF parser: it accepts all
 unicode encodings and not just Normal Form C. From Peter F.
 Patel-Schneider's e-mail to rdf-comments [2] we conclude that
 some discussions are going about dropping this constraint from
 RDF. We're waiting for the outcome of this discussion before
 we're starting fixing this issue.
Entailment test cases:
----------------------
We have not yet run a full test run with the Entailment test cases
mentioned in [0]: to be honest the RDF Core WG caught us
off-guard, we were unaware that such a test case set was being
prepared until last friday. Negligence on our part I presume.
Having examined the test cases, we are somewhat surprised that in
fact most test cases seem to deal with borderline details of
datatype entailment, and that there are virtually no tests
whatsoever concerning more mainstream entailment issues, like
class- and property subsumption, cycles in hierarchies, domain and
range restrictions, etc.
Furthermore there are several test that require further
clarification: based on the premise and conclusion data sets it is
often confusing to understand why a particular entailment is or is
not valid.
As a point in case: the tests datatypes/test004(a/b) are both
mentioned as positive and negative tests. It is not explained why
this is so in [0].
The results presented here are not automatically tested, we have
examined the test cases manually, tried out those where we were
unsure how Sesame would react, and for the others the results are
our opinion on how Sesame _would_ react.
Of the 17 approved positive tests, Sesame (0.9.2) passes 7, fails
8, and 2 are unknown.
The 8 failed tests are:
 - datatypes/test007a -> test007b
 - datatypes/test004b -> test004c
 - datatypes/test005a -> test005b
 - datatypes/test003a -> test003b
 - datatypes/test003b -> test003a
 - datatypes/test008a -> test008b
 These all deal with related issues of comparing datatype values
 to each other. Since a lot of the work concerning datatype
 entailments seems to be more or less in flux, and a number of
 editorial changes are anticipated, we will not fix these until
 a new release of the RDF Semantics WD.
- rdfms-seq-representation/empy -> test002
- rdfms-seq-representation/empy -> test004
 These cases demands that an empty graph
 entails that rdf:_1 is a containermembershipproperty, and a
 subproperty of rdfs:member. The Sesame inferencer is currently
 implemented in such a way that the rdf:_1 (etc.) properties
 are only added to the inference process when a container is
 added to the repository. Thus, the inference is only made
 later.
- xmlsch-02/test001 -> test003
 According to this test a well-formed XML-Schema-datatyped
 literal entails an empty Literal node. We are unclear how this
 follows from the RDF MT.
The 2 unknown tests are those tests that do not present a
"conclusion" document, but merely state FALSE. We are unsure how
to interpret these tests.
Of the 17 approved negative tests, Sesame (0.9.2) passes 11, fails
1, and 3 unknown.
The 3 unknown tests are:
 - datatypes/test004a -> test004b
 - datatypes/test004b -> test004c
 Both these tests are also mentioned as positive tests,
 therefore it is not quite clear what the expected behaviour of
 the system in fact should be.
 - datatypes/test002 -> test002b
 It is simply unclear to us at this moment what aspect of
 entailment this test is supposed to capture.
Remark: for some reason two tests (statement-entailment/test001a
and statement-entailment/test002a) are listed twice. Sesame passes
the tests, but we are unsure how to interpret the double listing.
The 1 failed test is:
 - xmlsch-02/test002 -> test003
 According to the description the supplied literal is not
 well-formed, but it is not explained why. Clarification in the
 documentation of the test case might be useful.
Editorial remarks:
We feel that the current set of entailment test cases 
over-emphasizes the importance of datatype reasoning, and that in
many projects and practical applications, the importance of
correct subsumption reasoning is at least as high, if not much
higher. The set of test cases should reflect that.
Second, we feel that the application of the entailment test cases
is severely under-documented.
We trust this overview is of some help, and welcome your feedback
on our first (admittedly somewhat hurried) findings.
Regards,
Jeen Broekstra and Arjohn Kampman
References:
[0]http://www.w3.org/2001/sw/RDFCore/TR/WD-rdf-testcases-20030117/
[1]http://www.w3.org/2000/10/rdf-tests/rdfcore/
[2]http://lists.w3.org/Archives/Public/www-rdf-comments/2003JulSep/0152.html
[3]http://sesame.aidministrator.nl/
-- 
jeen.broekstra@aidministrator.nl
aidministrator nederland b.v. - http://www.aidministrator.nl/
prinses julianaplein 14-b, 3817 cs amersfoort, the netherlands
tel. +31-(0)33-4659987 fax. +31-(0)33-4659987

Received on Monday, 4 August 2003 11:05:54 UTC

AltStyle によって変換されたページ (->オリジナル) /