System Track Final Results
Contents
System Track Final Results: Overall
System
Total
Instance
Time
Final results by categories
Overview
P
NP
Beyond NP
System
Total Score
Inst. Score
Time Score
System
Total Score
Inst. Score
Time Score
System
Total Score
Inst. Score
Time Score
clasp
213
150
63
claspfolio
609
385
224
claspd
103
60
43
claspfolio
209
150
59
clasp
597
370
227
cmodels
72
45
27
claspd
206
145
61
idp
597
370
227
claspfolio
-
-
-
sup
195
140
55
claspd
552
355
197
clasp
-
-
-
lp2sat2gminisat
185
140
45
cmodels
510
335
175
idp
-
-
-
cmodels
184
130
54
lp2diffz3
394
270
124
lp2diffz3
-
-
-
idp
184
130
54
sup
346
240
106
sup
-
-
-
smodels
180
130
50
lp2sat2gminisat
310
225
85
lp2sat2gminisat
-
-
-
lp2sat2minisat
179
135
44
lp2sat2minisat
302
220
82
lp2sat2minisat
-
-
-
lp2diffz3
178
135
43
lp2sat2lminisat
301
220
81
lp2sat2lminisat
-
-
-
lp2sat2lminisat
171
130
41
smodels
269
165
104
smodels
-
-
-
P Problems
System
Total
Reachability
Grammar-Based IE
Hydraulic Leaking
Hydraulic Planning
Stable Marriage
Partner Units Polynomial
clasp
Total
213
36
72
10
75
6
14
Instance
150
25
50
10
50
5
10
Time
63
11
22
0
25
1
4
claspfolio
Total
209
35
72
10
74
5
13
Instance
150
25
50
10
50
5
10
Time
59
10
22
0
24
0
3
claspd
Total
206
36
72
10
75
0
13
Instance
145
25
50
10
50
0
10
Time
61
11
22
0
25
0
3
sup
Total
195
29
71
10
74
0
11
Instance
140
20
50
10
50
0
10
Time
55
9
21
0
24
0
1
lp2sat2gminisat
Total
185
30
66
10
68
0
11
Instance
140
20
50
10
50
0
10
Time
45
10
16
0
18
0
1
cmodels
Total
184
29
71
10
74
0
0
Instance
130
20
50
10
50
0
0
Time
54
9
21
0
24
0
0
idp
Total
184
29
71
10
74
0
0
Instance
130
20
50
10
50
0
0
Time
54
9
21
0
24
0
0
smodels
Total
180
28
70
10
72
0
0
Instance
130
20
50
10
50
0
0
Time
50
8
20
0
22
0
0
lp2sat2minisat
Total
179
30
66
10
68
0
5
Instance
135
20
50
10
50
0
5
Time
44
10
16
0
18
0
0
lp2diffz3
Total
178
35
66
10
67
0
0
Instance
135
25
50
10
50
0
0
Time
43
10
16
0
17
0
0
lp2sat2lminisat
Total
171
28
66
10
67
0
0
Instance
130
20
50
10
50
0
0
Time
41
8
16
0
17
0
0
NP Problems
System
Total
Sokoban Decision
Knight Tour
Disjunctive Scheduling
Packing Problem
Labyrinth
MCS Querying
Numberlink
Hanoi Tower
Graph Colouring
Solitaire
Weight-Assignment Tree
Maze Generation
claspfolio
Total
609
66
65
37
0
63
75
64
47
55
21
21
95
Instance
385
45
35
25
0
40
50
35
35
40
15
15
50
Time
224
21
30
12
0
23
25
29
12
15
6
6
45
clasp
Total
597
78
63
38
0
78
75
65
39
23
21
21
96
Instance
370
50
35
25
0
50
50
35
30
15
15
15
50
Time
227
28
28
13
0
28
25
30
9
8
6
6
46
idp
Total
597
64
74
38
0
52
75
70
65
18
38
8
95
Instance
370
45
45
25
0
30
50
40
45
10
25
5
50
Time
227
19
29
13
0
22
25
30
20
8
13
3
45
claspd
Total
552
68
68
30
0
65
75
69
31
19
11
20
96
Instance
355
45
40
25
0
45
50
40
25
10
10
15
50
Time
197
23
28
5
0
20
25
29
6
9
1
5
46
cmodels
Total
510
67
56
21
0
62
75
30
51
29
18
6
95
Instance
335
45
30
20
0
45
50
20
35
20
15
5
50
Time
175
22
26
1
0
17
25
10
16
9
3
1
45
lp2diffz3
Total
394
42
55
0
0
0
70
45
47
27
25
0
83
Instance
270
30
35
0
0
0
50
30
35
20
20
0
50
Time
124
12
20
0
0
0
20
15
12
7
5
0
33
sup
Total
346
52
40
37
0
58
72
0
31
16
15
25
0
Instance
240
35
25
25
0
40
50
0
25
10
10
20
0
Time
106
17
15
12
0
18
22
0
6
6
5
5
0
lp2sat2gminisat
Total
310
36
10
32
0
46
71
22
47
17
29
-
0
Instance
225
30
5
25
0
35
50
15
35
10
20
-
0
Time
85
6
5
7
0
11
21
7
12
7
9
-
0
lp2sat2minisat
Total
302
39
0
32
0
52
71
15
47
17
29
-
0
Instance
220
30
0
25
0
40
50
10
35
10
20
-
0
Time
82
9
0
7
0
12
21
5
12
7
9
-
0
lp2sat2lminisat
Total
301
35
0
32
0
53
71
17
47
17
29
-
0
Instance
220
30
0
25
0
40
50
10
35
10
20
-
0
Time
81
5
0
7
0
13
21
7
12
7
9
-
0
smodels
Total
269
0
55
36
0
9
53
27
0
0
0
0
89
Instance
165
0
30
25
0
5
35
20
0
0
0
0
50
Time
104
0
25
11
0
4
18
7
0
0
0
0
39
Beyond NP Problems
System
Total
Strategic Companies
Minimal Diagnosis
claspd
Total
103
12
91
Instance
60
10
50
Time
43
2
41
cmodels
Total
72
0
72
Instance
45
0
45
Time
27
0
27
Detailed Results
Details: raw file
Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.
Details: tables
Detailed results are also available as separate PDF files:
Quick reference guide
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.
- System: the system name (quite easy to understand)
- Benchmark: the benchmark problem
- Instance: the specific instance of the problem
- Out|Outcome: the answer provided by the system for the specific instance
- Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
- Time: the total time spent by the system while solving the specific instance
- Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
- Total Score: the total score rewarded for the specific instance
- Inst\Opt Score - Time Score: the two components of the score, as introduced above.
- WF (Cost) : Witness Found (Cost)
- OF (Cost) : Optimum Found (Cost)
- Inc : Inconsistent
- TO : Time Out
- MO/F : Mem Out Fail
- N/A : Not Available / Not Applicable