Aggregated results by problem
In the tables below we provide the aggregated results of the solvers for each problem (SE, EE, DC and DS). The column Solver, #Corr, #Cra, #Inc, #Fail, #Inv, #TO, #OOM, Time, Memory, Score corresponds to: name of the solver, number of correct answers, number of crashes, number of incorrect answers, number of failures, number of invalid answers, number of timeouts, number of out of memory, running time (total), used memory (max) and final score, respectively. In detail, #Cra refers to the number of times the solver crashed due to memory issues, #Fail are all those runs that did not produce an answer (e.g., because of timeouts), #Inc refers to syntactically correct, but wrong, answers, and #Inv is the number of syntactically incorrect answers.
Back to all results
| SE | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 2279 | 0 | 0 | 0 | 0 | 3 | 0 | 7373.381 | 82718720 | 2279 |
| coquiaas | 2275 | 0 | 0 | 1 | 0 | 6 | 0 | 9804.689 | 97746944 | 2275 |
| mu-toksia | 2282 | 0 | 0 | 0 | 0 | 0 | 0 | 2085.454 | 93466624 | 2282 |
| pyglaf | 2281 | 0 | 0 | 0 | 0 | 1 | 0 | 11037.043 | 85348352 | 2281 |
| EE | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1590 | 0 | 34 | 0 | 5 | 1 | 0 | 5830.886 | 94019584 | 1397.918 |
| coquiaas | 1620 | 0 | 0 | 0 | 0 | 10 | 0 | 16179.240 | 97386496 | 1620 |
| mu-toksia | 1628 | 0 | 0 | 0 | 0 | 2 | 0 | 6456.604 | 90816512 | 1628 |
| pyglaf | 1627 | 0 | 0 | 0 | 0 | 3 | 0 | 10827.353 | 98836480 | 1627 |
| DC | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 2098 | 43 | 141 | 0 | 0 | 0 | 0 | 6760.712 | 823296000 | 1393 |
| coquiaas | 2276 | 0 | 0 | 1 | 0 | 5 | 0 | 8969.938 | 99647488 | 2276 |
| mu-toksia | 2282 | 0 | 0 | 0 | 0 | 0 | 0 | 2008.174 | 93822976 | 2282 |
| pyglaf | 1975 | 0 | 306 | 0 | 0 | 1 | 0 | 11283.370 | 88715264 | 445 |
| DS | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1593 | 2 | 35 | 0 | 0 | 0 | 0 | 4239.923 | 92950528 | 1418 |
| coquiaas | 1625 | 0 | 0 | 1 | 0 | 4 | 0 | 6730.054 | 98684928 | 1625 |
| mu-toksia | 1630 | 0 | 0 | 0 | 0 | 0 | 0 | 1415.499 | 71610368 | 1630 |
| pyglaf | 1595 | 0 | 35 | 0 | 0 | 0 | 0 | 5864.085 | 85712896 | 1420 |