Aggregated results by semantics
In the tables below we provide the aggregated results of the solvers for each semantics (CO, PR, ST, SST, STG, GR and ID). The column Solver, #Corr, #Cra, #Inc, #Fail, #Inv, #TO, #OOM, Time, Memory, Score corresponds to: name of the solver, number of correct answers, number of crashes, number of incorrect answers, number of failures, number of invalid answers, number of timeouts, number of out of memory, running time (total), used memory (max) and final score, respectively. In detail, #Cra refers to the number of times the solver crashed due to memory issues, #Fail are all those runs that did not produce an answer (e.g., because of timeouts), #Inc refers to syntactically correct, but wrong, answers, and #Inv is the number of syntactically incorrect answers.
Results for CO Track: overview.
| CO | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 2873.198 | 94019584 | 1304 |
| coquiaas | 1302 | 0 | 0 | 0 | 0 | 2 | 0 | 5060.037 | 969428992 | 1302 |
| eqargsolver | 1166 | 0 | 0 | 0 | 0 | 138 | 0 | 97244.536 | 474550272 | 1166 |
| mu-toksia | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 1309.272 | 90816512 | 1304 |
| pyglaf | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 4660.723 | 50774016 | 1304 |
| taas-dredd | 987 | 10 | 203 | 0 | 0 | 104 | 0 | 77462.209 | 99794944 | -163.190 |
| yonas | 1200 | 0 | 0 | 0 | 0 | 86 | 18 | 238724.722 | 9999351808 | 1035.829 |
Results for PR Track: overview.
| PR | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1299 | 0 | 5 | 0 | 0 | 0 | 0 | 4346.610 | 92950528 | 1274 |
| coquiaas | 1301 | 0 | 0 | 0 | 0 | 3 | 0 | 6601.919 | 93622272 | 1301 |
| eqargsolver | 1166 | 0 | 0 | 0 | 0 | 138 | 0 | 96887.766 | 3063808 | 1166 |
| mu-toksia | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 1405.415 | 89145344 | 1304 |
| pyglaf | 1257 | 0 | 47 | 0 | 0 | 0 | 0 | 5656.699 | 82247680 | 1022 |
| taas-dredd | 863 | 15 | 236 | 0 | 0 | 190 | 0 | 143690.076 | 99987456 | -436.032 |
| yonas | 1009 | 177 | 1 | 0 | 0 | 94 | 23 | 363580.659 | 9979367424 | 836.678 |
Results for ST Track: overview.
| ST | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 2111.697 | 86528000 | 1304 |
| coquiaas | 1303 | 0 | 0 | 1 | 0 | 0 | 0 | 1978.757 | 65941504 | 1303 |
| eqargsolver | 1160 | 0 | 0 | 0 | 0 | 144 | 0 | 100895.672 | 3059712 | 1160 |
| mu-toksia | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 999.592 | 78954496 | 1304 |
| pyglaf | 1304 | 0 | 0 | 0 | 0 | 0 | 0 | 4863.809 | 98836480 | 1304 |
| taas-dredd | 791 | 13 | 278 | 0 | 0 | 222 | 0 | 170496.572 | 99676160 | -718.032 |
| yonas | 915 | 185 | 12 | 0 | 1 | 163 | 28 | 411195.085 | 9995333632 | 700.669 |
Results for SST Track: overview.
| SST | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1138 | 21 | 139 | 0 | 5 | 1 | 0 | 4868.853 | 59879424 | 428.699 |
| coquiaas | 1298 | 0 | 0 | 1 | 0 | 5 | 0 | 8137.338 | 94203904 | 1298 |
| mu-toksia | 1303 | 0 | 0 | 0 | 0 | 1 | 0 | 3226.112 | 88989696 | 1303 |
| pyglaf | 1303 | 0 | 0 | 0 | 0 | 1 | 0 | 6317.506 | 92397568 | 1303 |
Results for STG Track: overview.
| STG | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 1215 | 24 | 65 | 0 | 0 | 0 | 0 | 3841.746 | 87470080 | 882.219 |
| coquiaas | 1289 | 0 | 0 | 1 | 0 | 14 | 0 | 15855.341 | 99647488 | 1289 |
| mu-toksia | 1303 | 0 | 0 | 0 | 0 | 1 | 0 | 3191.954 | 75272192 | 1303 |
| pyglaf | 1300 | 0 | 2 | 0 | 0 | 2 | 0 | 7479.289 | 95862784 | 1290 |
Results for GR Track: overview.
| GR | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| aspartix19 | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 930.546 | 3080192 | 652 |
| coquiaas | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 434.905 | 3055616 | 652 |
| eqargsolver | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 256.632 | 3059712 | 652 |
| mu-toksia | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 238.776 | 3059712 | 652 |
| pyglaf | 360 | 0 | 292 | 0 | 0 | 0 | 0 | 1900.577 | 409772032 | -1100 |
| taas-dredd | 502 | 0 | 150 | 0 | 0 | 0 | 0 | 218.212 | 3059712 | -248 |
| yonas | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 5736.522 | 3055616 | 652 |
Results for ID Track: overview.
| ID | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Solver | #Corr | #Cra | #Inc | #Fail | #Inv | #TO | #OOM | Time | Memory | Score |
| argpref | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 4241.309 | 46161920 | 652 |
| aspartix19 | 648 | 0 | 1 | 0 | 0 | 3 | 0 | 5232.252 | 82718720 | 643 |
| coquiaas | 651 | 0 | 0 | 0 | 0 | 1 | 0 | 3615.624 | 86106112 | 651 |
| mu-toksia | 652 | 0 | 0 | 0 | 0 | 0 | 0 | 1594.610 | 93822976 | 652 |
| pyglaf | 650 | 0 | 0 | 0 | 0 | 2 | 0 | 8133.248 | 82640896 | 650 |