Results
| run | problems × samples | accuracy % | cut at the cap % | mean tokens | correct when finished % | cut answers ending in repetition % | gain vs baseline (pp) | rescued / lost problems |
|---|---|---|---|---|---|---|---|---|
| bench/base/gsm8k | 1319×2 | 14.2 ± 0.8 | 87.7 | 29,370 | 97.5 | 98.3 | - | - |
| bench/base/ifeval | 541×2 | 21.2 ± 1.4 | 73.5 | 24,976 | 78.7 | 96.1 | - | - |
| bench/base/math500 | 500×4 | 7.3 ± 0.9 | 93.3 | 30,956 | 100.0 | 96.2 | - | - |
| bench/p50c/gsm8k | 1319×2 | 66.9 ± 1.2 | 6.3 | 11,742 | 71.3 | 29.7 | +52.7 ± 1.1 | 711 / 2 |
| bench/p50c/ifeval | 541×2 | 63.4 ± 1.8 | 1.4 | 3,984 | 64.3 | 80.0 | +42.2 ± 2.0 | 261 / 22 |
| bench/p50c/math500 | 500×4 | 51.8 ± 1.9 | 31.0 | 21,950 | 75.0 | 34.7 | +44.5 ± 1.7 | 263 / 0 |
| build/base | 128×2 | 11.7 ± 2.5 | 89.8 | 29,872 | 100.0 | 96.5 | - | - |
| dev/a-4 | 256×2 | 19.7 ± 1.9 | 65.8 | 26,063 | 55.4 | 96.4 | +7.6 ± 2.2 | 54 / 24 |
| dev/a1 | 256×2 | 22.7 ± 2.1 | 77.1 | 26,964 | 96.6 | 93.9 | +10.5 ± 1.8 | 42 / 7 |
| dev/a12 | 256×2 | 51.8 ± 2.8 | 12.3 | 13,515 | 59.0 | 9.5 | +39.6 ± 2.7 | 102 / 0 |
| dev/a2 | 256×2 | 35.2 ± 2.6 | 62.7 | 23,911 | 92.1 | 94.7 | +23.0 ± 2.3 | 71 / 5 |
| dev/a4 | 256×2 | 59.2 ± 2.8 | 29.1 | 17,730 | 82.9 | 80.5 | +47.1 ± 2.6 | 118 / 0 |
| dev/a8 | 256×2 | 60.7 ± 2.8 | 14.1 | 14,589 | 70.7 | 26.4 | +48.6 ± 2.6 | 119 / 0 |
| dev/all1 | 256×2 | 28.7 ± 2.3 | 71.9 | 25,609 | 97.9 | 95.7 | +16.6 ± 2.0 | 61 / 6 |
| dev/all2 | 256×2 | 49.4 ± 2.8 | 46.7 | 20,117 | 91.9 | 87.0 | +37.3 ± 2.5 | 97 / 0 |
| dev/base | 256×2 | 12.1 ± 1.6 | 88.7 | 29,683 | 100.0 | 96.9 | - | - |
| dev/base+max4k | 256×2 | 5.9 ± 1.1 | 95.5 | 4,019 | 100.0 | 18.2 | -6.2 ± 1.2 | 3 / 30 |
| dev/base+nothink | 256×2 | 52.1 ± 2.7 | 18.0 | 2,270 | 63.6 | 7.6 | +40.0 ± 2.5 | 115 / 2 |
| dev/base+pp0 | 256×2 | 4.5 ± 0.9 | 99.0 | 32,472 | 100.0 | 98.4 | -7.6 ± 1.6 | 11 / 42 |
| dev/base+pp2 | 256×2 | 16.2 ± 1.9 | 84.6 | 28,614 | 98.7 | 96.8 | +4.1 ± 1.7 | 24 / 13 |
| dev/base+rep1.1 | 256×2 | 34.4 ± 2.6 | 61.7 | 23,301 | 89.8 | 88.3 | +22.3 ± 2.2 | 62 / 3 |
| dev/base+temp1.0 | 256×2 | 27.9 ± 2.5 | 71.3 | 27,430 | 95.9 | 59.5 | +15.8 ± 2.1 | 52 / 9 |
| dev/capA-a6 | 256×2 | 43.0 ± 2.6 | 52.5 | 22,699 | 89.3 | 81.4 | +30.9 ± 2.3 | 95 / 1 |
| dev/capA-alpha-8 | 256×2 | 42.8 ± 2.8 | 53.5 | 22,772 | 89.9 | 78.5 | +30.7 ± 2.5 | 87 / 4 |
| dev/capA-max-samples-16 | 256×2 | 17.6 ± 1.9 | 82.6 | 28,525 | 96.6 | 96.0 | +5.5 ± 1.9 | 34 / 17 |
| dev/capB-a6 | 256×2 | 58.2 ± 2.9 | 30.7 | 19,899 | 83.7 | 49.0 | +46.1 ± 2.6 | 113 / 0 |
| dev/capB-alpha-8 | 256×2 | 54.5 ± 2.9 | 34.0 | 20,870 | 81.7 | 37.9 | +42.4 ± 2.6 | 107 / 1 |
| dev/capB-max-samples-16 | 256×2 | 53.7 ± 2.8 | 39.6 | 20,725 | 88.7 | 76.8 | +41.6 ± 2.6 | 110 / 0 |
| dev/l15only | 256×2 | 60.4 ± 2.9 | 28.5 | 17,362 | 84.2 | 67.8 | +48.2 ± 2.7 | 117 / 0 |
| dev/l19only | 256×2 | 46.9 ± 2.7 | 49.0 | 20,729 | 91.2 | 90.0 | +34.8 ± 2.4 | 100 / 0 |
| dev/p50c | 256×2 | 61.5 ± 2.8 | 19.7 | 16,104 | 76.6 | 24.8 | +49.4 ± 2.6 | 125 / 0 |
| dev/p50c+nothink | 256×2 | 53.5 ± 2.8 | 16.4 | 2,306 | 64.0 | 0.0 | +1.4 ± 1.9 | 22 / 26 |
| dev/randcols | 256×2 | 14.6 ± 1.9 | 86.7 | 29,113 | 100.0 | 97.5 | +2.5 ± 1.6 | 23 / 20 |
| dev/randdir | 256×2 | 14.5 ± 1.8 | 87.3 | 29,486 | 98.5 | 97.3 | +2.3 ± 1.6 | 22 / 18 |
| dev/rebuilt | 256×2 | 61.5 ± 2.8 | 17.6 | 16,350 | 74.6 | 32.2 | +49.4 ± 2.6 | 122 / 0 |
| dev/rebuilt-layers15-19-21 | 256×2 | 61.3 ± 2.8 | 21.1 | 15,947 | 77.5 | 38.0 | +49.2 ± 2.6 | 122 / 0 |
| dev/rebuilt-pairs2 | 256×2 | 62.3 ± 2.7 | 25.2 | 16,080 | 83.0 | 74.4 | +50.2 ± 2.6 | 129 / 0 |
| dev/rebuilt-windows36-36 | 256×2 | 61.9 ± 2.8 | 21.3 | 16,141 | 78.7 | 35.8 | +49.8 ± 2.7 | 122 / 0 |
| quick/base | 16×2 | 18.8 ± 7.7 | 84.4 | 28,375 | 100.0 | 100.0 | - | - |
| quick/p50c | 16×2 | 65.6 ± 11.8 | 9.4 | 11,664 | 72.4 | 0.0 | +46.9 ± 11.6 | 6 / 0 |
"Cut answers ending in repetition" counts two kinds of ending among the answers cut at the cap. The first is a word-for-word loop (one block of words repeated back to back at least 3 times, over at least 40 words). The second is a looser circling (at least half of the 4-word phrases in the last 200 words occur 3 or more times there). See verbatim_loop_end and repetitive_end in wsp/grading.py.