Lab 5: can decoding settings do the same job?

Big thing: We compare the 131,072-weight edit against six decoding changes on unedited Qwen3.5-0.8B (temperature 1.0, presence penalty 0 and 2.0, repetition penalty 1.1, a shorter 4,096-token cap, and <think> mode off) across all 512 dev256 answers. Penalties and temperature raise accuracy at most from 12.1% to 34.4%, turning thinking off reaches 52.1% in 13 times fewer tokens than the unedited thinking baseline, and the weight edit with thinking on reaches 61.5% (74.6% on the GSM8K half).

Why it matters: Always test whether a weight edit beats simpler decoding changes that leave the weights alone. Ending counts suggest penalties mostly add normal closes, while the weight edit adds both normal closes and loop escapes. These counts do not prove which loops were prevented, because the check reads only the text right before </think>.

Time and hardware: We already ran it (results/summary.md, results/lab5/endings.json, and results/figures/lab5_decoding.png). Re-running one unedited dev256 job on 1 NVIDIA A100 GPU takes 34 to 49 minutes with thinking on at the 32,768-token cap, or about 3 minutes with thinking off or a 4,096-token cap.

Theory guide: Section 13.2.

Before you run: predict#

Write down a guess for each question before looking at the table.

1. The paper's baseline ("think35") uses presence_penalty = 1.5, which subtracts 1.5 from the score of any token that has already appeared in the answer. If you turn that penalty off (presence_penalty = 0), what happens to the share of answers cut at the 32,768-token cap (now 88.7%)?
2. The unedited model's correct dev256 answers have a median length of 5,600 tokens, while looping answers run to 32,768 tokens. What happens to accuracy if you lower the token cap to 4,096 tokens?
3. What if you turn <think> mode off (enable_thinking=False), so the model writes a direct step-by-step solution instead of an open-ended internal scratchpad? Will it beat the unedited thinking model (12.1%), and will the weight edit (p50c) still help when thinking is off?

Run it#

All seven decoding runs are at the bottom of plans/sampler.json and ran on sampler alongside the Lab 4 variants (dev/base and dev/p50c at the top of plans/sampler.json are the baselines).

Open plans/sampler.json and check the last seven entries:

  {"name": "dev/base+temp1.0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["temperature=1.0"]},
  {"name": "dev/base+pp0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=0"]},
  {"name": "dev/base+pp2", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=2.0"]},
  {"name": "dev/base+rep1.1", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 14, "set": ["repetition_penalty=1.1"]},
  {"name": "dev/base+max4k", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "set": ["max_tokens=4096"]},
  {"name": "dev/base+nothink", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"},
  {"name": "dev/p50c+nothink", "model": "qwen35-p50c", "data": "dev256", "n": 2, "shards": 1, "est": 1, "preset": "nothink35"}

Each entry starts from a preset in wsp/protocol.py:

• "think35" (the default when "preset" is omitted): temperature = 0.6, top_p = 0.95, top_k = 20, presence_penalty = 1.5, repetition_penalty = 1.0, max_tokens = 32768, thinking = True.
• "nothink35" (Qwen's model-card settings for non-thinking mode): temperature = 0.7, top_p = 0.8, top_k = 20, presence_penalty = 1.5, repetition_penalty = 1.0, max_tokens = 8192, thinking = False.
• A "set" list overrides fields in the preset (scripts/run_jobs.py passes each "key=value" string to scripts/gen_worker.py as --set).

These runs finished on sampler and are in results/runs/ on this laptop. On 1 A100 GPU, a full dev256 run takes 17 to 49 minutes (33.8 to 48.9 minutes for the long unedited runs; dev/p50c took 20.6 minutes), while max_tokens=4096 and nothink35 take about 3 minutes (2.7 to 3.2 minutes).

To inspect how answers ended or read single answers on sampler (every VM command starts with . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code):

ssh sampler
. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
export CUDA_VISIBLE_DEVICES=""

# Classify how every answer ended across the Lab 5 runs (writes results/lab5/endings.json)
python labs/lab4_endings.py \
  dev/base dev/base+temp1.0 dev/base+pp0 dev/base+pp2 dev/base+rep1.1 dev/base+max4k \
  dev/base+nothink dev/p50c+nothink \
  --out lab5/endings

# List or read individual answers from any decoding run on sampler
python labs/peek.py dev/base+rep1.1 --id gsm8k_train_5628
python labs/peek.py dev/base+nothink --id gsm8k_train_5628

labs/lab4_endings.py skips dev/base+nothink and dev/p50c+nothink with a message (thinking is off, so its answers have no </think> to look for).

To redraw results/figures/lab5_decoding.png on this laptop:

# On the local laptop, inside ~/weight-space-programming/code
python3 scripts/summarize.py results/runs
uv run -q --with numpy --with matplotlib python scripts/plot_runs.py

What you should see#

1. All nine runs on dev256 (512 answers per run)#

Every run uses the same 256 unseen practice problems (128 GSM8K-train + 128 MATH-train, 2 answers each = 512 answers). Paired gain compares each run against dev/base, except dev/p50c+nothink, which scripts/summarize.py pairs against dev/base+nothink to isolate the weight edit when thinking is off:

2. Breakdown by problem source (GSM8K vs MATH) and median length of correct answers#

Splitting dev256 into its 128 GSM8K problems and 128 harder MATH problems (results/summary.json) shows where the tokens go and where thinking mode pays off:

In dev/base+pp0, removing the presence penalty sends 99.0% of answers (98.4% on GSM8K, 99.6% on MATH) to the 32,768-token cap. The median length of its 23 correct answers is 32,768 tokens because 18 of those 23 were cut while thinking after boxing an answer mid-scratchpad. In dev/base+max4k, because the unedited model's median correct answer is 5,600 tokens, a 4,096-token cap cuts off valid reasoning before completion and drops accuracy from 12.1% to 5.9% (30 problems lost, 3 rescued).

3. How every thinking-mode answer ended (results/lab5/endings.json)#

Ending counts from results/lab5/endings.json (plus dev/p50c from results/lab4/endings.json), with correct answers in parentheses:

Compare the last two columns (closed out of a loop and closed normally):

• Raising temperature to 1.0 increases closed normally from 56 to 157, while closed out of a loop stays at 3.
• Raising presence penalty to 2.0 gives 4 loop escapes; adding repetition penalty 1.1 gives 28 loop escapes and 190 normal closes.
• The weight edit (dev/p50c) produces 344 normal closes and 96 loop escapes. These endings suggest decoding tweaks mostly add normal closes, while the weight edit adds both. A normal close can still follow an earlier loop, because the check reads only the last 6,000 characters before </think>.

Why it works#

1. Decoding from first principles: how vLLM turns logits into a chosen token#

At each generation step, the language model outputs a vector of raw scores called logits, one number $z_v$ for every token $v$ in the vocabulary. Without decoding tweaks, the softmax function turns those logits into probabilities by exponentiating each score and dividing by the sum:

Because $e^{z_v}$ grows exponentially, adding $+3$ to a token's logit multiplies its relative odds by $e^3 \approx 20$.

In vLLM 0.33.0, sampling follows the four steps below. The helper pick_prob in wsp/protocol.py copies this order for presence penalty, temperature, top-k, and top-p, and rejects non-default repetition penalties.

Each step transforms a token's score:

1. Penalties (repetition_penalty and presence_penalty):
   • Repetition penalty $r$ (1.1 in dev/base+rep1.1): For every token $v$ that has appeared anywhere in the prompt or generated answer so far, divide $z_v$ by $r$ ($z_v \leftarrow z_v / r$) if $z_v$ is positive, or multiply by $r$ ($z_v \leftarrow z_v \times r$) if negative. Both push the logit down in proportion to its size.
   • Presence penalty $p$ (1.5 in "think35"): For every token $v$ that has appeared in the generated answer so far (ignoring the prompt), subtract $p$ ($z_v \leftarrow z_v - p$). This penalty is binary: a token used once and a token used 200 times both lose 1.5.
2. Temperature $T$ (0.6 in "think35", 1.0 in dev/base+temp1.0): Divide every logit by $T$ ($z_v \leftarrow z_v / T$). Dividing by $0.6$ multiplies logit gaps by $1 / 0.6 = 1.67$, favoring the top token. Setting $T = 1.0$ leaves gaps unstretched, giving runner-up tokens a larger chance.
3. Top-k (20) and Top-p (0.95): Keep the 20 highest-logit tokens, compute their softmax probabilities, sort them, and drop the lowest-probability tokens whose cumulative probability is at most $1 - 0.95 = 0.05$ (always keeping the top token).
4. Renormalize and sample: Divide the kept probabilities by their sum so they add to 1.0, then pick one token at random.

2. A worked example: why token penalties fail to break a strong loop#

Suppose the model is inside a loop and about to repeat "Let's" (using made-up numbers):

• "Let's" has raw logit $z_{\text{loop}} = 12.0$, and "</think>" (not yet in the answer) has raw logit $z_{\text{close}} = 5.0$.
• "Let's" and the other scratchpad words have already appeared in the answer.

This arithmetic explains three patterns in the data:

1. Why presence penalty does not grow as the loop repeats: "Let's" appeared in the first paragraph, so it pays -1.5 on cycle 1 and the same -1.5 on cycle 200. Normal math words ("Therefore", "answer", "total", "240") also appeared earlier in the scratchpad, so each pays -1.5 too, leaving the gap between "Let's" and "Therefore" unchanged.
2. Why repetition_penalty = 1.1 does better (34.4%) than presence_penalty = 2.0 (16.2%): Dividing a positive logit by 1.1 takes more from a large logit ($12.0 \to 10.91$, losing $1.09$ on top of the $1.5$ presence penalty) than from a moderate runner-up ($8.0 \to 7.27$, losing $0.73$). Even so, 61.7% of dev/base+rep1.1 answers hit the cap.
3. How penalties might prevent some loops yet fail to break others: When looping and productive continuations have similar logits, higher temperature (1.0) or repetition_penalty = 1.1 can tip the choice away from the first repeat. Once repeated cycles fill the context, loop-token logits can grow high enough that top_p = 0.95 drops every non-loop token to probability zero. Lab 6 Part 3 measures the per-token pick probability inside 48 real loops: 0.99995 in the unedited model vs 0.979 with the weight edit.

3. Why turning <think> off scores 52.1%, and why the edit adds nothing there#

Switching from "think35" to "nothink35" (dev/base+nothink) raises dev256 accuracy from 12.1% to 52.1% and cuts mean length from 29,683 tokens to 2,270 (median length on correct answers is 452 tokens).

1. The weight edit helps only when loops occur. With thinking off (dev/base+nothink), 18.0% of answers hit the 8,192-token cap, and 7.6% of those cut answers end in repetition (5.4% in a verbatim loop). Few answers loop, so with the weight edit and thinking off (dev/p50c+nothink) the paired difference against dev/base+nothink is +1.4 +/- 1.9 percentage points (22 problems rescued, 26 lost), which is indistinguishable from zero.
2. Once loops are repaired, the thinking preset beats the non-thinking preset, mostly on word problems. Holding the edited weights fixed, dev/p50c (think35, 61.5%) beats dev/p50c+nothink (nothink35, 53.5%) by +8.0 +/- 2.0 paired points (27 problems rescued, 11 lost). Most of that gain comes from GSM8K (74.6% vs 62.1%, +12.5 +/- 2.8 points), while the MATH gap is small (48.4% vs 44.9%, +3.5 +/- 2.9 points). The thinking preset uses about seven times as many tokens (16,104 vs 2,306 on average). The two presets also differ in sampling settings (think35 uses temperature 0.6, top-p 0.95, and a 32,768-token cap; nothink35 uses temperature 0.7, top-p 0.8, and an 8,192-token cap), so part of the gap could come from sampling changes instead of thinking itself.

4. The paper's Gemma 3 270M follow-up points the same way#

In Theory Guide section 10 (and external/lm_understand-logs/exp8_math/gemma_followup/README.md), the authors ran the same weight-space programming setup on a different small model, Gemma 3 270M. On MATH-500, unedited Gemma 3 270M scored 6.8% and hit the token cap on 8.5% of answers (91.5% finished, yet only 6.8% of all answers were right).

Across multiple agent runs, no weight edit moved Gemma 3 270M's MATH-500 score outside 6.8% to 7.7%. As the blog puts it: "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked." That matches dev/p50c+nothink: when answers do not get trapped in loops, subtracting the loop direction u does not make the model better at math.

Check yourself#

1. Why did removing the presence penalty (dev/base+pp0) hurt accuracy (12.1% down to 4.5%), whereas raising it from 1.5 to 2.0 (dev/base+pp2) barely helped (12.1% to 16.2%)?

2. In results/summary.md, dev/base+nothink shows a +40.0 +/- 2.5 percentage-point gain, while dev/p50c+nothink shows +1.4 +/- 1.9 percentage points even though its accuracy (53.5%) exceeds dev/base+nothink (52.1%). Why do the two gains differ?

3. If a teammate suggests saving compute on the unedited thinking model by lowering max_tokens from 32,768 to 4,096 (dev/base+max4k), what do the numbers show?

4. In vLLM's sampler (wsp/protocol.py), does presence_penalty get applied before or after dividing by temperature? Why does that order matter when temperature = 0.6?

Try this next#

1. Compare how gsm8k_train_5628 ends across four decoding settings and the edited dev/p50c on sampler. Run labs/peek.py (CPU only, a few seconds) on the Mr. Desmond problem (truth: 240) across dev/base, dev/base+temp1.0, dev/base+rep1.1, dev/base+nothink, and dev/p50c:
   bash

   ssh sampler
   . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
   export CUDA_VISIBLE_DEVICES=""
   for r in dev/base dev/base+temp1.0 dev/base+rep1.1 dev/base+nothink dev/p50c; do
     echo "=== $r ==="
     python labs/peek.py "$r" --id gsm8k_train_5628 --chars 600
   done
   

   Compare the length of dev/base+nothink against dev/p50c, and check whether dev/base+rep1.1 solved gsm8k_train_5628 or looped.
2. Test a new decoding combination on 16 problems (not run by us; write down your prediction first). Combining the two best decoding tweaks on the unedited model (temperature = 1.0 and repetition_penalty = 1.1), or combining the weight edit (qwen35-p50c) with repetition_penalty = 1.1, was not in plans/sampler.json. You can test both in about 5 minutes on 2 GPUs of sampler using "args": ["--limit", "16"] (as in plans/quick.json) and a "try/" prefix so runs/dev/ stays untouched:
   json

   {
    "about": "Quick 16-problem test of combined decoding settings.",
    "runs": [
     {
      "name": "try/base+temp1.0+rep1.1",
      "model": "base",
      "data": "dev256",
      "n": 2,
      "shards": 1,
      "est": 1,
      "set": ["temperature=1.0", "repetition_penalty=1.1"],
      "args": ["--limit", "16"]
     },
     {
      "name": "try/p50c+rep1.1",
      "model": "qwen35-p50c",
      "data": "dev256",
      "n": 2,
      "shards": 1,
      "est": 1,
      "set": ["repetition_penalty=1.1"],
      "args": ["--limit", "16"]
     }
    ]
   }
   

   Save this JSON on this machine as plans/try_lab5.json. From the project root on this machine, sync the plan, run it on sampler, and pull the results:
   bash

   bash scripts/sync_vm.sh sampler
   ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5.json --gpus 0,1' </dev/null
   bash scripts/pull_runs.sh sampler
   

   The single shard runs the first 16 rows of dev256 (all GSM8K problems, 2 answers each). The try/ prefix gets no paired column from scripts/summarize.py, so compare the shared problem IDs directly on this machine:
   bash

   python3 - <<'PY'
   from wsp.summary import compare, load_run
   for name in ("base+temp1.0+rep1.1", "p50c+rep1.1"):
       trial = load_run("results/runs/try/" + name)
       for baseline in ("base", "p50c"):
           print(name, "vs dev/" + baseline,
                 compare(load_run("results/runs/dev/" + baseline), trial))
   PY
   

   What we predict: Combining temperature 1.0 and repetition penalty 1.1 may reduce repetition further, though a gain is not guaranteed. On these 16 problems, our separate runs score 40.6% (13/32 answers) at temperature 1.0, 46.9% (15/32) with repetition penalty 1.1, and 65.6% (21/32) with p50c, with standard errors across problems of about 11 to 12 percentage points, so predictions from this small check are rough. Adding repetition penalty to p50c might reduce cuts, but it also penalizes repeated numbers and variables in valid solutions. Because this slice has no MATH problems, it cannot test hard MATH answers.
3. Test nothink35 on the three full benchmarks (not run by us; write down your prediction first). We ran nothink35 on dev256 (52.1% unedited, 53.5% edited), not on math500, gsm8k, or ifeval. Scaling from dev/base+nothink (512 answers in 2.73 minutes on 1 GPU) by answer count gives a rough wall-clock estimate of 6 to 15 minutes per benchmark with one shard on 1 GPU (1,082 IFEval, 2,000 MATH-500, or 2,638 GSM8K answers); answer lengths and throughput may differ. Save this plan on this machine as plans/try_lab5_nothink.json (matching the benchmark files and answer counts of the bench/ runs in plans/trainer.json):
   json

   {
    "about": "Thinking off (nothink35) on the three full benchmarks, unedited and edited.",
    "runs": [
     {"name": "try/nothink/base/math500", "model": "base", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"},
     {"name": "try/nothink/base/gsm8k", "model": "base", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"},
     {"name": "try/nothink/base/ifeval", "model": "base", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"},
     {"name": "try/nothink/p50c/math500", "model": "qwen35-p50c", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"},
     {"name": "try/nothink/p50c/gsm8k", "model": "qwen35-p50c", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"},
     {"name": "try/nothink/p50c/ifeval", "model": "qwen35-p50c", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"}
    ]
   }
   

   Sync, run on six free GPUs (replace 0-5 with the free GPU indices), and pull the results:
   bash

   bash scripts/sync_vm.sh sampler
   ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5_nothink.json --gpus 0-5' </dev/null
   bash scripts/pull_runs.sh sampler
   

   Print each score and the paired gain of the edit on this machine:
   bash

   python3 - <<'PY'
   import json
   from wsp.summary import compare, load_run
   for d in ("gsm8k", "math500", "ifeval"):
       for m in ("base", "p50c"):
           s = json.load(open(f"results/runs/try/nothink/{m}/{d}/summary.json"))
           print(f"{m} {d}: {100 * s['acc']:.1f} +/- {100 * s['acc_se']:.1f}%")
       c = compare(load_run(f"results/runs/try/nothink/base/{d}"), load_run(f"results/runs/try/nothink/p50c/{d}"))
       print(f"  p50c minus base, paired: {100 * c['gain']:+.1f} +/- {100 * c['gain_se']:.1f} points")
   PY
   

   What we predict: On GSM8K test and MATH-500, base with preset: "nothink35" should mirror the dev256 split: about 55% to 60% on GSM8K (above 14.2% unedited thinking, below 66.9% edited thinking) and 42% to 46% on MATH-500 (near edited thinking's 51.8%). On IFEval, which tests formatting constraints instead of long math derivations, base with nothink35 should approach bench/p50c/ifeval (63.4%), and p50c with nothink35 should match base with nothink35 within noise.