Good morning. You asked for two things before bed — (1) retrain the buzz-confidence
model on Sonnet, and (2) mine mechanical Science Bowl patterns from the rules + corpus
using many subagents. Both are done (buzzer retrain results below). Along the way I also
shipped the NSBA-4 question picker and fixed the "none"-trap bug. Everything is committed and
deployed. Numbers, findings, and one correction to a belief you had are below.
Scraped all 72 tossups (rounds 1–3) from isobowl.com's API and wired a dropdown into the demo —
pick a question, it auto-fills stem + category + correct answer. Filter by round.
72 tossups3 rounds
Fixed the "none"-trap + blind-list bug
Two bugs on numbered-list questions: (a) the bot blurted 1, 2, 3 before any list item
was even read (pure format-guessing), and (b) it couldn't answer "none" on necessity questions
("which MUST increase…"). Added a blind-list guard + a necessity nudge + skip-calc-on-lists.
list accuracy 77% → 95%none-cases 0/2 → 2/2
Measured on the 22 NSBA-4 list questions via eval_lists.py.
Applied mined patterns to the answerer
Added Science-Bowl answer-form rules and a multi-select strategy to the brain's prompt (details §2/§3).
Plus max_retries=8 on the API client — overnight Anthropic 529 overloads kept crashing
the batch jobs.
2. Patterns mined (4 parallel subagents)
One subagent read the official NSB 2026 Rules PDF; three mined the
24,359-question corpus. Full catalog in PATTERNS.md. The high-value, actionable ones:
Multiple-choice empirical priors STAT
Pattern
Finding
Use
Correct-letter skew
W .215 / X .272 / Y .274 / Z .239
Not uniform — measure edge vs 27%, not 25%
Numeric options
~88% pre-sorted ascending
Position carries no info; kill positional priors
Sorted-numeric answer
middle two ranks 64%
Avoid extremes, esp. the largest
Negated stem (NOT/EXCEPT)
skews Y/Z ≈ 59%
Shift prior late
"closest to"
X/Y ≈ 67%
Middle-value bias
Longest option
below chance
Do NOT use length heuristic
List-question structure STAT
Ordering: ~96% of answers are index permutations, 72% exactly 3 items.
Multi-select: the modal answer is a PAIR (41%), not a single item.
"Pick the one true statement" is wrong ~74% of the time → the bot must evaluate each item independently. (Now in the prompt.)
"none"/0 base rate ~6%, "all" ~8% — real but rare; never rule out, don't over-predict.
Opener → answer-format cues STAT
The first ~4 words predict the answer format with 97–100% reliability:
which of the following → MC letter · identify all → subset · order/rank → index permutation ·
how many/calculate → number · what is the term for → 1–3 word recall.
Also: researchers/scientists at… → 98% ENERGY category.
3. Answer-format rules from the NSB PDF RULE
These are ground truth (official scoring rules) — how to phrase an answer to not get
robbed of points. Now baked into the brain's prompt:
MC → say the LETTER alone (W/X/Y/Z). Never give both letter + value (both must then be correct).
Omit units unless demanded — stating a wrong unit is a failure mode.
Surname only — "Einstein", not "Albert Einstein".
Name the concept, don't describe it — "Newton's second law", not "F=ma" ("c" is fine).
Formula only when no isomers — "CO₂" ok; "C₂H₆O" not ok for ethanol.
Answer token only — any preface / spoken reasoning counts as WRONG; first utterance commits.
Interrupt risk is asymmetric — wrong interrupt gives opponents +4; a wrong answer on a
fully-read tossup costs 0. So guessing on a full read is nearly free; interrupting needs real confidence.
4. Buzzer retrain on Sonnet
Regenerated the training labels (sa_labels_sonnet.json) with the current
Sonnet-primary pipeline (calc-router + list-guard + necessity-nudge + refusal-fallback) instead of the
stale labels the old buzzer was trained on, then retrained the LogisticRegression buzzer.
Note: the server's "haiku_vote" is a misnomer — anticipate_sa
already runs on Sonnet (the default model). The real win of the retrain is that the labels now reflect
the current answerer behavior (calc-router, list-guard, necessity-nudge, refusal-fallback), not a
months-old pipeline. Regenerated 296 questions × per-word prefixes on the VPS overnight.
Metric
Old labels
New (Sonnet pipeline)
ECE (calibration error, lower=better)
0.080
0.044
ev/q @ T=0.7
+0.09
+1.44
ev/q @ T=0.8
+0.80
+1.66
ev/q @ T=0.9
+1.33
+2.16
accuracy when buzzing @ T=0.8
62%
73%
is_calc feature weight
−0.08
+0.45
Calibration error nearly halved; EV at the operating threshold went from break-even to
+1.44 points/question. The is_calc weight flipped positive — the buzzer now correctly
trusts computed (calculator) answers instead of distrusting them. New labels are live in production.
Backtest: no regression, buzzer improved
List-question accuracy held at 21/22 (95%) after all the prompt changes (measured
production-representative: Sonnet-verbose primary, the path the demo actually shows). None-cases 2/2.
The one miss is a genuine C-language knowledge error, not a format bug.
5. One correction for you RULE
You believed you can answer an ordering question by naming only the distinguishing item
("1, the one not mentioned, Chlorine"). The rules PDF says this is only true for SELECTION /
"identify all" questions (Appendix A-2f: distinguish an item by number, position, "all", or
"none" — any sufficient way). For ordering questions (A-2e) you must still give the
full order — though each element can be a number or position rather than the exact name.
There is no rule authorizing a partial or item-omitting answer to "order these". So: the trick is
real, just scoped to select-all, not ordering.
6. Three more builds — done
The deferred BUILD items are now implemented, tested, and deployed.
Canonical-sequence ordering solver sequences.py
For "order/rank by <axis>" questions on a fixed monotonic sequence, computes the index
permutation with zero LLM calls. Reference tables: electronegativity, covalent radius, first
ionization energy, Mohs, metamorphic grade, taxonomy, planets, EM spectrum, visible colors.
Conservative guards (skips ions / 2nd–3rd ionization energy, waits for all declared items, bails on
ties). corpus validation 26/26 (100%) correct on the ~2.5% of
ordering questions it fires on. Wired as mode='seq'; the buzzer trusts it like the calculator.
MC letter-prior de-biasing mc_prior.py
Blends the model's W/X/Y/Z votes with the empirical, stem-adjusted prior (negation→Y/Z, "closest to"→X/Y,
numeric-sorted middle bias). Validated on 8,241 corpus MC: prior-argmax 0.274 vs 0.25 uniform;
negation Y/Z 0.584; "closest to" X/Y 0.674; largest-value-correct only 0.223.
Opt-in (bot_buzz(debias=True)) so the reproducible experiment harness default is unchanged.
Reasoning-leak hardening _clean_answer
The terse path sometimes leaks a reasoning prefix ("For n = 3:", "The answer is…") that the buzzer
shouldn't treat as a commit. Now strips "answer is"/"Answer:" lead-ins and rejects trailing colons,
sentence-opener prefixes, and dangling connective/preposition endings — while keeping real terms
("vitamin A", "conservation of energy"). Covered by test_clean.py.
Full pattern catalog with counts: PATTERNS.md.
Still open (documented): table-lookup ordering axes (pKa, boiling point) need small reference tables;
MC de-biasing is built but only exercised in letter-mode experiments, not the value-prediction demo.