LeBot — Overnight Report
Good morning. You asked for two things before bed — (1) retrain the buzz-confidence
model on Sonnet, and (2) mine mechanical Science Bowl patterns from the rules + corpus
using many subagents. Both are done (buzzer retrain results below). Along the way I also
shipped the NSBA-4 question picker and fixed the "none"-trap bug. Everything is committed and
deployed. Numbers, findings, and one correction to a belief you had are below.
1. Shipped & deployed tonight
NSBA-4 question picker
Scraped all 72 tossups (rounds 1–3) from isobowl.com's API and wired a dropdown into the demo —
pick a question, it auto-fills stem + category + correct answer. Filter by round.
72 tossups3 rounds
Fixed the "none"-trap + blind-list bug
Two bugs on numbered-list questions: (a) the bot blurted
1, 2, 3 before any list item
was even read (pure format-guessing), and (b) it couldn't answer "none" on necessity questions
("which MUST increase…"). Added a blind-list guard + a necessity nudge + skip-calc-on-lists.
list accuracy 77% → 95%
none-cases 0/2 → 2/2
Measured on the 22 NSBA-4 list questions via eval_lists.py.
Applied mined patterns to the answerer
Added Science-Bowl answer-form rules and a multi-select strategy to the brain's prompt (details §2/§3).
Plus max_retries=8 on the API client — overnight Anthropic 529 overloads kept crashing
the batch jobs.
2. Patterns mined (4 parallel subagents)
One subagent read the official NSB 2026 Rules PDF; three mined the
24,359-question corpus. Full catalog in PATTERNS.md. The high-value, actionable ones:
Multiple-choice empirical priors STAT
| Pattern | Finding | Use |
|---|
| Correct-letter skew | W .215 / X .272 / Y .274 / Z .239 | Not uniform — measure edge vs 27%, not 25% |
| Numeric options | ~88% pre-sorted ascending | Position carries no info; kill positional priors |
| Sorted-numeric answer | middle two ranks 64% | Avoid extremes, esp. the largest |
| Negated stem (NOT/EXCEPT) | skews Y/Z ≈ 59% | Shift prior late |
| "closest to" | X/Y ≈ 67% | Middle-value bias |
| Longest option | below chance | Do NOT use length heuristic |
List-question structure STAT
- Ordering: ~96% of answers are index permutations, 72% exactly 3 items.
- Multi-select: the modal answer is a PAIR (41%), not a single item.
"Pick the one true statement" is wrong ~74% of the time → the bot must evaluate each item independently. (Now in the prompt.)
- "none"/0 base rate ~6%, "all" ~8% — real but rare; never rule out, don't over-predict.
Opener → answer-format cues STAT
The first ~4 words predict the answer format with 97–100% reliability:
which of the following → MC letter · identify all → subset · order/rank → index permutation ·
how many/calculate → number · what is the term for → 1–3 word recall.
Also: researchers/scientists at… → 98% ENERGY category.
3. Answer-format rules from the NSB PDF RULE
These are ground truth (official scoring rules) — how to phrase an answer to not get
robbed of points. Now baked into the brain's prompt:
- MC → say the LETTER alone (W/X/Y/Z). Never give both letter + value (both must then be correct).
- Omit units unless demanded — stating a wrong unit is a failure mode.
- Surname only — "Einstein", not "Albert Einstein".
- Name the concept, don't describe it — "Newton's second law", not "F=ma" ("c" is fine).
- Formula only when no isomers — "CO₂" ok; "C₂H₆O" not ok for ethanol.
- Answer token only — any preface / spoken reasoning counts as WRONG; first utterance commits.
- Interrupt risk is asymmetric — wrong interrupt gives opponents +4; a wrong answer on a
fully-read tossup costs 0. So guessing on a full read is nearly free; interrupting needs real confidence.
4. Buzzer retrain on Sonnet
Regenerated the training labels (sa_labels_sonnet.json) with the current
Sonnet-primary pipeline (calc-router + list-guard + necessity-nudge + refusal-fallback) instead of the
stale labels the old buzzer was trained on, then retrained the LogisticRegression buzzer.
Note: the server's "haiku_vote" is a misnomer — anticipate_sa
already runs on Sonnet (the default model). The real win of the retrain is that the labels now reflect
the current answerer behavior (calc-router, list-guard, necessity-nudge, refusal-fallback), not a
months-old pipeline. Regenerated 296 questions × per-word prefixes on the VPS overnight.
| Metric | Old labels | New (Sonnet pipeline) |
|---|
| ECE (calibration error, lower=better) | 0.080 | 0.044 |
| ev/q @ T=0.7 | +0.09 | +1.44 |
| ev/q @ T=0.8 | +0.80 | +1.66 |
| ev/q @ T=0.9 | +1.33 | +2.16 |
| accuracy when buzzing @ T=0.8 | 62% | 73% |
is_calc feature weight | −0.08 | +0.45 |
Calibration error nearly halved; EV at the operating threshold went from break-even to
+1.44 points/question. The is_calc weight flipped positive — the buzzer now correctly
trusts computed (calculator) answers instead of distrusting them. New labels are live in production.
Backtest: no regression, buzzer improved
List-question accuracy held at 21/22 (95%) after all the prompt changes (measured
production-representative: Sonnet-verbose primary, the path the demo actually shows). None-cases 2/2.
The one miss is a genuine C-language knowledge error, not a format bug.
5. One correction for you RULE
You believed you can answer an ordering question by naming only the distinguishing item
("1, the one not mentioned, Chlorine"). The rules PDF says this is only true for SELECTION /
"identify all" questions (Appendix A-2f: distinguish an item by number, position, "all", or
"none" — any sufficient way). For ordering questions (A-2e) you must still give the
full order — though each element can be a number or position rather than the exact name.
There is no rule authorizing a partial or item-omitting answer to "order these". So: the trick is
real, just scoped to select-all, not ordering.
6. Proposed next (documented, not built)
- BUILD Canonical-sequence ordering solver — ~12–13% of
ordering questions sit on a fixed monotonic sequence (periodic trends, EM spectrum, Mohs, taxonomy,
planetary order, geologic time). A small
sequences.py + axis-detector would nail these
deterministically with zero LLM calls. Another ~20% are table-lookup axes (pKa, boiling point).
- BUILD MC letter-prior de-biasing — feed the W/X/Y/Z skew
and negation/"closest-to" adjustments into confidence calibration.
- BUILD Reasoning-leak hardening —
_clean_answer
occasionally lets short reasoning prefixes through ("For n = 3:"). Tighten it.
- The pattern catalog (
PATTERNS.md) has the full list with counts.