36_buzzpoint_rank_difficulty.md

36 — Buzz RANK vs DIFFICULTY-WEIGHTED vs raw celerity vs outcome-only

36 — Buzz RANK vs DIFFICULTY-WEIGHTED vs raw celerity vs outcome-only

Follow-up to finding 35. Finding 35 showed raw celerity barely improves an opponent-adjusted outcome-only Plackett-Luce player ranking, but flagged the scibowl/SSB data as a weak position test (lockout short-answer, position only on ~29% of buzzes). Two strong players proposed a WHY + a fix:

• RADIUS: raw celerity is difficulty-blind — "you'd need to factor in the actual INFORMATION people buzz on." Fix = weight each buzz by the difficulty of the clue converted (IRT-style item difficulty).
• THEDOGE: use buzz RANK per question (were you the 5th-fastest or 30th to get it) instead of continuous celerity — rank auto-normalizes for question difficulty/length.

This finding tests whether a smarter buzzpoint feature beats both raw celerity and outcome-only ranking, on held-out prediction — the test that matters.

Script: scripts/analysis/36_buzzpoint_rank_difficulty.py (ENV /home/david/code/nsba/.venv/bin/python). Artifacts: data/processed/buzzpoint_rank_difficulty_36_{qb,ssb,corr,eval}.csv.

Datasets — and why we finally have a strong test

The big upgrade over finding 35: I pulled true pyramidal quizbowl buzz curves from the open-license quizbowl/open-data repo (ODbL).

• PRIMARY — ACF Regionals 2018 (detailed-stats/.../tossups.tsv): 16,854 buzz rows, buzz_location_pct present on 91% of buzzes (vs 29% for SSB), 1,446 distinct questions each read across ~9 rooms, 526 players, and — crucially — (tournament, round, room) reconstructs real 2-team games (736 of them), so opponent adjustment is genuine. Eligible field per tossup = players who buzzed in that game (rosters aren't in the file; this is a buzzer-only proxy). 14,213 per-tossup races, 8% dead. This is where the rank/difficulty ideas should win if they ever do.
• SECONDARY — SSB-2026 (scibowl): the finding-35 set, kept for continuity. 2,013 races, 46% dead, weak position signal. Honest caveat carried over from 35: SSB is a poor position test, so it can only corroborate, not drive, the verdict.

Why this matters for the dispute: Radius and thedoge are both quizbowl players arguing about pyramidal buzzpoints. ACF Regionals is exactly that format, so we are now testing their claims on their home turf, not on a lockout-format proxy.

Models (same player set / same game-split CV per dataset)

1. OUTCOME-PL — opponent-adjusted Plackett-Luce (top-1 conditional logit) on who-buzzed-first, with a shared dead-tossup difficulty "virtual competitor." This is finding 35's baseline / Radius's actual model. No buzzpoints.
2. +CELERITY — same PL, each won race weighted by winner celerity (0.5 + (1 − location_pct)). The thing that failed in 35; reconfirmed here.
3. +BUZZRANK (thedoge) — for each question, pool every read, rank the correct buzzers by speed, score each player by mean normalized rank (1 = always the fastest to convert that clue). Also +BUZZRANK-PL: a full-order Plackett-Luce fit on the per-question speed ORDER (fastest→slowest), i.e. the rank-aggregated speed-skill thedoge gestured at. (Gradient analytically verified vs numerical, recovers a known speed ordering at ρ=1.0 on synthetic data.)
4. +DIFFICULTY (Radius) — buzzpoint-IRT-flavored. Reference field = top third by conversions. For each question, build the celerity distribution of strong getters; a buzz at celerity c is credited with the fraction of the strong field that did NOT beat it to the clue (buzzed later or missed) — the difficulty it overcame. Player rating = difficulty-weighted credit / exposure. Hard, early conversions on clues that mostly went late score highest.

Eval (held-out games): split games 75/25, refit every rating on train, and on held-out non-dead races predict the winner among the eligible field via softmax(z-scored rating). Report winner-argmax accuracy and mean log-loss, averaged over 8 random splits. Plus full-data Spearman of each model vs OUTCOME-PL.

Results

Spearman vs OUTCOME-PL baseline

Held-out prediction (mean ± sd over 8 game splits)

(QB held-out acc is low in absolute terms because fields are larger and a huge share of held-out winners are players unseen/thin in train — the comparison across models on the same races is what's informative, not the absolute level.)

Interpretation

1. Raw celerity ≈ outcome-only. Reconfirmed, now on real pyramidal data. +celerity is nearly identical to OUTCOME-PL in ranking (ρ 0.992 QB / 0.972 SSB) and statistically tied held-out (QB 0.326 vs 0.327, SSB 0.529 vs 0.524 — within one SD either way). On the format where celerity is supposed to matter most, it still adds essentially nothing to the ranking. Finding 35's celerity result replicates.

2. thedoge's buzz-RANK is NOTICEABLY WORSE, not better. It diverges sharply from outcome (ρ 0.42–0.52) — but in the wrong direction: held-out it is the worst model on both datasets (QB acc 0.18 vs 0.33; SSB 0.46–0.50 vs 0.52), with the highest log-loss. The mechanism is visible in the rankings: rank-normalization conditions on questions you already got, discarding the dominant skill signal — how often you convert. It promotes low-volume fast buzzers (a player with 13 gets jumps from outcome-rank 409 → buzzrank 72) and punishes elite high-volume players (Itamar Naveh-Benjamin, 112 gets, outcome-rank 9 → buzzrank 319). "Were you fast on the ones you got" is a style axis orthogonal to quality; auto-normalizing for difficulty also auto-normalizes away the thing you wanted to rank. thedoge's bet (rank > celerity) loses: rank is worse than celerity AND worse than outcome.

3. Radius's difficulty-weighting tracks outcome but does NOT beat it. It is the sane buzzpoint feature — it stays correlated with skill (ρ 0.83 QB / 0.93 SSB, 13–16 of top-20 retained) because it rewards converting hard clues, which strong players do more. But held-out it is no better than OUTCOME-PL, and slightly worse (QB 0.321 vs 0.327, log-loss 1.833 vs 1.768; SSB 0.493 vs 0.524). Folding in clue difficulty did not unlock a predictive gain. Most of what difficulty-weighting "knows" is already implied by who converts against whom — the opponent-adjusted outcome model has it baked in.

4. The ceiling stays low even with the smart feature. The whole premise was that a difficulty-aware buzzpoint feature might finally separate from outcome-only. On true pyramidal data it doesn't: every buzzpoint variant either ties the outcome baseline (celerity, difficulty) or underperforms it (both rank variants). Nothing here noticeably beats opponent-adjusted who-buzzed-first out of sample.

Verdict (head-to-head, on held-out prediction)

• rank vs celerity (thedoge's bet): thedoge LOSES. Buzz-rank is worse than raw celerity on every metric and dataset — it strips out conversion volume and ranks style, not skill.
• difficulty-weighted vs celerity (Radius's fix): roughly a wash, leaning worse. Difficulty-weighting is a better-behaved feature than rank (it stays skill-correlated) but it buys no held-out improvement over plain celerity or outcome-only; in QB it's slightly behind.
• anything vs outcome-only: outcome-only wins or ties everything. On the pyramidal format where these ideas were supposed to shine, opponent-adjusted who-buzzed-first remains the best or statistically-tied ranking, with zero buzzpoints. Difficulty-awareness did not unlock a gain — the ceiling is still low.

Honest power limits. (a) QB eligible field is a buzzer-only proxy for the roster (no schedule/roster file), which inflates field size and depresses absolute held-out accuracy — but it biases all five models equally, so the ranking of models is robust. (b) One pyramidal set (ACF Regionals 2018) means question difficulty is identifiable per-question but only one tournament's worth; (c) SSB remains a weak position test and is used only to corroborate. Within those limits the direction is consistent across two very different formats: smarter buzzpoint features do not beat opponent-adjusted outcome — and the one that tried hardest to be "difficulty-aware" (Radius) tied at best, while the one that bet on rank (thedoge) clearly lost.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub