35_plackett_luce_buzzpoints.md

35 — Plackett-Luce & the buzzpoint claim

35 — Plackett-Luce & the buzzpoint claim

Claim under test (Radius): "There's nothing noticeably better than the rankings I came up with for quizbowl that you can produce WITHOUT incorporating buzzpoints — without them it's just fitting the data." He pointed to his Plackett-Luce model: dn285/buzzpoints-playground/plackett-luce.

Script: scripts/analysis/35_plackett_luce_buzzpoints.py (ENV /home/david/code/nsba/.venv/bin/python). Data: raw per-buzz export at scibowl-org/.../ssb-2026-combined/facts/ (13,632 buzz rows, 831 questions). Outputs: data/processed/plackett_luce_buzzpoints_35.csv, pl_buzz_corr_35.csv.

Part 1 — What Radius's model actually is

I cloned the repo and read model.py, scraper.py, and the 2026-acf-nationals-results/ report.

The big subtlety — his "buzzpoints" model does NOT use buzz position. Despite the repo name, the report is explicit (report.md, line 3):

"unlike other analyses, this does not factor in buzzpoint data whatsoever — only who got which tossup and against which opponents."

So what he fits is a Plackett-Luce / conditional-logit on toss-up OUTCOMES:

• Model. Plackett-Luce is the full-ranking generalization of the top-1 conditional logit (softmax). For toss-up t with the in-room eligible field S_t, the probability player i buzzes first is the softmax P(i wins) = exp(s_i) / Σ_{k∈S_t} exp(s_k). Each player gets one latent skill s_i; fit by maximizing Σ_t [ s_{w_t} − log Σ_k exp(s_k) ] with L-BFGS-B and an L2 penalty (λ=0.1), centered to mean 0. (Bradley-Terry is the 2-player special case; PL is the multi-competitor / full-order generalization — here he only uses the top-1 event, i.e. the conditional-logit slice of PL.)
• Input data. Per (game × toss-up): who was in the room, who buzzed first correctly (winner_idx), and whether it went dead. Buzz position (the actual buzzpoint / celerity column his scraper collects) is loaded but deliberately not used in the likelihood.
• Negs. Instead of dropping negged toss-ups, the negging player is removed from S_t and the rebound winner faces the smaller field (PL handles variable set sizes natively).
• Question difficulty. Each unique toss-up gets a difficulty d_t acting as a "virtual competitor": P(dead) = exp(d_t)/(exp(d_t)+Σ exp(s_k)). This lets dead toss-ups (~30% of events) become data and credits players who play in tough fields. λ_d chosen by game-level 5-fold CV (he picked 0.5; the CV curve is flat 0.05–1.0).
• Validation / uncertainty. A game-level cluster bootstrap (resample whole games, refit) gives 95% skill CIs and P(top-5/10/20). Plus per-category models (7 aggregated cats, ≥~10 gets to qualify), a specialization analysis, and a cross-category correlation matrix.
• Outputs. Overall player skill ranking, per-category rankings, and a by-product toss-up difficulty table (hardest questions).

His thesis, in his own words (report.md, "What this doesn't model"): position "would add real information about margin… For ranking purposes, outcome is most of the signal." That is precisely the claim we can test, because — unlike his ACF data, and unlike our NSBA games — the scibowl.live SSB-2026 export does carry buzz position.

Part 2 — Testing it on data that HAS buzzpoints (SSB-2026)

What the scibowl data actually supports (be honest): Science Bowl is a lockout short-answer format, not pyramidal tossups. The export records a word_index (buzz position) only for the 3,972 mid-question buzzes (location_kind = question/option); the other ~9,660 buzzes fire at end (player answered on the full read, no position). So per-buzz position exists but is sparser and coarser than quizbowl celerity. I built celerity = 1 − word_index/word_count for the winner (end-buzzes → 0), and also used the pre-aggregated avg_celerity in scibowl_player_stats.csv.

I fit three ratings on 2,013 per-question races (352 players, 920 dead), restricted to 185 players with ≥10 correct buzzes:

• A. Outcome-PL — Radius's exact model (softmax who-buzzed-first + a shared dead-tossup difficulty term). No buzzpoints.
• B. Position-PL — same races, but each won race is weighted by the winner's celerity (early buzz counts up to 1.5×, end-buzz 0.5×). Injects buzzpoints. Plus a pure aggregate-celerity ranking.
• C. Non-buzzpoint baselines — raw conversion %, net points-per-game (a PPTF-analog), gets-per-game. The kind of stat available from NSBA-style "who-won-the-tossup" logs with no model.

Rank correlations (Spearman, 185 qualified players)

Held-out prediction (mean over 8 random 75/25 game splits)

Fit each rating on 75% of games; on the held-out 25%, for every non-dead race predict the winner among the eligible field via softmax(rating).

Interpretation

1. Buzzpoints barely move the RANKING. Folding buzz position into the PL model (B vs A) leaves the ranking essentially unchanged: ρ = 0.978, 16/20 of the top 20 identical. Held-out, position buys ~0.7 pp of winner accuracy and ~0.03 nats of log-loss — within one bootstrap SD. This directly confirms Radius: for ranking, who-won is ~all the signal; position adds a sliver.

2. The real lift is MODELING, not buzzpoints. The outcome-PL diverges sharply from raw counting stats (ρ = 0.13 vs gets-per-game, 0.23 vs net-PPG) not because PL uses buzzpoints — it doesn't — but because it adjusts for opponent/field strength and field size. Raw PPG rewards players who simply hear more toss-ups or sit in weak rooms; PL nets that out. So "fitting the data" with a principled opponent-adjusted model beats raw stats by a lot, with zero buzzpoints involved.

3. Held-out, the model is not clearly better than a raw rate at calibration. Conversion % actually had the lowest log-loss (it is well-calibrated on average) but the worst argmax accuracy (it can't discriminate the winner in a strong field). PL/Position-PL win on discrimination. The differences are small — on 2k SSB races nothing is a blowout.

Caveats / honesty about the SSB data. (a) Science Bowl ≠ quizbowl: lockout short-answer, position recorded for only ~29% of buzzes, so this is a weaker test of position than pyramidal celerity would give — it can show position doesn't help much here, but can't prove it never would on richer celerity. (b) A single SSB set means question difficulty is a single shared intercept, not Radius's per-question d_t. (c) 185 players over ~2k races is modest; CIs are wide enough that the A-vs-B tie is "no detectable difference," not "proven identical."

Verdict on Radius's claim

• "WITHOUT buzzpoints you can't do noticeably better" — Supported on our data. His outcome-only model and the buzzpoint-augmented version produce near-identical rankings (ρ 0.978) and statistically indistinguishable held-out prediction. For ranking, buzz position is close to redundant with outcome.
• "Without [a model] it's just fitting the data" — Supported, but the operative ingredient is opponent-adjustment, not buzzpoints. His PL beats raw rate stats decisively, and it does so without buzzpoints — so the value is the model, exactly as he argues.
• Would buzzpoints help WITH enough data? — Plausible but unproven here. He himself flags position's real use is profiling / question-quality / counterfactuals, not ranking. Our SSB position signal is sparse and gave only a within-noise nudge. A richer celerity dataset (full pyramidal buzz curves, many sets) is where position could separate fast-guessers from deep-knowledge players — but that's a margin/profiling question, not the ranking question, and on the ranking question his skepticism holds up.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub