38 — Cleanup Gets vs Clean First-Buzz Wins
38 — Cleanup Gets vs Clean First-Buzz Wins
Idea (Anurag): A toss-up "get" (+4) that comes after the opposing team has already negged (−4 interrupt) is a cleanup — the player converted a question the opponent already gave up, rather than out-buzzing them. A cleanup arguably deserves less credit than a clean first-buzz win. Does distinguishing the two change player valuation / the draft board?
Data: tossups_long.csv (one row per buzz), filtered to the 181
reconciles==True games → 5,902 buzz rows, 3,244 gets, 726 negs.
Three seasons (nsba1_2022, nsba2_2023, nsba3_2025).
1. Classification (CLEAN vs CLEANUP)
Buzz order is not recoverable. Among 2-buzz neg+get toss-ups, the row order shows neg-first vs get-first ~50/50 (247 vs 271) — physically impossible if order were real (a continuation can only happen after the neg). So row order ≠ buzz order, and we cannot use it.
Approximation used: a get is a CLEANUP iff the opposing team has an
is_neg row on the same (game_id, tossup_num); otherwise CLEAN. Grouped by
(game_id, tossup_num), comparing teams. This is the correct call given Standard
Science Bowl rules: if your opponent negged the same toss-up you got, the
question was live-after-neg when you answered, so it's a cleanup. (One anomalous
row has the same team neg+get — a data artifact; negligible, 1/519.) The only
loss vs true ordering: a tiny number of cases where your team also could have
out-buzzed before the opponent negged — not separable here, so we accept it.
2. How common are cleanups?
- Cleanup share of all gets = 15.2% (492 / 3,244). The large majority of gets are clean first-buzz wins.
- By subject: lowest for Math (9.3%) and Energy/Chem (~12%), highest for Physics (20.4%), Earth/Space (17.3%), Bio (16.8%). Plausible: math toss-ups are computational and rarely negged-then-cleaned; physics/ess invite more speculative early buzzes.
- By season: 12–19%, no trend that matters.
- Cleanup-merchant variation is modest. Among 43 player-seasons with ≥25 gets, mean cleanup fraction = 13.7%, SD = 6.3%. Range roughly 4% → 31%.
- Most cleanup-reliant: Rohan G (31% of gets, n=48), Akhil Batchu (27%), akul (26%), Vishnu M nsba3 (24%).
- Most first-buzz: Vishnu M nsba2 (4%), Daniel S nsba2 (5%), Edwin He (5%), sanj (5%). Note Vishnu M flips from 4% to 24% across two seasons — a sign this is noisy, not a fixed trait (see §4).
3. Cleanup-adjusted value vs raw PPTF
Raw PPTF = (4·gets − 4·negs) / tossups_faced. Adjusted rate credits cleanup gets
at weight w<1: (4·clean_gets + 4·w·cleanups − 4·negs) / tuf. n=43 player-seasons
(≥25 gets).
| w (cleanup credit) | Spearman(raw, adjusted) |
|---|---|
| 0.00 (cleanups worth 0) | 0.956 |
| 0.25 | 0.981 |
| 0.50 | 0.989 |
| 0.75 | 0.998 |
- Top-20 board overlap: 20/20 at w=0.5, 19/20 even at the extreme w=0 (cleanups worth nothing). The draft board barely moves.
- Biggest movers (only at the unrealistic w=0): Rohan G drops 11 ranks (8→19), Vishnu M-nsba3 drops 10, Kian Dhawan −7. Risers: Fishface +7, Daniel S-nsba2 +6, Anurag-nsba2 +6. These are the high-cleanup-fraction outliers — and only the most aggressive discount reshuffles them at all.
At any defensible discount (cleanups still worth half a clean get, w=0.5), the ranking is essentially identical (rho 0.989, board overlap 20/20). It's a wash.
4. Honest check — is "cleanup ability" a real, separable, less-valuable skill?
Three tests, all point the same way:
- Cleanup fraction is uncorrelated with overall skill. corr(cleanup_frac, raw_pptf) = −0.006 (p=0.97). High-cleanup players aren't worse; it's orthogonal to value — consistent with it being mostly opportunity/noise, not a skill axis.
- Cleanup fraction does not persist across seasons. Same-player nsba1→nsba2 (n=9, ≥10 gets both): pearson r=0.22 (p=0.58). No evidence of a stable "I'm a cleanup merchant" trait; this is what you'd expect if which toss-ups happen to be negged-first is largely luck of the draw.
- Cleanup-adjusting does not improve out-of-sample prediction. Predicting next-season raw PPTF from prior-season rating (n=10 consecutive player pairs): raw r=0.31, adj-w0.5 r=0.35, adj-w0.0 r=0.36. The adjusted versions are a hair higher but all three are statistically indistinguishable at n=10 (p≈0.31–0.38, none significant). No real predictive gain.
So: cleanup conversion is not demonstrably a separable, less-valuable skill. It is uncorrelated with value, unstable year to year, and gives no OOS lift. What little signal exists looks like opponent/opportunity noise rather than a player attribute worth pricing into a draft.
Small-n caveats
- OOS test has only n=10 consecutive-season player pairs — directional at best, not significant. Stability test n=9. Treat §4.2–4.3 as suggestive.
- Cleanup classification can't separate "you'd have first-buzzed anyway" from "you only got it because they negged"; we conservatively call all opposing-neg gets cleanups, which if anything over-counts cleanups.
- Per-player cleanup counts are small (most have <10 cleanups/season), so individual cleanup fractions are noisy.
Verdict
Not worth incorporating. Cleanups are only 15% of gets, vary little across players (SD 6pp), and discounting them — even to zero — leaves the ranking essentially unchanged (Spearman 0.96–0.99, top-20 overlap 19–20/20). Cleanup propensity doesn't correlate with skill, doesn't persist across seasons, and doesn't improve next-season prediction. The idea is conceptually reasonable but empirically a wash; raw PPTF already captures what matters. File under "checked and ruled out."
Artifacts: scripts/38_cleanup_gets.py,
data/processed/cleanup_gets_player_season_38.csv,
data/processed/cleanup_adjusted_ranking_38.csv.