How we actually modeled a
science-bowl league

Three seasons of games, four of tryouts, and a mess of names to reconcile before any of it means anything.

NSBA is a science-bowl league run like the NBA. Every player takes a combine — a standardized 24-question tryout — then fourteen captains build rosters through a snake draft and play a season of head-to-head matches. Two scoring systems live side by side and are never mixed. Game rounds use standard Science Bowl scoring: a correct toss-up is +4, a wrong interrupt (a "neg") is −4, and a converted team bonus is +10. The combine is pyramidal — buzz earlier for more credit (+4 / +3 / +2), with −1 for a wrong multiple-choice guess — which makes it gameable: because only wrong MC buzzes are penalized, late guessing ("biking") is positive-EV. So the combine is used only as a predictor, never as a measure of value.

The corpus is small and we say so up front. Game scoresheets span three seasons (2022, 2023, 2025), reconciled to a clean set of 180 games — the games that both balance and aren't empty templates:

That yields 41 team-seasons and 207 player-seasons over 181 distinct people, plus 216 draft picks across three drafts. Combine data covers four seasons including the current one — whose games hadn't been played when the models were built. The parser is faithful: the combine reconstructs with zero mismatches, and per-player points reconcile exactly against the buzz-attributed totals.

The two-scoring-worlds guardrail

The single most load-bearing discipline in the whole project: the pyramidal, gameable combine and the standard-scored game are never folded into the same statistic. The two even use different code vocabularies (b/ch/cs/ess/m/p in games, bio/chem/… in the combine), which forces an explicit crosswalk on every join and makes accidental mixing hard. The one time the guardrail slipped — an early finding that "CS points are worth 3.5×" had quietly fed team bonus points into a per-player regression — an adversarial re-audit caught it and the ratio collapsed to ≈1.8× (and lost significance). More on that in §10.

Identity: the hardest unglamorous problem

A player appears as a Discord handle in one place, an in-game nickname in another, a real name in the combine. Every raw string is mapped to a canonical_id by a union-find over five bridge rules (shared Discord tag, exact real name, a nsba1 nickname↔name "Rosetta" sheet, and two team-scoped fuzzy matches). Matches are never forced: 65 ambiguous identities are kept raw, and a bare first name appearing on two different teams is treated as two humans, not merged. The draft→game roster bridge is scored by Jaccard overlap of canonical rosters,

and its per-season quality is wildly uneven — median 0.11 in nsba1 (nicknames vs real names: effectively unlinkable), 1.00 in nsba2, 0.50 in nsba3. This is why nsba1 is essentially games-only and nsba2 carries most of the draft-outcome story.

PPTF and how much of it is real

The base currency is points per toss-up faced — not per game, because bonuses are team-answered and can't be attributed to an individual:

where c = correct buzzes, g = negs, n = toss-ups faced. How much of the spread in PPTF is talent versus sampling noise? A method-of-moments variance decomposition splits observed variance into between-player (true) and within-player (sampling) parts; the ratio is the stabilization constant M, and reliability at n trials follows:

PPTF stabilizes at M ≈ 35 faced toss-ups (reliability 0.5, ~1.5 games); at the median sample of ~150 faced, ρ ≈ 0.81 — we keep about 80% of the signal. An independent empirical split-half, corrected by Spearman–Brown

agrees at 0.82, which is the check that the whole reliability edifice actually holds. Neg rate, by contrast, has M ≈ 23 but a tiny denominator (~12–15 buzzes), so ρ ≈ 0.35 — mostly noise, and shrunk hard toward the league mean.

Before the season, all you have is a cheatable tryout. Here's how much to trust it — spoiler: not much.

Every player takes a 24-question combine (6 categories × 4 pyramidal questions, buzz earlier for more credit). The catch: a wrong multiple-choice guess costs only −1 and short-answer buzzes are free, so late guessing — "biking" — is positive-EV, and a weak player can inflate a raw score. So we don't score it arithmetically; we fit a graded-response IRT model, recoding each question to an ordered outcome (wrong-bike < skip < no-credit < +2 < +3 < +4) and estimating each player's latent ability from the pattern, not the point total:

Fit by marginal maximum likelihood, read out as a posterior-mean (EAP) ability with its error bar; a re-fit that drops the bikeable tier ("de-biking") corrects the worst gaming. It's a tidy model. But here's the honest part, and the reason this section is short: the combine only correlates ≈0.6 with what players actually do in games — it explains about a third of the variance and no more. For a suspected tanker it's a gamed ceiling; for a reliable returner it's an honest floor; for everyone it's a prior we lean on hardest for rookies with no tape and abandon the moment real games arrive. Which they do — and that's where the interesting modeling actually happens.

Baseball's projection playbook — stabilize, regress, age — ported to a buzzer.

Projecting a player is a four-step "Marcel" spine borrowed from sabermetrics. Weight prior seasons by recency (newest gets weight 5, then 4, 3) and by sample; compute the stabilization constant from variance components; then regress the weighted rate toward the league mean with M ≈ 35 toss-ups of padding. The empirical-Bayes shrinkage is a single line:

Per-category projections shrink harder (padding ≈ 5.8, since a subject sees ~⅙ the volume), so a two-buzz "CS specialist" is pulled most of the way back to average — appropriately. Replacement level is the projected rate of the 70th-ranked player (a 14×5 league), and VORP = (θ̂ − 0.298) × 150. Because the talent pool is thin, replacement sits near average, so VORP cleanly separates the elite but flattens the middle.

Growth, net of the illusion of growth

A naïve year-over-year delta shows players improving +0.43 z per edition — but much of that is regression-to-the-mean of a negatively-selected returning cohort. Fitting the change on prior level separates the two effects:

where β₁ ≈ −0.43 is mean reversion and β₀ ≈ +0.41 z is the true net development for a returner starting at the mean (95% CI [+0.13, +0.68]). The rule that follows is strict: apply reversion and development once, together — never stack a growth bump on top of shrinkage, or you count the same climb twice.

Turning points into wins

Two lenses. Win Shares distribute a team's actual wins by scoring share, so they sum exactly to the team total: WS_p = point_share_p × team_wins. Win-Probability-Added models the final margin as a Normal that collapses onto the running score as toss-ups resolve, and credits each buzz's swing to the player:

The residual SD is deliberately un-cocky — a 20-point lead is only ~65% pregame — so early swings aren't inflated; 1.00 WPA ≈ one full win added.

Value versus luck

Three orthogonal skill-minus-variance separators keep us honest: points above expectation (actual gets minus what the win-model expected from the player's actual lineup); Pythagorean wins from point differential, win% = PF^k/(PF^k+PA^k) with k = 2.37 pinned (a free fit overfits 4–5 games/team); and the model residual, realized output minus combine prior, both z-scored. When all three agree a player is under- or over-rated, we believe it; when they don't, we don't.

The field drafts the tryout, not the talent. The inefficiency is real; the size of it, less than we first claimed.

Here is the market inefficiency in two numbers. The field drafts almost mechanically off the visible raw combine — the rank correlation between pick order and combine score is 0.67–0.75 every season. But the combine predicts realized value only at ρ ≈ 0.36–0.43. The room over-anchors on a gameable tryout: that gap is real and directional. How much it's actually worth is where I have to walk myself back.

In betting terms the combine is the number everyone anchors on, so the draft prices like a soft book, and fading it — taking the talent the tryout underrates, passing on the biked-up darlings — is directionally the +EV play. But the magnitude an earlier draft of this post bragged about, "~13 toss-points per slot," does not survive a hard look, and the quant we sent this to caught it cleanly. That figure comes from a single clean-linkage season (nsba2) on about nine names. Pooled across the whole corpus the recoverable edge is closer to ~2–3 points a slot; it roughly vanishes in the season that most resembles today; and the one fully-independent test of "the field mis-drafts talent" comes back null — the combine's own link to the pick-slot residual is 0.007 (p = 0.95). So the inefficiency is real in direction, shaky in magnitude, and quite possibly not exploitable once you're honest about the linkage artifacts. It's the closest thing to an edge in the entire project — which mostly tells you how little edge there is.

The pick-value curve

Joining each pick to its realized win-shares and fitting a curve over the normalized snake slot gives a clean exponential (not linear) decay:

Rescaled so pick 1 = 100, the round-openers are worth roughly 100 / 42 / 22 / 15 / 12 / 11 — a steep top and a long flat tail, and the curve is convex (the top pick is worth more than a linear board implies). The real "loser's curse" isn't a lower mean at the top; it's variance: round-1 busts run 15%, round-6 busts 73%. Early picks are insurance as much as talent.

Steals and reaches

The live draft grade — the one the community bot reports — is a rank delta. A player's value is a balanced z-score of skill and volume, ranked, and compared to where they were drafted:

Positive means drafted later than they've played (a steal); negative means a reach. It runs on a small ± integer scale (~±25) — and is deliberately not the same quantity as the ability-adjusted "value grade" on the draft-value page, which uses IRT/PPTF on a ±70 scale. Two different yardsticks for the same idea. The season's biggest steals were late-round finds (Uddip K at pick 61, +33); the biggest reaches were early picks who underdelivered.

Corrections that didn't matter, and one clustering that did

Two sophisticated adjustments were built and then honestly shelved. Contestation-adjustment — a Plackett–Luce model, P(i wins | field) = e^θ_i / Σ e^θ_j, to strip out the "empty stats" a lone star vacuums on a weak team — reorders the board by almost nothing (Spearman 0.93 with raw PPTF; the confound test is a clean null). Duo synergy — do complementary teammates over-produce? — is a bounded null: zero of eight tests survive multiple-comparison correction. The lesson is that team-building here is additive: draft the best individual value, don't chase fit.

The one structure that held is archetypes. K-means on a fused combine+game per-subject vector finds five soft clusters, and only one — "elite multi-science" (broad and deep, chem/physics core) — reliably wins, carrying ~4× the win-shares and essentially all the VORP. Notably, there is no pure-CS archetype: strong CS players ride with the broad elites.

This is the part everyone actually cares about: we posted a line on every game for six weeks. Here's the scoreboard, and it's humbling.

The model's job was to call games. Across the first six weeks — 36 games with a published line — it went 20-16 straight up (55.6%). Sounds fine, until you meet the baseline: always pick the home team went 27-9 (75%). The dumbest possible heuristic beat our log-sum-exp-softmax-Bayesian machine at every horizon. So let's answer the only question a bettor actually cares about, up front and in bold: would you have made money betting this model? No. 20-16 is roughly break-even before the vig, and blind-betting home would have buried it. Its Brier score is 0.250 — identical to shrugging "50/50" every week. Two caveats, cutting both ways: at n=36 that record is statistically indistinguishable from a coin in either direction (55.6% carries a 95% CI of about [39%, 71%]), so we can't prove it's worthless either — and a Brier tie can hide real signal behind bad calibration, which, it turns out, is exactly what's happening here.

How the engine works (in one breath)

Under the scoreboard is a clean generative model. Each team's per-subject strength is a log-sum-exp pool over its lineup, the three-way toss-up outcome (A / B / dead) is a softmax, the margin is a sum of independent toss-ups and therefore Normal by the CLT, and the win probability is one normal CDF:

It's a nice model. It just runs into a wall: on the historical backtest, combine ability explains only 1-(56.2/57.9)^2 ≈ 6\% of margin variance. A ~150-point game has a ~56-point residual SD — most of the outcome is same-day noise no rating can see.

Why it lost to "pick the home team"

The ~6% ceiling is the irreducible problem. But there was a very reducible one hiding in the calibration, and it's the sharper lesson. In NSBA the home team bans a category, so home teams won 75% of games — yet the model's mean probability on its own favorite was just 0.553. It barely leaned home. Bucket the predictions and the miss is glaring:

The probabilities are both under-dispersed (they hug 0.5 — the standard deviation of p − 0.5 is a measly 0.072) and mis-centered (too low on the home side). The spread shows the same tell in tradeable form: home covered the model's own number 26/36 = 72%. In plain terms, we were so reliably light on home that fading our own spread and taking home would have hit 72% — a cleaner signal than the moneyline, and proof the miss is a systematic side bias, not variance. Two honest hedges, though: those bucket cells are 6–11 games and non-monotonic (noise texture, not a clean slope), and a home offset fit on these same 36 games would overfit exactly like the Week-3 blunder below — the direction (we under-credit home) is safe, the magnitude isn't. And home's 75% rests on a real mechanism (the category ban is a rules artifact), not just a 27-9 record that on 36 games is itself a wide [59%, 86%]. We were tuning the engine while the wheel was pointed slightly the wrong way — an ordinary, humbling failure.

The Week-3 mirage

The arc is worth telling straight, because it's the kind of trap every modeler falls into. Week 1 posted road favorites with full rosters and went 1-5 — the worst week. We fixed the two obvious things: projected lineups (use who actually dressed last time, catch the no-shows) and a dispersion fix, and Week 3 came back 6-0, Brier 0.176 — the peak. Feeling good, we ran an in-sample backtest, saw that a model-only line beat the 70/30 market-futures blend 83% to 50%, and dropped the blend. Then Weeks 4, 5, and 6 all went 3-3 at a coin flip.

Pricing the league's play-money prediction market — and a live trader that lives off variance collapse.

The league runs a play-money prediction market, which needs prices. Take the flagship binary — "will player X score more than 32 toss-up points in some single game this season?" Model per-game points as Normal, price the tail, and compound over the season's games under independence:

The mean comes from the player's rating (a regression with R² = 0.98, on all of thirteen players — read with suspicion); the variance is Bayes-blended with history or a population model. This gave Sanjay S ≈ 79% and Kian D ≈ 40% to hit the line — and the fat tail matters: a high-variance player gets a bigger over-shot than his mean alone suggests. When bets are placed, size them by fractional Kelly against the market price:

Trading the market live

The platform is a Hanson LMSR automated market maker: prices are a softmax of outstanding shares, and the cost to move them is

Because LMSR is shift-invariant, the bot reconstructs the full share vector from just the observed prices and the liquidity parameter b (q_i = b ln p_i), then buys until the marginal price hits fair − 0.02 (the average fill beats the marginal, so it's +EV by construction). The edge itself comes from variance collapse: as toss-ups resolve, the live win probability conditions on the running score and the remaining uncertainty shrinks toward zero:

The model is essentially "collapse the Normal onto the score" — the ability correction is a near-negligible tie-breaker, since β = 0.296 means skill barely moves a single toss-up.

When a buzz happens is mostly a fact about the question, not the person — with one important open door.

How early someone buzzes feels like it should measure how good they are. We modeled buzz order as a Plackett–Luce race: each player carries a latent buzz-speed strength, and the probability they beat a field is a softmax,

with dead questions retained through a shared "virtual competitor" term. Fitting this and folding in the actual buzz position (celerity) turns out to add almost nothing: the celerity-weighted rating correlates 0.99 with a pure who-won-the-buzz model, and buys zero held-out predictive lift. Everything the buzzpoints "know" is already implied by who beats whom. A competing "buzz-rank" feature was actively worse — it promotes low-volume fast buzzers and punishes high-volume elites, measuring style, not quality.

Why averaging buzz timing fails

The deeper reason is a reliability wall. Split-half reliability of a player's average buzz position is r ≈ 0.08 — about 92% noise — because a player buzzes only ~5 times a season, so their mean is dominated by which questions they happened to face. The very same buzz data, aggregated by question, is reliable at r ≈ 0.58: a packet is reused across ~6 games, so buzz position is a genuine question-difficulty signal. The signal that's real in the data is question hardness, not player speed — which is why buzz timing informs our difficulty priors and stays out of the player ratings.

A specific, falsifiable claim — "a few great buzzes reveal category mastery better than the average" — put to the data.

The argument is intuitive and worth taking seriously: information is disproportionate, so a player's two or three fastest conversions on hard clues should reveal what they truly know better than a season of middling averages. To test it, we need an ability truth that is independent of in-game buzzing — the combine IRT θ per category — and we ask which family of in-game features predicts it better.

For every (player, subject) cell with at least three correct buzzes, we built two feature families. Mean/volume: get count, get rate, average buzz depth. Peak/extreme: the single fastest buzz, the mean of the top-2 and top-3 fastest, and conversions on the hardest questions. Then a head-to-head, leave-one-out out-of-sample R² at predicting combine θ. Coverage, stated honestly: 87% of buzzes join to an identity, leaving 81 cells across 43 players — thin, and that thinness is the binding constraint.

The verdict: we can't tell — and being honest about that is the finding. Every family lands below zero out-of-sample, and that does not mean "peak is worse than mean." It means neither recovers the target, so the −0.12 vs −0.05 gap is a comparison of two noise realizations. At n=81 the experiment is underpowered to the point of being uninformative about the contrast at all. And the one wrinkle actually cuts toward the theory: the single strongest correlate is a peak feature — fastest buzz, r = −0.22, 95% CI [−0.42, 0.00], p=0.046 uncorrected — so if the thin data leans anywhere, it leans for "select buzzes," not against. We didn't refute the idea. We found we don't have the sample to test it.

The mechanism is a one-way analysis of variance on buzz depth, and it's decisive: which question it is explains 76% of the variance in how deep a buzz lands; the player explains only about 10%.

Simulating the season 50,000 times — and letting adversaries try to break every finding.

Playoff odds come from Monte-Carlo. Each remaining game draws a margin from the calibrated engine, seeds are set by the real tiebreak (wins, then total points scored), and the bracket is run tens of thousands of times. Any probability is a sample mean, with a Monte-Carlo standard error small enough to ignore:

The title odds come out flat — the favorite sits near 16%, the field bunched behind — precisely because the ability engine can barely separate teams (recall the ~6% margin variance). The championship is mostly seeding and variance, not a runaway. That flatness is a finding, not a bug.

The joint model, and the discipline to report a null

The technical centerpiece is a joint hierarchical Bayesian model — the kind of thing you'd see in a psychometrics paper, fit here with the No-U-Turn sampler. One latent per-(player, subject) ability vector feeds two likelihoods at once — the games and the combine — with partial pooling across players and seasons and a per-player "bias" term that literally learns how much each person games the tryout:

It's the right architecture — a player becomes a six-dimensional vector, not a scalar, which correctly re-rates specialists a flat PPTF buries, and it hands combine-only rookies honestly wider intervals (~1.7×). It converges cleanly (zero divergences). And here is the discipline that matters most: out-of-fold, it does not beat raw combine. Drop a season's games, predict from combine alone, and the joint model lands at LOSO rank-correlation 0.32–0.39 versus 0.38–0.49 for raw combine. Its in-sample win was largely same-season leakage. So we report it for what it honestly is — better uncertainty and specialist re-rating for mid-board decisions, not a proven predictive upgrade.

The red-teams earned their keep

Every major finding was handed to adversarial reviewers whose job was to break it. They did, repeatedly, and it changed the conclusions:

• The "CS is worth 3.5×" headline did not reproduce — it had leaked team bonus points into a per-player regression. Corrected, the ratio fell to ~1.8× and lost significance.
• The undervaluation validation (ρ ≈ 0.60) was shown to be ~90% circular, a shared-PPTF artifact; combine's independent link was a flat p = 0.95.
• The player "growth" curve was partly survivorship + regression to the mean of a negatively-selected cohort.
• A cleaner out-of-sample test showed the usage-bias correction should change the draft board by essentially zero (P(adjusted better) = 13%) — so it was shelved.

Four findings were formally revised; several magnitudes were downgraded to "direction only." A backtest scoreboard scores any ranking on realized draft value, with a naïve combine-greedy board as the bar and a perfect-hindsight oracle as the ceiling. The gap between them is real but — same caveat as §4 — it lives almost entirely in the one clean-linkage season; pooled across the corpus the recoverable per-slot value is a couple of points, not the dozen an earlier draft implied.

What we don't know, stated as plainly as what we do.

A paper that only advertised its wins would be dishonest about a dataset this small. The load-bearing caveats:

• Everything rests on three game seasons and 41 team-seasons. Some per-category cells are a handful of buzzes. Trust rankings and directions; distrust decimals.
• The most decision-relevant season is the most noisily measured. nsba3 — the one most like the current target — records no per-player toss-ups-heard, so its rate stats use an estimated denominator that adds noise and inflates apparent unreliability.
• The game engine is nearly a coin flip. Combine ability explains ~6% of margin variance; full-strength win probabilities span just 0.46–0.54. The largest movable quantity in a season is attendance — worth ~9 points of accuracy — and it isn't predictable from history.
• The most sophisticated model doesn't beat the simplest baseline out-of-fold. The joint hierarchical model earns its place on uncertainty and specialist re-rating, not prediction. We kept it anyway, labeled honestly, rather than overclaim.
• Known bugs are known. The playoff-scenario labeler mislabels three-way record ties (trust the Monte-Carlo percentages instead), and the live trading stack has never run on real money.
• Half our "fitted" constants are in-sample knobs. The 0.7 market blend, the 56/64.5 residual SDs, the 0.35 Kelly fraction — point estimates picked on 40–50 games, not out-of-sample-validated. We caught this lapse on exactly one of them (the Week-3 blend); assume it applies to the rest.

The contribution here is not a single number that wins games. It's a ranked, confound-audited, adversarially-tested map of what a science-bowl season does and doesn't let you know — with every magnitude carrying its error bar and every shortcut carrying its caveat. That's the honest version, and it's the only one worth publishing.