11_reliability.md

11 — Reliability & Stabilization of Game Rate Stats

Question. For the key game rate stats — overall PPTF (toss-up points per toss-up faced), per-category PPTF, and neg rate — how reliable are they at a typical NSBA sample size, and at what number of trials do they "stabilize" (become trustworthy)? The stabilization point M is the shrinkage constant for regressing a player's raw rate toward the population mean.

Scope. GAME data only (standard Science-Bowl scoring). Combine is never mixed in. Source: player_season_master.csv (207 player-seasons, nsba1/2/3), game_player_box.csv and games_meta.csv inner-joined to the clean game set (reconciles==True AND not both-zero → 180 games). Script: scripts/reliability.py.

Method

A rate stat is a per-trial mean: rate_i = (sum of per-trial values) / n_i. The observed spread of player rates mixes true talent spread with sampling noise. Decompose (Lord–Novick / KR / method-of-moments):

Var_observed(rate) = sigma2_between (talent)  +  E[ sigma2_within / n ]  (noise)

sigma2_within = per-trial variance. For PPTF each faced toss-up scores in {+4, 0, −4}; per player E[x²]=16·(correct+neg)/n, so within-var = E[x²] − rate². For neg rate the trial is Bernoulli, within-var = p(1−p).
Solve for sigma2_between, then the stabilization point

M = sigma2_within_bar / sigma2_between

Reliability of a player's rate at n trials: rho_n = n / (n + M).
Shrinkage toward the population mean: weight = M / (M + n) (the empirical-Bayes shrinkage constant). The classic "stabilizes at M trials" point is where rho = 0.5, i.e. n = M.

Cross-check. An independent empirical split-half: for each player-season with ≥2 clean games, randomly split games into two halves, recompute the stat on each half, correlate across players, and Spearman–Brown adjust (full sample = double a half). Averaged over 400 random splits. This needs only game-level box counts, so it works across all three seasons including nsba3 (where per-game TUH is unrecorded). For PPTF the split-half uses points-per-game as the n-robust proxy (PPTF's exact per-game faced count is not recoverable for nsba3).

Headline numbers

Stat	trial unit (n)	M (stabilization)	median n	reliability ρ @ median n	shrink-to-mean @ median n
PPTF overall	tossups faced	35	150	0.81	0.19
neg rate	buzzes (correct+neg)	23	~12–15	~0.35	~0.65

The contrast is the whole story. PPTF is denominated in tossups faced (median 150), so its denominator is large and PPTF is already trustworthy at a typical sample. Neg rate is denominated in buzzes the player took (median only ~12–15), so even though it stabilizes faster in trials (M≈23 < 35), a typical player has far fewer of those trials → neg rate is mostly noise and must be heavily regressed.

Empirical split-half (independent cross-check, Spearman–Brown corrected)

Stat	split-half reliability	matches variance-components?
Points-per-game (PPTF proxy)	0.82	yes — vs ρ=0.81 from M=35
Conversion / neg rate	0.44	yes — vs ρ≈0.35 (noisy, small n)

The two methods agree, which is the main validity check. (Conversion% and neg rate share the same correlation by construction: on the correct+neg denominator, conv = 1 − neg_rate, so they are mechanically identical in reliability — not a bug.)

Per-category PPTF stabilization

Per-category denominators are ~1/5 the overall, so every category is shakier.

Category	n_players	M	median faced	ρ @ median
Biology	106	24	32	0.57
Math	94	33	32	0.50
Chemistry	99	36	32	0.47
Earth/Space	107	43	35	0.45
Physics	111	44	32	0.42
Comp Sci	61	69	27	0.28
Energy (nsba1)	43	72	30	0.29

Bio and Math stabilize fastest (most concentrated talent spread → larger between-variance → smaller M). CS and Energy have the largest M (≈70) and the fewest faced toss-ups → per-category CS/Energy rates from games are the least reliable single-season signals and should be shrunk almost entirely to the mean unless a player has an unusually large CS sample. (Note the prior that combine CS is the cleaner signal — but that's the combine; here we are measuring the game per-category rate, whose small denominator makes it noisy regardless.)

The per-category M's use a coarse per-trial-variance proxy (we have category points but not category correct/neg splits in the master), so treat the category M's as ±25% accurate ranking-grade estimates, not precise constants. The overall PPTF and neg-rate M's use exact correct/neg counts and are tighter.

Implied shrinkage rule (the practical output)

Regress a player's raw rate to the population mean with weight M/(M+n):

estimate = (n/(n+M)) * raw_rate  +  (M/(n+M)) * population_mean

PPTF overall (M=35, pop mean 0.332 pts/faced):

tossups faced n	ρ	shrink to mean
20	0.36	0.64
50	0.59	0.41
75	0.68	0.32
100	0.74	0.26
150 (median)	0.81	0.19
200	0.85	0.15

neg rate (M=23, pop mean 0.177):

buzzes n	ρ	shrink to mean
10	0.30	0.70
20	0.47	0.53
32	0.58	0.42
50	0.69	0.31
100	0.81	0.19

So for a typical drafted player: trust their PPTF (keep ~80% of the signal, regress ~20% to mean), but distrust their neg rate (keep only ~1/3, regress ~2/3 to the 17.7% league baseline) unless they have 40+ buzzes on record.

Limitations / caveats

Small samples. 207 player-seasons total; per-category n's run 43–111. Empirical split-half needs ≥2 games, leaving 192 player-seasons. The M's are point estimates without bootstrap CIs here — treat as ranking-grade, not 3-significant-figure.
PPTF denominator is partly estimated. tossups_faced is exact (lineup TUH) only for nsba1 and ~13 nsba2 games; nsba3 (paired layout records no TUH) and most of nsba2 use a season-mean-per-game estimate (tuh_source = estimated/mixed for 129/207 rows). Estimated denominators add a little extra noise to nsba3 PPTF, which would inflate M slightly (make PPTF look less reliable than it is); the points-per-game split-half (which is denominator-free per game) agreeing at 0.82 is reassuring.
Talent is not stationary across seasons or game types. This is a within-pool, cross-player reliability (split on games within a season), not a season-to-season predictive reliability — a player's true PPTF can also drift between seasons, which this number does not capture.
Per-category within-variance is a proxy (category correct/neg splits aren't in the master), so category M's are ±25%. Overall PPTF and neg-rate M's use exact counts.
Variance-components vs split-half can diverge for thin stats. For neg rate the two estimates (ρ≈0.35 vs 0.44) bracket each other rather than coincide, reflecting genuine estimation noise at median ~12 buzzes — the qualitative conclusion (heavily regress neg rate) is robust to which you pick.

Artifacts: scripts/reliability.py (regenerates all numbers). Inputs: data/processed/player_season_master.csv, game_player_box.csv, games_meta.csv, canonical_players.csv.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub