10_combine_irt.md

10 — Combine Ability Model (latent IRT, de-biked)

10 — Combine Ability Model (latent IRT, de-biked)

Goal. Estimate a latent per-player combine ability (theta) that is robust to small n and to "biking" (late multiple-choice guessing that is +EV under the combine's −1-only-on-wrong-MC penalty). Produce both a raw combine number and a de-biked ability estimate, with standard errors / shrinkage, per overall and per category.

• Script: scripts/combine_ability.py
• Output: data/processed/combine_ability.csv (255 rows; one per player×season)
• Plot: outputs/combine_ability.png
• Env: /home/david/code/nsba/.venv/bin/python · model: girth.grm_mml_eap (GRM)

Two scoring worlds, never mixed. The combine is pyramidal & gameable (points {0,2,3,4}, −1 penalty; earlier buzz scores more). Games are standard Science Bowl (+4/−4/+10, not pyramidal). The IRT model is fit only on combine data; game scoring (PPTF) is used only for out-of-sample validation.

Method

1. Ordinal outcome per question (pyramidal + biking tier)

Each combine question (per-question seasons nsba1/3/4 only) is recoded to an ordered category that encodes both buzz-earliness and the biking signal:

2. Graded Response Model (GRM)

Items = category × question_num (24 items; 20 for nsba1's 5-category Energy era). girth.grm_mml_eap (mixed Bayesian / marginal-MML — built for small samples) gives per-item discrimination + ordered thresholds. Player abilities are the EAP (posterior mean over a N(0,1) prior, custom quadrature) with a posterior-SD standard error — this is where the shrinkage / small-n robustness comes from. theta_overall is fit on all items; each theta_{cat} is fit on that category's items only. Single-level (no-information) items are dropped from the item fit.

3. Aggregate-only season (nsba2_2023)

nsba2 has no per-question data — only category totals + an Overall sheet. So theta_overall is the within-season z-score of the recorded Overall (CS included), and theta_{cat} are z-scored category totals. Reduced resolution: a nominal wide SE (0.8) flags that these are aggregate, not item-level, estimates.

4. De-biking (debiked_overall)

Biking = late WXYZ guessing, +EV because only a wrong MC costs −1 while short-answer buzzes are unpenalized; CS is "unbikable" (cleaner signal). The de-biked GRM is re-fit with the gameable cheap-credit neutralised: - the wrong-MC tier (ord 0) is treated as MISSING, not informative; and - CS items are up-weighted (duplicated) in the overall fit so the unbikable category carries more of the signal.

Honest limitation: the data does not record question type (short-answer vs MC), so we cannot cleanly separate a wrong-MC guess from a wrong short-answer buzz. debiked_overall is therefore a directional adjustment, not a clean correction — treat it as "ability with the most-gameable signal down-weighted."

Identity

Joined to canonical_players.csv (source='combine') on (season, player_raw), with discord_tag used to split nsba2's 4 shared-name pairs. 251/255 rows matched. The 4 unmatched are the second member of each nsba2 shared-name pair (Owen/Aditya/Rishabh/Rohan) — they have no separately-reconciled canonical row, so per the identity guardrail they are kept raw, not force-matched (canonical_id left blank). Their per-category thetas are also blank because the category table carries no discord_tag to disambiguate them.

Results

Validation A — does theta track raw combine? (in-sample)

Pearson corr(theta_overall, raw_overall):

theta closely tracks raw where the per-question signal is rich (nsba3/4). The low nsba1 correlation is a feature, not a bug (next section).

Validation B — does theta predict REAL game scoring? (out-of-sample)

123 players link combine→clean games (≥2 games). Corr with standard-SB game PPTF (points per game):

Combine ability is a moderate predictor of real-game scoring (r ≈ 0.6 in the data-rich seasons) — consistent with the prior that the combine is a noisy, gameable seed, not ground truth.

The de-biking payoff is concentrated where biking happens

On nsba1 — the most-biked, Energy-era season — de-biking flips the ranking toward real ability: debiked (0.415) > raw (0.306) > plain theta (0.037) against game PPTF. Concrete cases (nsba1): - "Ne Dassanayake": raw = 37 (top score) but attempted only 12/20 questions → cheap late accumulation. GRM theta = +0.45 (modest), not elite. - "nihal" / "Avaneesh": attempted all 20 (engaged with hard items), low raw (−1, 1) but theta ≈ −0.3 — less punished than raw, because attempting hard items is informative.

So de-biking earns its keep exactly where the prior (D7) says it should. In the clean per-question seasons (nsba3/4) plain theta and raw are already close and de-biking slightly hurts PPTF correlation — expected, since there is less biking to remove there and down-weighting real cheap-but-correct answers loses signal.

Output schema (data/processed/combine_ability.csv)

canonical_id, season, player_raw, theta_overall, se_overall, theta_{bio,chem,cs,ess,math,phys}, debiked_overall, raw_overall - theta/debiked are on a ~N(0,1) latent scale (shrunk; std ≈ 0.69 within per-question seasons due to EAP shrinkage). raw_overall is the recorded combine number (different units per season — do not pool raw across seasons). - se_overall range 0.40–0.55 (per-question) / 0.80 (nsba2 aggregate). - theta_cs is blank for all nsba1 (no CS in the Energy era) and for nsba2's 4 unassignable shared-name rows.

Limitations / caveats

1. De-biking is partial. Without question type (MC vs short-answer) we cannot isolate true biking; debiked_overall down-weights the most-gameable signal but is directional, not exact. Trust it most on nsba1 (where biking is rampant); on nsba3/4 it adds little.
2. Small n per player. 24 items max (20 for nsba1), fewer attempted. EAP shrinkage is the right tool but means extreme players are pulled toward 0; read theta with its SE.
3. nsba2 is aggregate-only — theta there is just a z-scored Overall (no item resolution, wide SE). Its per-category thetas come from category totals.
4. Cross-season scale. thetas are standardized within season; they are comparable in rank within a season but not an absolute cross-season scale. Game-PPTF validation suggests the seasons are roughly commensurable, but don't over-read small cross-season theta gaps.
5. Validation n is small, especially nsba1 (10 linked players) and nsba4 (17, cross-season since nsba4 has no games). Treat per-season PPTF correlations as directional.
6. Trust games over combine (prior D7): where a player has real game data, that is the stronger signal; this model is for the nsba4 draft pool where combine is often all we have.

Reproduce

/home/david/code/nsba/.venv/bin/python scripts/combine_ability.py

Writes data/processed/combine_ability.csv and outputs/combine_ability.png; prints the validation tables above.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub