NSBA Draft Analyticsembargoed · 2026-06-06

A7_ml_methodology.md

A7 — ML / Statistics Methodology for NSBA Player Projection

A7 — ML / Statistics Methodology for NSBA Player Projection

Project: NSBA (science-bowl league drafted like the NBA) Author: ML/stats methodology research agent Date: 2026-05-30 Status: Research + recommendation. No code shipped here; this is the modeling blueprint.

0. The data, as it actually sits on disk

Before any method, anchor on what we have (verified against the raw files in data/raw/):

Combine (data/raw/combine/*.xlsx) — one sheet per player. Rows are {Category}{1..4} (e.g. Bio1..Bio4, Phy1..Phy4), i.e. 6 categories × 4 pyramidal positions = 24 items. Each cell carries a Points value in {0 (skip/miss), +2, +3, +4} and a Penalty column for the -1 WXYZ mistakes. Harder questions come later in each 4-item ramp. The combine is gameable: strong players sometimes tank to drop in the draft.

Games (data/raw/games/{season}/*.xlsx) — one workbook per round, a rosters sheet plus one sheet per head-to-head match. Match sheets are a question × player matrix: each row is a tossup tagged by Subject (phy/bio/mat/ess/che/ene/cs), each player column holds that player's buzz result on that tossup (+4 / 0 / -4). ~10 games/season, 3 seasons (2022, 2023, 2025), with NSBA4 (2026) in progress.

Scale of the problem: dozens of players, ~24 combine items, ~10 games × ~20 tossups of game data per player-season. This is small, noisy, and partially biased data. Every method below is chosen for that regime, not for the large-n regime most ML tutorials assume.

Key structural facts that drive modeling choices: - Both the combine and the games are already IRT response matrices (persons × items, ordered-categorical outcomes). The combine and games are the same kind of object; we can fit IRT on both and put them on a common ability scale. - Categories/subjects give a natural multidimensional structure (6–7 latent abilities per player), but with this little data we cannot afford 7 free, uncorrelated abilities per player — we need pooling across dimensions. - Players recur across seasons and improve a lot (8th grade → college, ~4 year careers). Season is not exchangeable noise; it is a developmental trajectory. This is an aging-curve problem layered on top of the rating problem.

1. Item Response Theory (IRT) — the core combine + game model

1.1 Plain explanation

IRT models the probability of a person's response to a test item as a function of a latent person ability θ and item parameters. The workhorses:

The magic of IRT for us: it jointly estimates player ability and question difficulty from the same buzz outcomes. A player who only beat easy questions and a player who beat hard ones can get correctly different ability estimates even with the same raw point total — exactly what a combine scored by raw points cannot do.

Sample-size caveat from the literature: classical (MMLE/JMLE) 2PL wants N≈500–1000 for stable item parameters; Rasch is usable around N≈250–500 (ResearchGate – sample size for Rasch/IRT, Schroeders & Gnambs 2025, Sample-Size Planning in IRT). We have dozens, not hundreds. The resolution is Bayesian hierarchical IRT, where priors on item parameters stabilize estimation: an optimized hierarchical 2PL recovers accurate item and trait estimates at N≈100 (Sheng et al., Optimized Bayesian Hierarchical 2PL for small-sample calibration, PMC7262992; robustness follow-up, PMC10700496). We will lean on this hard.

1.2 Encoding pyramidal buzz points: Graded / Partial-Credit IRT (GPCM)

A buzz is not binary. The combine yields ordered outcomes {miss/skip < +2 < +3 < +4} (and games {−4 < 0 < +4}). The right family is polytomous IRT for ordered categories:

Why this matters: a player who consistently buzzes in the superpower window (+4) on a category has higher latent ability than one scraping +2s, even at equal item counts. GPCM extracts that; raw-points scoring throws it away. This directly powers floor/ceiling and "who buzzes early" captain selection.

The −1 WXYZ penalty is a separate process (a recklessness/accuracy nuisance), not a higher ability tier. Two clean options: 1. Keep it out of the GPCM categories and model penalties as a small per-player risk random effect (a Poisson/Bernoulli count of bad WXYZ buzzes) estimated alongside ability. 2. Or fold a guess/penalty floor in via a 4PL-style lower asymptote — but that's the unstable route; prefer option 1.

1.3 Category-specific ability: Multidimensional IRT (MIRT)

Six categories (seven with CS) → each player has a vector of abilities θ_p = (θ_bio, θ_phy, θ_che, θ_mat, θ_ess, θ_ene/cs). This is confirmatory MIRT: each item loads only on its own category's dimension (a "simple-structure" / between-item multidimensional model). The estimated inter-factor correlation matrix Σ_θ is itself a deliverable — it tells you how much science-bowl ability is one general factor (g) vs. genuinely category-specific, which informs draft strategy (specialists vs. generalists).

Critically, at our n we should not fit 6 free uncorrelated abilities. Two stabilizers, used together: - A bifactor / two-tier structure: one general science ability θ_g loading on all items, plus specific category factors orthogonal to it. This concentrates most signal in θ_g (stable) and treats category factors as small residual specialties (shrunk hard). mirt supports bifactor and two-tier dimension-reduction directly (Chalmers, mirt JSS, mirt docs). - A hierarchical prior on the θ covariance so category abilities are partially pooled toward θ_g and toward each other (see §2).

This is the single most important modeling decision in the whole project: a hierarchical, (bi)multidimensional, partial-credit IRT is our core combine model, and the same machinery scores the game data.

1.4 Mapping to our data (precise)

IRT concept NSBA object
Person p player
Item i a combine question (Bio3) or a game tossup (round R, q 12, subject che)
Latent ability θ_p player science-bowl ability vector by category
Ordered response category combine {0,+2,+3,+4}; game {−4,0,+4}
Item difficulty b_i / steps β_{i,k} how hard/early that question is to buzz
Item discrimination a_i how sharply that question separates strong/weak
Dimension d category (bio/phy/che/mat/ess/ene-cs)
Inter-factor corr Σ_θ g-factor vs. specialization structure

Combine items are shared across all players in a season (everyone sees the same 24) → item parameters are well-identified per season. Game tossups are mostly seen by only the 2–4 players in that match → game item parameters are weak; anchor them by pooling difficulty by subject and round-difficulty tier rather than estimating each tossup freely.

1.5 Implementation sketch

Two viable stacks; we recommend the Bayesian one as primary because it gives us uncertainty for free (§7) and the hierarchical priors we need at this n.

Primary — Bayesian hierarchical (partial-credit, multidimensional) IRT in PyMC

import pymc as pm, numpy as np
# long-form: rows = (player_idx, item_idx, category_idx, k_response in {0,1,2,3})
with pm.Model() as combine_irt:
    # population structure
    sd_g   = pm.HalfNormal("sd_g", 1.0)
    theta_g = pm.Normal("theta_g", 0, sd_g, shape=n_players)          # general ability
    # small, shrunk category-specific deviations (bifactor specifics)
    sd_cat = pm.HalfNormal("sd_cat", 0.5)                              # weakly-informative
    theta_s = pm.Normal("theta_s", 0, sd_cat, shape=(n_players, n_cat))
    theta = theta_g[:, None] + theta_s                                 # player×category
    # item params with hierarchical priors (this is what makes small-n work)
    a = pm.LogNormal("a", 0.0, 0.3, shape=n_items)                     # discrimination > 0
    # ordered thresholds per item (GPCM steps); ordered transform keeps them sorted
    b = pm.Normal("b", 0, 1, shape=(n_items, n_steps),
                  transform=pm.distributions.transforms.ordered)
    eta = a[item] * (theta[player, cat] - b[item])                     # build category logits
    pm.OrderedLogistic("y", eta=eta, cutpoints=..., observed=resp)
    idata = pm.sample(target_accept=0.95)

Notes: use pm.OrderedLogistic (or a hand-rolled GPCM likelihood) for the ordered buzz tiers; put LogNormal/HalfNormal priors on a and sd_* per the small-sample hierarchical-2PL recipe (Cauchy/exponential hyperpriors on variance components also work — PMC7262992). Reparameterize non-centered (theta = mu + sd*z) to fix divergences.

Secondary / sanity-check — classical MIRT. - Python: py-irt (Pyro/variational, scales, supports 1PL/2PL/4PL — arXiv:2203.01282, github nd-ball/py-irt) or girth / girth_mcmc (github eribean/girth) for quick dichotomous/polytomous fits and data simulation. - R (via rpy2 if needed): mirt is the gold standard for confirmatory MIRT, GPCM, bifactor, two-tier (JSS v48i06). Worth using to validate the PyMC model against a mature implementation.

Plan: prototype/validate in girth/mirt, ship the hierarchical model in PyMC.

1.6 Pitfalls

1.7 Priority: P0 (highest). This is the spine of the model.

2. Hierarchical / multilevel Bayesian models & partial pooling

2.1 Plain explanation

With dozens of players and a handful of observations each, per-player independent estimates are garbage (high variance, overfit). Partial pooling ("random effects" in frequentist terms) shrinks each player's estimate toward the population mean by an amount that depends on how much data that player has and how much true between-player variance there is. Players with little data get pulled hard toward the mean; players with lots of consistent data barely move. No-pooling is noisier than partial pooling, especially in small samples (rstanarm – Hierarchical Partial Pooling).

This is the same mechanism that makes the hierarchical IRT in §1 work — it is the unifying idea of the entire stack. Three nested grouping levels apply to NSBA: 1. player within category (shrink rare-category abilities toward the player's own g and toward population), 2. player within season (shrink a thin season toward the player's career trajectory), 3. player within population (shrink everyone toward the league mean).

2.2 Mapping + choosing priors

2.3 Implementation sketch

Same PyMC model as §1.5 — the hierarchy is the priors on theta_g, theta_s, and Σ_θ. For a non-IRT, faster scaffold, brms/rstanarm (R) or bambi (Python, formula syntax over PyMC) fit hierarchical GLMs quickly for exploration.

2.4 Pitfalls

2.5 Priority: P0. Inseparable from the IRT core.

3. Empirical-Bayes shrinkage for rate stats (James–Stein, beta-binomial)

3.1 Plain explanation

For simple rate stats — buzz-conversion (buzzes won / buzzable opportunities), category accuracy, neg rate — full Bayesian IRT is overkill. Empirical Bayes gives 80% of the shrinkage benefit for ~5% of the effort.

3.2 Mapping

3.3 Implementation sketch

statsmodels (BetaBinomial, or GLM with a beta-binomial family), or fit α,β by method-of-moments / MLE in a dozen lines of NumPy; scipy.stats for the Beta. For the regression version, a small PyMC/bambi beta-binomial.

3.4 Pitfalls

3.5 Priority: P1. Fast, robust, great for dashboards and as a baseline

the fancy model must beat.

4. Mixed-effects models for player value

4.1 Plain explanation

A GLMM with random intercepts for player, game, and season is the frequentist twin of §2 and a fast way to get a single "player value" number with proper partial pooling, e.g. buzz_value ~ subject + home + (1|player) + (1|game) + (1|season). The (1|player) BLUP is a shrunk player-value estimate; (1|game) absorbs "this was a hard packet" and (1|season) absorbs era effects.

4.2 Mapping

4.3 Implementation sketch

statsmodels MixedLM (Python) for Gaussian; for the binary/ordinal buzz outcome use a GLMM — statsmodels BinomialBayesMixedGLM, or R lme4 glmer via rpy2 (statsmodels MixedLM). Sports precedent uses exactly lmer(y ~ ... + (1|player) + (1|opponent)) (Curtis – multilevel modeling for sports scientists).

4.4 Pitfalls

4.5 Priority: P1 as baseline/decomposition; superseded by the

Bayesian IRT for the final product.

5. Regression to the mean & the "stabilization point"

5.1 Plain explanation

Extreme observed performances are part skill, part luck; next time they regress toward the mean. The amount you should regress = the reliability of the stat at the sample size you have. Sabermetrics formalizes this as the stabilization point: the sample size where split-half reliability hits ~0.7 (≈50% of future variance predictable). At that n you weight observed vs. league-mean 50/50; at half that n, you weight observed only ~⅓ (FanGraphs – Sample Size; Medvedovsky – NBA Stabilization Rates and the Padding Approach).

The beautiful part: partial pooling (§2) and shrinkage (§3) are regression-to-the-mean done right — the shrinkage factor n / (n + n_stabilize) is mathematically the optimal regression weight. So we don't bolt RTM on; it falls out of the hierarchical model. But computing the stabilization point explicitly is still valuable for communication and for the "how many games until we trust a rookie" question.

5.2 Mapping

5.3 Implementation sketch

Split-half reliability: random-split each player's tossups many times, correlate halves, Spearman–Brown correct, solve for the n giving r=0.7 (NumPy/scipy). Or read it straight off the fitted hierarchical model's variance components: n_stabilize = within_var / between_var.

5.4 Pitfalls

5.5 Priority: P1. Mostly emergent from §2–3; compute explicitly for

explanation + rookie-trust rules.

6. Handling the tanking bias

6.1 The problem

Some strong players deliberately tank the combine to drop in the snake draft. Their combine θ is biased low and is not missing-at-random — it's strategically suppressed. Naively trusting the combine misranks exactly the players a savvy GM most wants. Three complementary attacks:

6.2 (a) Detect tanking as a residual — the cheapest, most robust signal

We have an external, harder-to-game ability signal: game performance. Tanking shows up as a player whose game/external ability ≫ combine ability. Concretely: fit IRT θ from the combine and (separately) θ from games, put them on a common scale, and flag large positive θ_game − θ_combine residuals. Corroborate with reputation/age priors and prior-season θ (returning stars who suddenly post a weak combine are prime suspects). This is structurally the same as AI "sandbagging" detection: compare expected vs. exhibited performance and flag the gap (sandbagging detection survey). This is the primary recommendation — it needs no special estimator, just the two-source comparison we already get for free.

6.3 (b) Robust estimation — don't let tanked points dominate

When combining combine + game signals, use robust/heavy-tailed likelihoods so a suspiciously-low combine doesn't drag a player's estimate down: - Student-t (instead of Normal) observation noise on the combine contribution, so outlier-low combine scores are down-weighted automatically. - Trimmed / Winsorized estimators if doing a non-Bayesian aggregate (Cizek – Semiparametric robust estimation of truncated/censored regression).

6.4 (c) Censored / truncated / selection model — the principled version

Treat a tanked combine as a censored observation: the observed combine is a lower bound on true ability for flagged players (true θ ≥ observed θ). Fit a mixture / selection model: each player is "honest" (combine ∝ ability) or "tanking" (combine is censored below), with mixture membership informed by the game-vs-combine residual and priors. This is a Tobit/Heckman-style censored likelihood inside the Bayesian model.

6.5 Implementation sketch

6.6 Pitfalls

6.7 Priority: P0 for (a) detection-as-residual (cheap, high value);

P1 for (b) robust likelihood; P2 for (c) full censored mixture.

7. Projection under uncertainty — distributions, not points

7.1 Plain explanation

A draft pick is a decision under uncertainty; a point estimate hides the thing GMs care about — floor vs. ceiling. The Bayesian stack (§1–2) yields a full posterior distribution over each player's (multidimensional, possibly future-season) ability. Summarize as: - Floor / ceiling = posterior quantiles (e.g. 10th / 90th percentile of projected ability or projected fantasy-points). - Posterior predictive simulation → distribution of next-season outcomes, not just current ability, by pushing posterior θ through the development curve and the game-scoring likelihood.

This is the single biggest advantage of going Bayesian over a point-estimate ML model: uncertainty is first-class and free.

7.2 Mapping

7.3 Implementation sketch

From PyMC idata: az.hdi(idata, hdi_prob=0.8) for floor/ceiling; pm.sample_posterior_predictive for next-season outcome distributions; ArviZ for forest/ridge plots of player abilities with credible intervals.

7.4 Pitfalls

7.5 Priority: P0. It's the output format of the whole project.

8. Why heavy ML will overfit — and what to use instead

8.1 Plain explanation

Random forests, gradient boosting, and neural nets have effective complexity far exceeding what dozens of players × a few features can support. RF in particular routinely shows near-perfect in-sample R² with poor out-of-sample R² (To Bag is to Prune, arXiv:2008.07063) — it will look great in training and fail to generalize. With n this small, flexible learners memorize noise; regularization "encourages simpler, more generalizable patterns... particularly important for small sample sizes" (Analytics Vidhya – regularization).

The deeper point: our problem has strong, correct structure (items have difficulty; ability is latent and multidimensional; players develop). IRT / hierarchical models encode that structure as inductive bias, which is exactly the regularization that lets you learn from little data. A tree ensemble throws that structure away and tries to rediscover it from 40 rows — hopeless.

8.2 Preferred regularized alternatives

8.3 Pitfalls

8.4 Priority: P0 as a guardrail (do not build the product on trees/NNs);

penalized GLM is a fine P2 alternative scorer.

9. Validation strategy with tiny n

9.1 The right scheme: Leave-One-Season-Out (LOSO), time-respecting

Because players recur and improve, random k-fold leaks the future into the past and leaks a player's other-season rows into his held-out rows. Use: - Leave-One-Season-Out / forward-chaining: train on seasons {1,2}, predict season 3; train on {1}, predict 2. This mirrors the real task (project the next draft) and respects time, like TimeSeriesSplit (scikit-learn TimeSeriesSplit / leakage). With only 3 seasons this gives 1–2 honest evaluation folds — few, but real. - Leave-One-Player-Out within a season for the combine→game-rank task (does combine θ predict a held-out player's game performance?), making sure no row of that player is in training.

9.2 Pitfalls (these are where projects die)

9.3 Metrics

9.4 Priority: P0. Without honest LOSO validation, every number above is

unfalsifiable.

10. Recommended modeling stack (assembled)

  1. Core: Bayesian hierarchical, bifactor-multidimensional, partial-credit (GPCM) IRT in PyMC, fit on combine and game response matrices, sharing a player-level ability prior across seasons with a development (aging) term so returning players' thin seasons borrow strength and rising youth aren't over-regressed. (§1, §2, §5; aging-curve precedent: Bayesian NBA aging curves, survivor-bias caveats, BP).
  2. Tanking: detect as the game-minus-combine θ residual (P0), add a Student-t combine likelihood for robustness (P1), reserve a censored mixture for later (P2). (§6)
  3. Output: posterior floor/ceiling per player and per category, plus posterior-predictive next-season projections. (§7)
  4. Baselines / fast paths: beta-binomial EB shrinkage for rate stats and a statsmodels mixed-effects model for a transparent player-value number and variance decomposition. The fancy model must beat these. (§3, §4)
  5. Explanation: explicit split-half stabilization points for communication and rookie-trust rules. (§5)
  6. Validation: leave-one-season-out + leave-one-player-out, all preprocessing/IRT-scaling done inside folds; WAIC/PSIS-LOO only for model comparison; report interval calibration. (§9)
  7. Explicitly avoided: unregularized RF/GBM/NN as the ranking engine — they overfit at this n; structure-as-regularization (IRT/hierarchy) wins. (§8)

Tooling: PyMC + ArviZ (primary), girth/py-irt and R mirt via rpy2 (validation), statsmodels (mixed-effects, beta-binomial baselines), scikit-learn (penalized-GLM alternative + CV plumbing), NumPy/pandas (ETL from the xlsx).

Sources

IRT / small-sample / GPCM / MIRT - Optimized Bayesian Hierarchical 2PL for small-sample (N≈100) calibration — https://pmc.ncbi.nlm.nih.gov/articles/PMC7262992/ - Robustness of optimized hierarchical 2PL — https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10700496/ - Schroeders & Gnambs (2025), Sample-Size Planning in IRT — https://journals.sagepub.com/doi/10.1177/25152459251314798 - Sample size for Rasch/IRT (dichotomous) — https://www.researchgate.net/publication/8890461_The_Effect_of_Sample_Size_for_Estimating_RaschIRT_Parameters_with_Dichotomous_Items - Auxiliary item info, GRM small/medium n: EB vs Hierarchical Bayes — https://pmc.ncbi.nlm.nih.gov/articles/PMC10664746/ - GPCM overview (Assessment Systems) — https://assess.com/what-is-the-generalized-partial-credit-model/ - GPCM parameter recovery (MMLE vs MCMC) — https://arxiv.org/pdf/1809.07359 - mirt package (JSS v48i06) — https://www.jstatsoft.org/v48/i06/ - mirt docs (bifactor/two-tier/confirmatory) — https://philchalmers.github.io/mirt/docs/reference/mirt.html - py-irt (arXiv:2203.01282) — https://arxiv.org/abs/2203.01282 ; repo — https://github.com/nd-ball/py-irt - girth — https://github.com/eribean/girth

Hierarchical Bayes / partial pooling / priors - rstanarm — Hierarchical Partial Pooling for repeated binary trials — https://cran.r-project.org/web/packages/rstanarm/vignettes/pooling.html - Gelman (2006), Prior distributions for variance parameters — https://projecteuclid.org/journals/bayesian-analysis/volume-1/issue-3/Prior-distributions-for-variance-parameters-in-hierarchical-models-comment-on/10.1214/06-BA117A.pdf - Informative priors on hierarchical variances (Stan dev blog) — https://statmodeling.stat.columbia.edu/2017/10/11/partial-pooling-informative-priors-hierarchical-variance-parameters-next-frontier-multilevel-modeling/

Empirical Bayes / shrinkage / RTM / stabilization - Robinson — Understanding empirical Bayes (baseball) — http://varianceexplained.org/r/empirical_bayes_baseball/ - Robinson — Beta-binomial regression (baseball) — http://varianceexplained.org/r/beta_binomial_baseball/ - Efron & Morris / James–Stein (CASI ch.7) — https://efron.ckirby.su.domains/other/CASI_Chap7_Nov2014.pdf - FanGraphs — Sample Size / stabilization — https://library.fangraphs.com/principles/sample-size/ - Medvedovsky — NBA Stabilization Rates and the Padding Approach — https://kmedved.com/2020/08/06/nba-stabilization-rates-and-the-padding-approach/

Mixed-effects - statsmodels MixedLM example — https://www.statsmodels.org/stable/examples/notebooks/generated/mixed_lm_example.html - Multilevel modeling for sports scientists — https://ryan-curtis.netlify.app/post/multilevel-modeling-and-effects-statistics-for-sports-scientists-in-r/

Tanking / robust / censored - Cizek — Semiparametric robust estimation of truncated/censored regression — https://www.sciencedirect.com/science/article/abs/pii/S0304407612000589 - Sandbagging detection (expected vs exhibited) — https://www.emergentmind.com/topics/sandbagging-detection-techniques

Overfitting / regularization - To Bag is to Prune (RF in-sample vs out-of-sample) — https://arxiv.org/pdf/2008.07063 - Prevent overfitting using regularization — https://www.analyticsvidhya.com/blog/2021/07/prevent-overfitting-using-regularization-techniques/ - Lasso/Ridge vs overfitting — https://www.analyticsvidhya.com/blog/2021/09/lasso-and-ridge-regularization-a-rescuer-from-overfitting/

Validation / leakage - TimeSeriesSplit / leakage in time series — https://codecut.ai/cross-validation-with-time-series/ - Avoiding data leakage in CV — https://medium.com/@silva.f.francis/avoiding-data-leakage-in-cross-validation-ba344d4d55c0 - Nested vs non-nested CV (sklearn) — https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html - Wainer & Cawley — nested CV is overzealous for most practical applications — https://arxiv.org/pdf/1809.09446

Aging curves / survivor bias - Bayesian aging curves of NBA players — https://link.springer.com/article/10.3758/s13428-018-1183-8 - Survivor bias in baseball (BP) — https://www.baseballprospectus.com/news/article/59491/an-approach-to-survivor-bias-in-baseball/ - Aging curves via regression + imputation — https://link.springer.com/article/10.1007/s10479-022-05127-y

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub