42_backtest_scoreboard.md

42 — Backtest Scoreboard: the bar a new ranking must clear

42 — Backtest Scoreboard: the bar a new ranking must clear

Date: 2026-06-03 Script: scripts/backtest_scoreboard.py (reusable; score_ranking(ranking, season) entrypoint) Outputs: outputs/backtest_baselines.csv (4 rankings × 3 seasons + pooled-trust) Env: /home/david/code/nsba/.venv/bin/python

LEAD CAVEAT — same tiny, leaky 3-season sample as F8/F9. This is a harness, not new data. It replays the historical NSBA drafts and scores any canonical_id -> score ranking by the realized win-share value it would have front-loaded into early picks. Everything rests on nsba2 (gold, 57/59 pool picks scorable); nsba3 is partial (45/69); nsba1 is untrustworthy (18/60, the nickname-namespace problem of F-23/24) and is reported separately, never pooled. The value-board baseline's rank-correlation rests on n=9 scorable-and-ranked nsba2 players — directional only. Read direction and the baseline ordering, not magnitudes.

Claim

A new player-ranking model should be trusted only if it beats combine_greedy (the field's model) on this scoreboard, and its ceiling is oracle. On the gold season (nsba2) the recoverable band is wide and the existing value board already sits well inside it:

The total recoverable edge (combine→oracle) on nsba2 is ≈ 12.7 realized win-shares of early-pick value (33.2 → 45.9), i.e. the field's combine-anchored draft leaves ~57% of the floor→ceiling gap on the table. A new model's job is to move combine_greedy's 33.2 toward 45.9. value_board_greedy already recovers ~7.0 of those 12.7 WS — so the new hierarchical model must clear early-WS ≈ 40 and ρ ≈ 0.49 (vs combine) on nsba2 to be worth trusting, ideally approaching the value board's 40+.

What the harness does

The reusable entrypoint

from backtest_scoreboard import score_ranking
metrics = score_ranking(ranking, season)   # ranking: dict | Series | DataFrame

ranking maps canonical_id -> score (higher = pick earlier). It accepts a dict, a pd.Series indexed by canonical_id, or a DataFrame[['canonical_id', 'score']]. This is exactly the format a new model's ranking CSV must provide:

canonical_id,score
P0107,1.42
P0250,0.93
...

Drop the CSV in, call score_ranking(pd.read_csv(...), 'nsba2_2023'), and the output dict is directly comparable to the baselines below.

Two metric families (a robust one and an intuitive one)

(a) Simulated-roster realized value (the snake replay). For a season we hold fixed the real pick order, which team owns each slot, and each team's roster size, and reassign who is picked: walking the draft in real order, each counterfactual GM takes the highest-ranked still-available player from that season's drafted pool. Because the snake consumes the whole pool, the grand total realized WS is invariant to the ranking — so the discriminating headline is where the value lands: early_ws = realized win-shares the ranking routes into rounds 1–2 (and ws_recovered_per_slot = slope of realized WS on counterfactual slot). A better ranking front-loads more realized value into early picks. This is the F8 "one seat's edge over combine-greedy rivals" reframed at the pool level, and it isolates ranking quality from slot luck.

(b) Rank-correlation / recoverable toss-points (the F8 intuition). Over the scorable picks: Spearman ρ(score, realized WS) and ρ(score, realized toss_points), plus a regression of realized toss_points on the ranking's induced rank → recoverable toss-points per slot (the F8 "~13 pts/slot" frame).

Bridge honesty

Realized value = same-season win_shares / toss_points from player_win_shares.csv (summed over teams for mid-season movers). Only players with a clean draft→game link are scorable; coverage is reported, never silently dropped. Captains (overall_pick 0) are excluded (no draft slot). Unscorable pool members still occupy slots (realized value 0) so the snake stays pool-preserving.

Evidence — the full baseline scoreboard

outputs/backtest_baselines.csv. n_scorable/n_pool = picks with a clean draft→game link / non-captain drafted pool. early_ws = realized WS in rounds 1–2. ρ and pts/slot use only scorable-and-ranked picks (n_corr).

Reading the scoreboard. - The bar = combine_greedy. nsba2 ρ=0.49, early-WS=33.2; pooled-trust ρ=0.41, early-WS=44.1. A new model must beat these. - The ceiling = oracle. nsba2 early-WS=45.9 (ρ=1.0). The combine→oracle gap (33.2→45.9 = +12.7 WS front-loaded; ρ 0.49→1.00) is the total recoverable edge. Pooled-trust gap: 44.1→61.0 = +16.9 WS. - value_board_greedy (F13 shrunk_pptf) already recovers most of the nsba2 ρ edge (0.49→0.80) and ~55% of the early-WS edge (33.2→40.2) — but on only n=9 scorable+ranked players, and it cannot score nsba3 (n=3, ρ=NaN). It is a strong but thin baseline; the new model should match-or-beat it on nsba2 and actually rank the nsba3 pool. - random (floor) confirms the harness has signal: combine and value-board both clear random by a wide margin on both metric families. - The pts/slot=15.3 for value_board on nsba2 is inflated by the n=9 / steep small-sample slope — do not read it as "15 recoverable toss-points/slot"; the credible F8 figure remains ~13 pts/slot. The ρ numbers are the robust comparison.

Confidence

• Medium on nsba2 ordering (random < combine < value_board < oracle holds on both metric families) — this is the one gold season and the result is internally consistent. The headline ordering (combine is the bar, oracle the ceiling, edge is real and ~12–17 WS of early-pick value) is trustworthy as a relative scale.
• Low on magnitudes and on nsba3/value-board ρ (n=3–9). Low on anything pooled-with-nsba1.
• The harness itself is high-confidence reusable: score_ranking accepts dict/Series/DataFrame, pool/total invariance verified, all three input formats tested.

Caveats (read before trusting)

1. Tiny n; nsba2 carries it. The trustworthy scoreboard is 102 scorable picks, 57 of them in the one gold season. Treat the numbers as a ranked leaderboard, not measured magnitudes.
2. The snake total is invariant by construction — that is why the headline is the early-pick (rounds 1–2) value, not the grand total. If a future use wants a single-seat edge number, take early_ws minus the random floor, not the total.
3. value_board coverage is the binding weakness. It only ranks 9–13 of the trustworthy pool, so its ρ/pts-per-slot rest on a handful of players and it cannot evaluate nsba3. The new model must rank the whole eligible pool.
4. Same-season realized value = drafted-for-season. As in F8, oracle and the value board partly see the games they're scored on (nsba3 value board is single-season → circular, per D19/F8); nsba3 oracle is a ceiling-of-a-proxy. The clean ex-ante test lives in nsba2.
5. nsba1 is excluded in substance (Jaccard 0.11, 18/60 scorable). Its row is shown only flagged; never pool it.
6. Survivorship (F8 T5): combine-high no-shows/tankers are missing from the scorable pool, so combine_greedy looks more predictive than it is for the real draftable population — the true recoverable edge is likely larger than the +12.7 WS measured here.
7. win_shares embeds team context (F9): a great player on a bad team scores low. Appropriate for "did the pick help you win," but it is not a pure player rating.

How Phase 2's new ranking plugs in

1. Produce a CSV canonical_id,score (higher = better) covering the season's eligible pool.
2. m = score_ranking(pd.read_csv("my_ranking.csv"), "nsba2_2023").
3. Compare m["early_ws"], m["rho_ws"], m["pts_per_slot"] to the combine_greedy row above. Trust the model only if it beats combine_greedy on nsba2 (early-WS > 33.2, ρ > 0.49) and ranks the nsba3 pool (n_corr ≫ 9). To add it to the baseline file, write a ranking_<name>(season) fn in the script mirroring ranking_value_board and add it to build_baselines().

Reproduce

/home/david/code/nsba/.venv/bin/python scripts/backtest_scoreboard.py

Prints the scoreboard and writes outputs/backtest_baselines.csv.

outputs/backtest_baselines.csv schema

ranking, season, trustworthy, n_pool, n_scorable, n_ranked_in_pool, coverage, pool_total_ws, early_ws, early_ws_per_pick, early_ws_share, early_tp, n_early, ws_recovered_per_slot, rho_ws, rho_tp, pts_per_slot, n_corr. season ∈ {nsba1_2022, nsba2_2023, nsba3_2025, pooled_trust}; early_* = realized value the ranking front-loads into rounds 1–2; rho_*/pts_per_slot computed on the n_corr scorable-and-ranked picks only.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub