23_draft_to_outcome.md

23 — Draft → Team Outcome (D4): does the draft *matter*?

23 — Draft → Team Outcome (D4): does the draft matter?

Question. This is the end-to-end validation of the whole thesis. Link drafted rosters (draft_rosters.csv) to realized standings (team_season.csv win_pct) and ask: did teams that drafted more projected value / better coverage / earlier-combine players actually win more? Is the draft → wins chain signal, or noise? And which roster-construction features (top-end talent, depth, coverage, CS presence) best separated winners from losers historically?

TL;DR — three claims, in descending order of how much the data supports them.

1. Talent wins games. A roster's realized toss-up production rate is the single best predictor of win% — corr(win%, mean drafted-player PPTF) = +0.68 on the clean-linkage subset (90% bootstrap CI [+0.37, +0.85], excludes 0), and the star/top-player rate separates winners from losers (t=2.65, p=0.017). So the roster's talent → wins arrow is real and reasonably strong, even at this n.

2. But the combine-seeded draft only loosely controls that talent. The arrow that is supposed to make the draft work — combine ability of who you drafted → how good your roster actually was — is weak and not distinguishable from zero: corr(top-2 drafted combine θ, realized roster PPTF) = +0.13 to +0.27 (CIs include 0). The draft's input signal (a gameable combine) is a noisy handle on the output (wins). Top-end combine θ does point at win% (nsba2 r=+0.53) but with a CI [−0.10, +0.85] that crosses zero — directionally right, statistically unproven.

3. Coverage / CS design did NOT separate winners from losers — because every drafted team already had it. Design coverage (r=+0.07 with win%), and the apparent "no-CS-answerer → wins" blip, are saturation artifacts, not levers. This confirms finding 18: the snake draft makes broad coverage table stakes, so coverage cannot be a differentiator. The differentiator is how much production rate you concentrate, not breadth.

The honest verdict: the draft "matters" in that rosters with more scoring talent win — but the historical record can only weakly attribute that to the combine-driven draft choices, partly because the cleanest test exists in one season (nsba2, n=12) and the combine→realized link is loose. Lead with caveats; this is a 12-to-34 team-season result built on top of a fragile draft↔game team-name bridge.

The binding constraint: drafted rosters barely link to game rosters (except nsba2)

Before any outcome test, the draft team (a captain handle, e.g. Coby, polaris, karthik) must be mapped to the in-game team name (e.g. mount ellis, okc). They are not the same strings. The only reliable bridge is canonical_id overlap: match each game team to the drafted roster it shares the most players with (best Jaccard). Doing that exposes how recoverable each season's draft→outcome link actually is:

• nsba1 is effectively unlinkable. Only 19% of drafted players show up in clean games — the nickname/handle fragmentation documented in the data dictionary (nsba1 game sheets use nicknames, draft + canonical use real names; ~82% name-only draft match, 0-loss game join, but the two namespaces barely intersect). The 8 "mapped" nsba1 teams overlap their drafted roster by a single player on average (Jaccard 0.11). nsba1 carries no trustworthy draft→outcome signal and is reported only for completeness.
• nsba2 is the gold standard. Draft teams are the game teams (Jaccard 1.0 for 8/12); every drafted player played. This is the one season where "the roster you drafted" and "the roster that played" are the same object.
• nsba3 is partial. 63% of drafted players played; team-folding (16 game-teams from fewer real franchises) and trades drop the median Jaccard to 0.50.

Everything below is therefore reported four ways: all 34 mapped team-seasons, the clean-linkage subset (Jaccard ≥ 0.5, n=19), nsba2-only (n=12, gold), and per-season. Trust the nsba2 and Jaccard≥0.5 columns; treat the pooled-34 as contaminated by the nsba1 near-random mapping.

Method

• Data. team_season.csv (41 team-seasons), draft_rosters.csv / draft_picks.csv, player_season_master.csv (realized PPTF), combine_ability.csv (de-biked IRT θ; the pre-game draft-design signal). Env /home/david/code/nsba/.venv/bin/python. Script: scripts/draft_to_outcome.py.
• Bridge. Greedy best-Jaccard match of each game team to a drafted roster by shared canonical_ids (data/processed/draft_team_bridge.csv).
• Two feature families per (season, team):
• (A) Pre-game draft design (legal predictors) — built from same-season combine θ of the drafted players (76–92% of drafted players carry a θ): design_top1_theta (best player), design_top2_mean (top-end pair), design_mean_theta, design_depth_theta (3rd-best = depth), design_n_cov (# of 6 subjects with ≥1 credible answerer at θ>0.5 — the roster-design coverage the causal red-team T2 asked for, computed pre-game, not realized-in-game), design_has_cs.
• (B) Realized roster quality (ceiling/diagnostic, not a pure draft input) — from player_season_master: real_mean_pptf / real_top_pptf (drafted players' actual toss-up rate), real_total_pts.
• Outcome. win_pct over the clean 180-game set. Reported with pearson + spearman, win/lose median-split t-tests, a permutation test, the design→realized→win causal chain, and bootstrap 90% CIs (because n is tiny).

Results

1. Realized roster talent → wins (the strong, robust arrow)

• Bootstrap 90% CI (clean subset): realized mean-PPTF → win = [+0.37, +0.85] — the one headline here whose interval excludes zero. nsba2: +0.53, CI [+0.20, +0.85].
• Winner/loser split (median win%): on the clean subset, winning teams' drafted players posted higher PPTF — top-PPTF t=2.65, p=0.017, mean-PPTF t=2.43, p=0.026.
• Permutation test (nsba2, mean-PPTF→win): observed r=0.53, perm p=0.075 — borderline at n=12 but consistent with the bootstrap.
• Star-weighted, not depth-weighted. Regressing win% on (top-PPTF, mean-PPTF) jointly, the mean term dominates (coef 1.35 vs 0.04) — but design_depth_theta (the drafted 3rd-man) is ~0 with win% (r=0.03 clean). Depth of credible answerers is not what separates teams; aggregate scoring rate is.

Read: rosters that actually scored faster won more. This is the part of the thesis that holds up.

2. Combine-design → realized talent (the weak arrow that should make the draft "work")

This is the crux of "does the draft matter vs noise." The draft's job is: pick high-θ players → get a high-production roster → win. Arrow 2 (talent→wins) is strong (§1). But arrow 1 — combine θ of who you drafted → how good your roster actually was — is weak:

The combine→realized-PPTF correlation (+0.13 to +0.27, all CIs spanning 0) is much weaker than finding 12's player-level combine→game r≈0.62 — expected, because (a) this is at the roster-aggregate level (averaging washes out, and a team is only as linked as its noisiest member), and (b) the combine is gameable (the guardrail). So the historical record shows that drafting high-θ players gave only a loose grip on realized roster quality.

Does top-end combine θ point at wins at all? Directionally yes, but unproven:

Top-end combine θ → win% is +0.53 in the gold-standard season but flips slightly negative in nsba3, and every CI crosses zero. So "draft the highest-θ players and you'll win" is not refuted, but not demonstrated in the historical sample — it is exactly as strong as the combine→game link is loose (§ arrow 1).

3. Coverage / CS design did NOT separate winners (saturation, per finding 18)

• Design coverage is flat with win%. Drafting credible answerers across more of the 6 subjects did not predict winning — because nearly every team already did it (finding 18: 95–100% cover all six). Coverage is table stakes the draft auto-provides, so it cannot separate teams. This is the redteam-T2 "design-coverage" rebuild, and it confirms finding 18's conclusion on a pre-game design metric: coverage is solved, not a lever.
• The design_has_cs "−0.34" is an n=4 fluke, not a finding. Only 4 of 26 nsba2/3 mapped teams lacked a credible CS combine answerer (θ_cs>0.5) — and all 4 happened to be strong teams (win% .67–.78). That produces a spurious negative correlation off 4 points. It does not mean "CS hurts"; it means CS-design is near-constant and any correlation on the 4 exceptions is noise. Consistent with the redteam's warning (90/91) that the CS headline is fragile — and with finding 18 (CS covered 96.6% of the time).
• Realized n_subjects_scored does correlate (+0.56) but that is the realized, talent-entangled breadth (good teams convert ≥1 subject everywhere) flagged in findings 15/16/L2 — not a draft-design lever. The clean design version (design_n_cov) is flat.

4. Per-season summary of "what separated winners"

The only signal positive and material in both trustworthy seasons (nsba2, nsba3) is realized scoring rate. Top combine θ works in nsba2 but not nsba3; the HHI/breadth effect is an nsba3-only phenomenon (matching findings 16/18 and red-team T2's "the coverage result is essentially nsba3").

Interpretation — does the draft matter?

Yes, conditionally, and weakly attributable to the combine. Decompose the thesis chain:

   combine θ of picks  --(weak, r≈0.1–0.3)-->  realized roster scoring rate  --(strong, r≈0.5–0.7)-->  wins
        [arrow 1: the draft's predictive grip]                [arrow 2: talent wins]

• Arrow 2 is solid: scoring talent wins games (clean-subset r=0.68, CI excludes 0). The league is star/rate-driven, not depth- or coverage-driven — drafted depth (3rd-man θ) and design coverage are ~0 with wins.
• Arrow 1 is loose: the gameable combine is a noisy handle on realized roster quality (r≈0.1–0.3, CIs include 0). So the draft decision (who you picked, scored by combine) only weakly determined the roster quality that won games. This is the honest limit on "the draft matters": talent matters; the combine-seeded draft is a blurry instrument for acquiring it.
• Coverage is not a lever, it is a floor the draft already clears (finding 18 confirmed on a pre-game design metric). The exploitable margin is concentrating scoring rate / top-end talent, not spreading for breadth or hunting a CS specialist for coverage's sake (the field already covers CS).

Actionable for nsba4: prioritize projected production rate (PPTF / VORP from finding 13) over combine rank, over depth, and over coverage. The draft's job is to maximize roster scoring rate; combine θ helps rank-order candidates but is a weak predictor of realized output, so blend it with game-history PPTF where available (finding 12/13) and do not over-pay for it. Coverage and a CS body are cheap insurance the draft provides almost automatically — don't reach for them at the expense of rate.

Limitations & caveats (lead with these)

• n is tiny and the linkage is the dominant uncertainty. 34 mapped team-seasons, of which only nsba2 (12) is gold-standard. The headline rests on 12–19 team-seasons. Every correlation carries a bootstrap CI ~±0.3 wide; treat direction and ranking as the finding, not magnitudes.
• nsba1 is excluded in substance. 19% drafted-player game-appearance, Jaccard 0.11 — the draft→game bridge is near-random there (nickname namespace mismatch). The pooled-34 numbers are diluted by this; prefer the Jaccard≥0.5 / nsba2 columns.
• The combine is gameable (guardrail) — it is a seed signal only, never mixed into value. Arrow-1's weakness partly reflects that gaming, not just aggregation noise.
• Win% is schedule-confounded (red-team T6): nsba3 schedules range 1–10 games; no SOS adjustment. Short-schedule teams have high-variance win%. This compounds the small-n.
• nsba3 PPTF uses the estimated-TUH proxy (paired layout, no per-player TUH) — the realized-PPTF→win result leans partly on a proxy denominator (red-team T8). It reproduces in nsba2 (exact/partial TUH), which is reassuring.
• Realized roster features are not pure draft inputs. Arrow 2 uses what players did, which is downstream of the draft. The clean pre-game test is arrow 1 + top-θ→win, both of which are the weaker results — by design, that is the honest read.
• design_has_cs −0.34 is an n=4 artifact, not evidence CS is bad. Do not act on it.
• Greedy Jaccard bridge can mis-assign a team when two drafted rosters overlap a folded game team; the per-pair overlaps are in draft_team_bridge.csv for inspection.

Artifacts

• scripts/draft_to_outcome.py — full reproducible analysis.
• data/processed/draft_team_bridge.csv — draft_team ↔ game_team mapping with overlap/Jaccard.
• data/processed/draft_outcome_features.csv — 34 mapped team-seasons × design + realized features + win%.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub