16b_coverage_design_revised.md

A16b (R2) — Coverage, rebuilt from roster DESIGN (not realized production)

A16b (R2) — Coverage, rebuilt from roster DESIGN (not realized production)

Revision of A16 / F16 in response to red-team T2 (91_causal_redteam.md, also M-list in 90_methodology_redteam.md). Date: 2026-05-30. Reproduce: /home/david/code/nsba/.venv/bin/python scripts/analysis_16b_design_coverage.py → writes data/processed/team_design_coverage_16b.csv.

Lead caveats (read first)

• Tiny n, and the test effectively lives in 24 nsba2+nsba3 team-seasons. 32 bridged teams at games≥5 (nsba1=8, nsba2=11, nsba3=13). Design CS coverage exists only in nsba2/nsba3 (nsba1 combine has no CS — and no Energy category at all), and the nsba1 bridge is weak (see next).
• Design coverage requires a draft→game bridge, and it is imperfect. The win% outcome lives on game team names; the design measure lives on draft team names; the two name spaces do not match. I bridge them by canonical-id roster overlap (greedy 1:1). Bridge quality: nsba2 mean overlap 0.91 (12/13), nsba3 0.66 (14/16), nsba1 only 0.22 (8/12) — nsba1 game sheets use nicknames, so nsba1 design coverage is noisy and I report results with and without it.
• Combine is used as a PREDICTOR only. theta_cat defines roster design; it is never mixed into the win% outcome or any game value stat. Per guardrails.
• theta_cat is a gameable, ±-noisy seed signal (combine is tankable). Drafted rosters average 3.8–5.4 of 6 players with a same-season combine row, so a few "covered/punted" calls rest on partial roster combine data.

TL;DR — the coverage rule does not survive a design-based rebuild

The original F16 headline ("don't leave a subject uncovered → ~7 win-pts/punt, corr −0.54") is a property of REALIZED in-game coverage and does not reproduce when coverage is measured from roster DESIGN.

• At no design threshold (theta ≥ −1.0, −0.5, 0.0, +0.5, +1.0, or within-season top-25/40/50%) does design-punt → win% reach significance; |corr| ranges 0.04–0.24, all p > 0.16.
• Design and realized punts barely agree: corr(design_punts, realized_punts on the same measurable subjects) = −0.119 (essentially zero, and not the positive agreement you'd want). Only 1 of the teams flagged as a punter is flagged under both definitions (the other team, nsba3).
• The 5 realized-punt teams (usmic, okc, fishnoobs, no like blue, the other team — all nsba3) were, by design, NOT short on coverage: each drafted at least one above-average combine answerer in every measurable subject. Their "punt" is a failure to convert in games, not a hole in the roster they built.

Conclusion: the "don't punt a subject" rule is tautological with being good. "Realized punt = a subject your team didn't score in" is downstream of being a weak (or short-schedule) team. When coverage is measured from the pre-game roster you drafted, it carries no detectable, talent-independent signal for winning at this sample size. The red-team's T2 critique is confirmed, not refuted.

What was tested

For each historical game team (which carries win%), bridge to its drafted roster (draft_rosters.csv), and for each combine-measurable subject mark it covered if ≥1 drafted player has combine_ability.theta_<cat> ≥ threshold. #punts = # measurable subjects with no such player. Re-run the F16 tests: univariate corr(#punts, win%), talent-controlled partials, drop-one leverage, within- vs cross-season. Design subject slate: nsba2/nsba3 = {bio,chem,cs,ess,math,phys}; nsba1 = {bio,chem,ess,math,phys} (no CS/Energy combine).

Results in detail

1. Univariate punt → win% (games ≥ 5, n = 32)

Compare F16's realized measure, which strengthened monotonically as the floor rose (−0.40 → −0.54 → −0.57 → −0.60). The design measure shows no such monotone strengthening and no significance anywhere. Win% by design-punts at the primary threshold is essentially flat: 0 punts → 0.504 (n=25), 1 → 0.366 (n=3), 2 → 0.471 (n=4) — the n=3 dip is not monotone and not distinguishable from noise.

2. Talent controls kill what little is left (games ≥ 5)

F16's realized partial-over-talent was −0.37 (p=0.026). The design partial is ~0 and sometimes flips sign. There is no talent-independent design-coverage effect.

3. Drop-one DFBETA leverage (base r = −0.108)

Most influential single teams: dropping Okeh City Okehios moves r to −0.197 (Δ −0.089); dropping usmic moves it to −0.162 (Δ −0.054); dropping the other team moves it to −0.060 (Δ +0.048). No single team rescues the effect into significance, and the leverage is mixed-sign — i.e. the near-zero correlation is fragile in both directions on individual teams. (For contrast, F16's realized −0.54 collapsed to −0.32 when its two 0%-win nsba3 teams were dropped — also leverage-driven.)

4. Within-season vs cross-season

Unlike the realized measure (whose entire punt-variation was 5 nsba3 teams), design punts are spread across seasons — but the effect is null in every slice, including the within-nsba3 slice where the realized effect was strongest (F16: nsba3 realized −0.76). Restricting to the well-bridged nsba2/3 teams (dropping the noisy nsba1 nickname bridge) does not change the conclusion.

5. CS specifically (nsba2 + nsba3, games ≥ 5, n = 24)

Per F17/F18, CS is the subject of interest. By design, CS coverage is universal: every bridged nsba2/nsba3 team drafted ≥1 above-average (theta_cs ≥ 0.0) CS answerer — there are zero CS design-punters. corr(best drafted CS theta, win%) = −0.036 (p=0.87). If anything the contrast runs backward: teams whose best CS designer is strong (theta_cs ≥ +0.5, n=20) won 0.429; the 4 teams without such a player won 0.731 — small-n noise, but certainly no positive CS-design-coverage edge. This is consistent with F18's "the draft already solves CS coverage 96.6% of the time": once everyone drafts a CS answerer, having one cannot separate winners — only converting in-game (F17's player-level conversion gap) can.

6. Which subjects get design-punted

At theta ≥ 0.0 (games ≥ 5): nsba2 — chem×2, ess×2, phys×1, bio×1; nsba3 — chem×1, ess×1; nsba1 — chem×1, math×1, phys×1. CS is never design-punted. The design holes are scattered (chem/ess most common) and do not line up with the realized holes (which were CS- and ESS-depth-flavored), reinforcing that the two measures capture different things.

Interpretation: why design ≠ realized here

The realized measure conflates three things the design measure strips out: 1. Being good. A strong team converts ≥2 in every subject because it wins buzzer races everywhere; a weak team fails to in several. "Realized punt" is partly just "weak team," which is mechanically anti-correlated with win%. 2. Games played / schedule length. Realized per-subject totals scale with games; nsba3's short, uneven schedules (1–10 games) make a subject read as "punted" simply from low exposure. (All 5 realized punters are nsba3.) 3. Sub/lineup endogeneity. Realized coverage depends on who actually buzzed, which is itself an in-game choice.

The DESIGN measure (who you drafted × their tryout ability) removes (1)–(3). When it does, the coverage→win% relationship disappears. So F16's "don't punt" correlation was largely the tautology good teams don't have empty stat lines, not a roster-construction lever you can pull on draft night.

Verdict on the F16 coverage rule

Bottom line. Measured from roster design rather than realized production, the "don't punt a subject" rule does not survive. At n=32 (effectively 24 nsba2/3) the design-coverage→win% signal is indistinguishable from zero and vanishes under any talent control. The actionable residue is narrower than F16 claimed: drafting a credible answerer in every subject is table stakes the draft already provides (every bridged team did, CS included) — it is not a differentiating lever, and the leveraged value (if any) is in conversion and depth in-game (F17), not in filling a coverage hole on draft night. The realized-coverage "punt penalty" should be read as a symptom of weak/short-schedule teams, not a roster-construction rule.

Artifacts

• data/processed/team_design_coverage_16b.csv — one row per bridged team (season, game team, drafted team, bridge overlap, design #punts/#covered at 5 thresholds + 3 percentile rules, realized & design talent z-controls).
• scripts/analysis_16b_design_coverage.py — full reproduction.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub