15b_sos_robustness_revised.md

15b — SOS-Adjusted Win% + Factual Fixes (REVISION R4)

15b — SOS-Adjusted Win% + Factual Fixes (REVISION R4)

Date: 2026-05-30. Revision of F15/F16/F17/F13/F19 in response to red-team T6, T8, M3, M2.

Lead caveat (unchanged and load-bearing): everything here rests on 41 team-seasons (nsba1=12, nsba2=13, nsba3=16) and 3 game seasons. Schedules are uneven — nsba3 ranges 1–10 games (median 7); 12 of 41 team-seasons have <8 games. Treat every correlation as directional and every Δ below as "did the magnitude move a little or a lot," not as a precise estimate. Two top nsba3 teams (chonk 8-1, boneca anuragu 3-1) sit in a near-perfect dominance cluster, which drives their Bradley-Terry strength to the numerical ceiling (~3.5e5, both pinned at win-prob ≈1.0); their BT-adjusted win% should be read as "≈1.0, ill-determined," not a point value. The RPI metric is immune to this.

This finding does three things the red-team asked for: - (a) Builds a strength-of-schedule (SOS) adjustment for the 41 team-seasons and re-runs the headline win% correlations from F15 (north-star), F16 (coverage), F17 (CS), reporting how far each magnitude moves (red-team T6). - (b) Runs an nsba3 down-weight / exclude sensitivity on the value-board top-N and the north-star ranking (red-team T8). - (c) Fixes two factual slips: the top-25 nsba4-combine count is 3, not 5 (M3), and states which theta column finding 19's buy-low residual actually uses (it is raw theta_overall, r=0.74, not the de-biked column, r=0.67 — M2).

Reproduce: /home/david/code/nsba/.venv/bin/python scripts/sos_robustness.py (first run scripts/north_star.py so team_north_star_features.csv exists). Writes data/processed/team_sos_adjusted.csv and value_board_nsba3_{full,down,excl}.csv.

SOS method

Teams only play within their season, so SOS is computed per season on the head-to-head results of the clean 180-game set. The clean-game reconstruction of wins/win% reproduces team_season.csv exactly (0 mismatches / 41 team-seasons) using the same team-name canonicalization build_master.py applies, so the raw baseline below is the real F15/F16/F17 baseline, not a re-derivation.

Two SOS metrics (both intra-season):

1. Bradley-Terry MLE (bt_winprob_vs_avg) — fit log-strengths via MM iteration (Hunter 2004) so P(i beats j)=σ(r_i−r_j); ties split 0.5/0.5. The adjusted score is the BT win-probability vs a median-strength opponent in that season (rescaled to [0,1]). This is the cleaner SOS object but is the one that diverges for undefeated teams.
2. Iterated opponent-win% (rpi_adj, RPI-style) — start from raw win%, replace each team's score with its average opponent's strength, iterate to a fixed point, then mix 0.5·own_raw + 0.5·opp_strength. Robust to the undefeated-team divergence; a useful check.

How much does SOS move teams at all? Raw win% vs adjusted:

So SOS reshuffles the margins but not the broad ordering. The biggest movers are all short-schedule nsba3 teams (e.g. boneca 4 games 50%→BT rank −6; big shot 6 games 50%→−5; no like blue 8 games 37.5%→+4 because it played a hard slate). The teams whose standing SOS changes most are exactly the <8-game nsba3 teams the red-team flagged.

(a) Headline correlations: raw vs SOS-adjusted win%

Each row: the F15/F16/F17 metric's pearson correlation with raw win% (reproduces the published finding), then with BT- and RPI-adjusted win%, and the shift ΔBT.

F15 — North-star draftable metrics (all 41 team-seasons)

Read. SOS shaves ~0.03–0.06 off every magnitude — small. The metric ranking is fully preserved: ppg_total > pptf_team > n_subjects > |subject_hhi| under raw, BT, and RPI alike. PPTF (the recommended draftable currency) stays a strong, stable correlate (+0.62 → +0.58). F15's conclusion is unchanged.

F16 — Coverage / "don't punt a subject" (games ≥ 5, n = 37)

Read. The punt/coverage effect barely moves under SOS (−0.537 → −0.512 BT, −0.540 RPI; |Δ|≈0.025). Notably, talent correlations move more than coverage under BT (roster_toss_points −0.14, top2 −0.10) — i.e. some of "talent → wins" is "talent → easier realized schedule," whereas the coverage signal is comparatively SOS-robust. This does not rescue F16 from its real fragility (T2: the punt variation is 5 teams, all nsba3 — see the T8 section and the original F16 caveats); SOS just confirms the coverage magnitude is not a schedule artifact. The redundancy/deep_subjects correlation is the most SOS-sensitive (+0.30 → +0.19 BT), reinforcing F16's "redundancy adds little" conclusion.

F17 — CS value (nsba2/nsba3 only)

Read. The team-level CS→win correlation is the most SOS-sensitive of the three findings: cs_ppg drops +0.40 → +0.30 (BT), share_cs +0.42 → +0.36. This is consistent with the causal red-team's T3 (the team CS premium is partly "we were the better/faster team," which SOS absorbs). It does not zero out — CS still correlates positively with winning after SOS — but it widens the already-wide error bar on the team "CS is worth ~Nx" magnitude and adds to the case (made in red-team H1/T3) that the team CS-leverage number is soft. The confound-robust legs of F17 — the player-level CS-mains-vs-generalists conversion gap and theta_cs r=0.60 to in-game CS — are not team-win regressions and are untouched by SOS; those remain the parts of F17 to lean on.

(cs_ppg n=28 because the 1-game team-only nsba3 team 2yellow4brown has no player rows and thus no CS attribution; share_cs comes from team_season directly, n=29.)

Summary of (a)

SOS does not overturn any of the three conclusions, but it discounts the team-level CS magnitude the most — exactly where the red-team said the most aggressive draft action lives.

(b) nsba3 down-weight / exclude sensitivity (T8)

nsba3 is simultaneously the season with the estimated-TUH proxy, the only season with coverage/punt and playoff variation, and the season most like the nsba4 draft target. So we test how much the draftable conclusions depend on it.

North-star ranking, excluding nsba3 (nsba1+nsba2 only, n=25)

Two important structural facts surface here:

1. All breadth/coverage signal is nsba3-only by construction. In nsba1 and nsba2, every team scores all 6 subjects (n_subjects_scored has zero variance → the correlation is undefined), and subject_hhi → win% collapses to −0.01 without nsba3. So F16's coverage finding and F15's "breadth as secondary tiebreak" are entirely carried by nsba3's 16 teams (and within those, by ~5 punting teams). This is the strongest confirmation of red-team T2/T8: the breadth story is one season of data. PPTF and ppg_total, by contrast, transfer cleanly — ppg_total even strengthens without nsba3.
2. PPTF still works without nsba3 (+0.62 → +0.50) and is positive every season individually: nsba1 +0.535, nsba2 +0.561, nsba3 +0.752. The production-rate north-star does not depend on nsba3; the breadth secondary signal does.

Value-board top-N stability (nsba3 weight ∈ {1.0, 0.5, 0.0})

We rebuilt a career-PPTF board (TUF-weighted, shrunk toward the pool mean at M=35 per F11) under three nsba3 weightings and compared the top-N sets:

The board is highly stable to nsba3 down-weighting at the top and degrades only deeper. Reason: the top of the value board is nsba1/nsba2-dominated. In the actual published outputs/player_value_table.csv, the top-14 all have nsba3-TUF share = 0.00; only 8 of the top-25 have any nsba3 data, 6 of them purely nsba3 (Akhil Batchu, Anurag S, Kaden W, Sohil Rathi, Kian Dhawan, Anish A). So:

• Down-weighting nsba3 (0.5) barely moves the top-15 (Jaccard 1.00) — the elite board is carried by exact-TUH nsba1/nsba2 careers.
• Excluding nsba3 entirely only starts to bite at top-25 (Jaccard 0.52): the 6 purely- nsba3 players drop out because they have no non-nsba3 games at all (e.g. at the #15 slot, Akhil Batchu is replaced by Gideon Tzafriri). That is expected — you can't rank a player with zero retained games — not instability of the metric.

Decision-relevant read: the proxy-TUH uncertainty in nsba3 propagates into the value of the 6 purely-nsba3 players, not into the board's top tier. For those 6 (several of whom are nsba4-relevant returners — Akhil, Kian), the VORP/PPTF should be treated as ordinal with a wider band, exactly as F13's TUH-proxy caveat says. For the nsba1/nsba2-anchored top of the board, the nsba3 proxy is irrelevant.

(c) Factual fixes

M3 — top-25 nsba4-combine count is 3, not 5 (finding 13)

outputs/player_value_table.csv top-25 has has_nsba4_combine.sum() == 3: Akhil Batchu (#15), Rohan G (#17), Kian Dhawan (#23). Finding 13 stated "5" twice; both occurrences are now corrected to 3. This strengthens F13's own central caveat ("the board is historical talent, not the nsba4 pool") — the live re-registered pool is even thinner than the finding claimed.

M2 — finding 19's buy-low anchor is raw theta_overall (r=0.74), not de-biked (r=0.67)

Recomputed on the 18 returning nsba4 combine entrants (those with prior NSBA1–3 game history):

The 0.74 in finding 19 is unambiguously theta_overall — so despite the finding's repeated "de-biked IRT" framing, the Track-A residual is measured against the raw theta. Finding 19 has been annotated (text + a new caveat #7) to say so. Impact on the board: raw and de-biked theta correlate 0.81 across these 18, so re-pricing on the de-biked column would drop the anchor to r=0.67 and shift every mispricing modestly without likely flipping the buy-low/fade ordering materially — but the exact mispricing magnitudes are anchor-dependent and should be read as ordinal (as finding 19 already cautions for its n=18, 1–4-game reads).

What changed vs the original findings

• F15 (north-star): unchanged. PPTF survives SOS (+0.62→+0.58) and the metric ranking is identical under both SOS schemes. Now explicitly shown to transfer without nsba3 (+0.50, positive every season).
• F16 (coverage): direction unchanged, generality narrowed. The punt correlation is SOS-robust (−0.54→−0.51), but the no-nsba3 cut makes explicit that all breadth/coverage variation lives in nsba3 (nsba1/nsba2 teams all cover 6 subjects; subject_hhi→win% is −0.01 without nsba3). Read coverage as an nsba3-anchored, ~5-team threshold result.
• F17 (CS value): team magnitude softened. The team CS→win correlation loses the most to SOS (cs_ppg +0.40→+0.30), consistent with reverse-causation (T3). Lean on the player-level conversion gap and theta_cs, not the team correlation.
• F13: "5 of top-25" → 3 of top-25.
• F19: anchor labeled as raw theta_overall (r=0.74), not de-biked; magnitudes are ordinal.

Limitations specific to this revision

• BT divergence for near-perfect dominance clusters. chonk (8-1) and boneca anuragu (3-1) form a near-perfect win cluster, so their BT strength runs to the numerical ceiling (~3.5e5, win-prob ≈1.0). The RPI metric is immune and gives the same qualitative answer (raw↔RPI r=0.994), which is why both are reported. These teams sit at the top under every scheme, so the divergence does not affect the correlation conclusions.
• SOS is intra-season and the schedules are short. With nsba3 medians of 7 games, the opponent-network is sparse; the SOS estimates themselves carry wide error. The honest takeaway is the direction and size of the shift (small for F15/F16, ~25% for F17), not the adjusted point values.
• The value-board sensitivity uses a simplified career-PPTF board (shrunk at M=35), not the full VORP pipeline of F13; it is meant to test top-N membership stability to nsba3 weighting, which it does cleanly, not to re-rank VORP.

Artifacts

• scripts/sos_robustness.py — full reproducible analysis (SOS build + all re-runs).
• data/processed/team_sos_adjusted.csv — 41 teams × {raw win%, BT, RPI, rank shifts}.
• data/processed/value_board_nsba3_{full,down,excl}.csv — top-25 career-PPTF board under the three nsba3 weightings.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub