41_scibowl_difficulty_anchor.md

41 — scibowl.live Difficulty Anchor (transferable priors for the joint model)

41 — scibowl.live Difficulty Anchor (transferable priors for the joint model)

Goal. Fit subject/question difficulty on the LARGE external scibowl.live corpus (13,632 buzzes; 8,426 tossup buzzes; 366 players; 420 distinct tossup questions; 100% reconciled to recorded game scores — see F1) and export it as transferable priors a joint NSBA model can consume. The combine IRT (combine_ability.py) fits difficulty separately each season on tiny NSBA samples (24 items, ~50–90 players); difficulty is the hardest-to-estimate layer. This anchor lets the joint model borrow strength from a ~50–100× larger sample for the 5 shared subjects.

• Script: scripts/A41_scibowl_anchor.py
• Outputs: data/processed/scibowl_difficulty_priors.csv, data/processed/scibowl_player_subject_ability.csv
• Env: /home/david/code/nsba/.venv/bin/python · girth.grm_mml_eap (subject ordinal + player ability), girth.twopl_mml_eap (per-question), EAP under N(0,1).
• Source: scibowl-org/stats/ssb-2026-combined/facts (same raw log as F1).

Scale / parameterization (what the next agent consumes)

All latent quantities are on a girth IRT logit scale fit under a N(0,1) ability prior — the SAME convention as combine_ability.py theta — so difficulty/discrim priors are on a compatible scale even though the OUTCOME WORLD differs (see caveats).

scibowl_difficulty_priors.csv (one row per scibowl subject + a CS sentinel):

Subject & question difficulties are RELATIVE (centered) — transfer the ordering and spread, not an absolute zero.

scibowl_player_subject_ability.csv (366 players): canonical_id, match_method, player_name, team_name, tournament, theta_overall, se_overall, theta_{bio,chem,phys, math,ess,energy}. theta_overall/se_overall are the GRM EAP posterior mean/SD (N(0,1) scale). Per-subject thetas are standardized best-buzz ordinals (5-level resolution) — a cross-validation seed, not a precise per-subject IRT score.

Claims / evidence / confidence

C1 — Subject difficulty ordering on the shared 5 (PRIMARY)

Claim. On the shared 5 NSBA subjects, hardest→easiest is Earth/Space ≫ Biology > Physics ≈ Math ≈ Chemistry. Energy (no NSBA map) is the easiest of the 6. Evidence. Two independent methods agree:

corr(subject_difficulty_prior, q_difficulty_mean) = +0.74; Earth/Space is the clear hardest in BOTH. Confidence: high for ESS-hardest and the broad ordering; medium for the tight Physics/Math/Chemistry cluster (within ~0.06 logits — effectively a tie).

C2 — scibowl difficulty ordering DISAGREES with NSBA finding 20 on the shared 5

Claim. Finding 20 (NSBA games) found Bio easiest, then ess/phys/chem/math, CS hardest. On the shared 5, scibowl finds Earth/Space the HARDEST and Bio second- hardest — close to the reverse for those two. Evidence. F20 nsba3 subject conversions: bio 0.921 (easiest), ess 0.844, phys 0.833, chem 0.803, math 0.794. scibowl: ess 0.764 (hardest), bio 0.806, phys 0.831, math 0.835, chem 0.839. Chemistry/Math flip from "harder" (NSBA) to "easier" (scibowl); Earth/Space flips from mid to hardest. Why. Different question writers (NSBA packet-submission vs real Science Bowl tournaments), a much stronger/deeper scibowl field, and — load-bearing — F20 itself warns subject conversion is not stable across contexts (NSBA Bio swung 0.79→0.92 across seasons). The priors should be used as a soft Bayesian prior the NSBA data can override, NOT a hard constraint. Confidence: high that the orderings differ; medium on the causal attribution.

C3 — Celerity tracks ability (sanity vs F1)

Claim. Higher-ability players buzz earlier; harder subjects do NOT mean slower buzzing. Evidence. corr(theta_overall, avg_celerity) = +0.41, corr(theta_overall, tu_conv%) = +0.29 (both positive — F1 found team-level corr(win_pct, celerity) = +0.78). NOTE a deliberate non-result: subject difficulty and subject celerity are POSITIVELY correlated (+0.85) — the hardest subject (Earth/Space) is buzzed EARLIEST. This is exactly why the celerity-tiered GRM threshold was rejected as the difficulty signal (it conflates speed with difficulty); the conversion-logit prior is speed-independent. Confidence: high.

C4 — Question-level difficulty IS recoverable (unlike NSBA games)

Claim. Per-(player, question) difficulty is estimable here; 373 of 406 tossup questions are usable in a 2PL. Evidence. Each logical tossup is replayed across many games (median 8, mean 15.7) with ~20 distinct buzzers, so a [player × question] binary matrix exists. Exposure = a player appeared in a game where the question was played; correct=1 iff they had a correct buzz on it. girth.twopl_mml_eap fit cleanly (kept questions with ≥10 exposed players and both classes). Confidence: high for the ordering; see W3 for the level caveat.

C5 — 24 scibowl players reconcile to NSBA canonical_ids

Claim. 24 scibowl players map to NSBA canonical_id (extra cross-val observations). All via the authoritative canonical_players bridge (the 23 nsba_name_guess matches are a subset). Evidence. 24 source='scibowl' rows in canonical_players.csv share a canonical_id with a non-scibowl source (match_method='real_name'). Sanity: their mean theta_overall = +1.12 vs field +0.02 — NSBA draftees who also play real Science Bowl are elite in the scibowl field, as expected. Top reconciled: Anurag Sodhi (P0093), Ryan Karim (P0505), Ishaan Kabra (P0283), Kian Dhawan (P0346). Confidence: high.

Limitations / caveats (load-bearing)

• W1 — CS has NO external anchor. scibowl has 6 categories incl. Energy and no Computer Science. Energy does NOT map to CS (different subject). The joint model must estimate CS difficulty from internal NSBA data ONLY. The priors file ships a COMPUTER_SCIENCE sentinel row with empty priors and has_external_anchor=0 so CS cannot silently borrow. (CS is the HARDEST subject in NSBA per F20 — exactly the one we can't anchor.) Energy carries an anchor but nsba_subject is blank (consume it only if a future analysis adds an Energy axis).
• W2 — Two scoring worlds (D5). scibowl games are STANDARD Science Bowl (+4/−4/+10), not pyramidal like the NSBA combine ({0,2,3,4}, −1 MC). These priors describe standard-SB difficulty. The combine GRM's ordinal outcome (pyramidal earliness tiers) is a different outcome space — transfer the difficulty scale and ordering, not the combine item params verbatim.
• W3 — Question-level difficulty is competitive-confounded. Standard SB lets only one team answer per instance, so a strong player whose opponent/teammate buzzed first records an "undeserved" 0. This inflates apparent question difficulty and deflates discrimination. The raw 2PL b sat ~+3.4 (diluted exposure pool); we center it to grand-mean 0 so it is interpretable as relative difficulty. Trust the question-level numbers as an ordering, not absolute levels.
• W4 — Player ability has a ceiling. With only 6 subject items in a coarse best-buzz ordinal, top players saturate: 13 players who hit the earliest-correct tier in all 6 subjects share the max theta_overall = +1.92. 267/366 thetas are distinct. Use the per-subject/overall theta as a cross-val SEED, read with se_overall (~0.6); it is not a high-resolution ability score.
• W5 — Subject difficulty is NOT stable across contexts (per F20). scibowl and NSBA disagree on the shared-5 ordering (C2). Use as an informative prior that NSBA data can override, not a hard constraint.

Reproduce

/home/david/code/nsba/.venv/bin/python scripts/A41_scibowl_anchor.py

Writes the two CSVs and prints the subject + per-question difficulty tables and the reconciliation count.

NSBA Draft Analytics · embargoed until after the SSB draft · ← hub