Fair-Value Model — Build, Red-Team & Backtests

kalshi-mm • 2026-06-03 • all briefs

TL;DR

I red-teamed the fair-value spec with 7 subagents, then built and backtested a first model on the data we already captured. Three things you should know — and one of them challenges the plan:

1. Raw Stockfish WDL is badly miscalibrated for elite human chess — *worse than just guessing the base rate* (skill score −1.12). The "skip the model, just use the engine's win/draw/loss" shortcut fails on real data. A human-calibrated mapping is genuinely needed and adds +8.9% skill even on a tiny sample.

2. The Kalshi market broadly tracks the engine, but with real 10–31¢ gaps — biggest when a favorite (Carlsen) was collapsing. That hints at exploitable lag, with caveats.

3. The red-team's loudest warning: we may be building the wrong thing *first*. The business is incentive capture + speed; a fair-value model only earns its keep if it beats a free baseline by more than the bid-ask spread. (Result #1 shows the free baseline is bad — which *strengthens* the case for a model.)

Caveat on everything below: N = 24 games. Far too small. These results are suggestive, not conclusive. Treat as direction, not truth.

The honest framing (from the red-team)

The synthesis named the biggest risk bluntly: *"Building the wrong thing — investing all of v0 in a win-probability model when the goal is capturing KXCHESSMATCH liquidity incentives, where income is the rebate and the suspected edge is SPEED, not out-predicting the crowd."*

Its recommended alternative worth trying: ship raw Stockfish WDL as the anchor + tablebase/fortress overrides, and spend the effort on a market-reaction model + defensive quoting trained on the data we already have.

I took this seriously — which is exactly why tonight's first real analysis was the efficiency test (is there even an edge?) rather than just building the model. The twist: the data partly *undercuts* the "just use sf_wdl" alternative, because raw sf_wdl turned out miscalibrated. So the truth is nuanced: we likely need *both* a better fair value *and* the speed/microstructure work.

What I built tonight

engine_eval.py — Stockfish wrapper at ONE pinned config (fixed depth 12, multipv 2, 1 thread, WDL on) so training and live can't silently diverge (the #1 red-team fix). ~90ms/position.

pgn_features.py — PGN → per-ply position with correct clock attribution (the old latency parser swapped sides by ply parity — red-team caught it) + reconstructed wall-clock so engine evals align to market timestamps.

efficiency_test.py — aligns the engine's encounter win-prob to the captured Kalshi mids, per game.

train_winprob.py — builds the dataset, trains the binned "matrix," and runs the calibration backtest with the honest baselines.

Result 1 — Efficiency: does the market already track Stockfish?

Engine encounter-prob vs captured Kalshi mid, Norway Round 8 (June 2), real data:

Game	Result	Corr(engine, market)	mean gap	max gap
Firouzja–Gukesh	1–0	0.86	10¢	22¢
Carlsen–Pragg	0–1 (upset)	0.43	18¢	31¢
So–Keymer	½ (→Armageddon)	−0.78*	7¢	11¢

Read: the market is *not* perfectly efficient — gaps of 10–31¢ exist, largest when the favorite (Carlsen) was losing and the market was slow to accept it. That's where an engine edge would live.

Caveats: (a) the comparison used *raw* sf_wdl, which Result 2 shows is miscalibrated — so some "gap" is sf_wdl's error, not market inefficiency; (b) wall-clock is reconstructed from clocks (drifts over hours); (c) *the draw game's negative corr is my crude Armageddon combine (P=0.5), not a real signal.* A v2 efficiency test should use the calibrated model and proper Armageddon handling.

Result 2 — Win-probability calibration (the acceptance gate)

Leave-one-game-out CV (red-team: the game is the unit, not the position), 24 classical games, 791 positions, draws first-class. White-POV marginal: W 32% / D 53% / L 15% (sane: White scores ~58%, draws dominate).

model	log-loss	Brier	skill vs base rate
climatology (base rate)	1.089	0.656	0.000
raw Stockfish WDL	2.310	0.704	−1.121
binned human-calibrated	0.992	0.576	+0.089

The headline: raw Stockfish WDL is *worse than guessing the base rate* on elite humans — it's overconfident about decisive results that GMs draw. A simple human-calibrated cp→W/D/L map beats both. Gate passed (binned beats raw sf_wdl: 0.99 vs 2.31 log-loss). But again — 24 games.

The "matrix" (cp → [W, D, L] lookup tensor — your "just a matrix" idea)


 cp[-1000,-300): W=0.12  D=0.34  L=0.54
 cp[ -300,-150): W=0.07  D=0.68  L=0.25
 cp[ -150, -70): W=0.12  D=0.59  L=0.29
 cp[  -70, -25): W=0.19  D=0.61  L=0.20
 cp[  -25,  25): W=0.16  D=0.69  L=0.15   (dead-equal: 69% draw)
 cp[   25,  70): W=0.38  D=0.50  L=0.11
 cp[   70, 150): W=0.50  D=0.42  L=0.08
 cp[  150, 300): W=0.81  D=0.15  L=0.04
 cp[  300,1000): W=0.80  D=0.17  L=0.04

Trained offline, saved to winprob_matrix.json; live inference = one array lookup (microseconds), exactly the fast-matrix design you described. Note the small non-monotonic blip in the bottom two bins — that's the sparse-cell noise the red-team predicted at this sample size.

Every design choice (and the alternative)

Choice	What I did	Alternative / red-team view
Engine config	Fixed depth 12, multipv 2, pinned	Movetime is non-deterministic; deeper = more accurate but slower & may not match live budget. Train==live config is mandatory.
Eval → prob	Learn human mapping from outcomes	"Just use sf_wdl" — refuted tonight (miscalibrated). Could also use SF WDL as a base-margin offset and learn only the residual (more sample-efficient).
Model family	Binned lookup + climatology/sf_wdl baselines	Red-team: prefer monotone-constrained ordinal GBT + a Dirichlet calibration layer; logistic alone can't model eval×time×elo interactions.
Draws	First-class 3-way output	A win/loss-only model is simply wrong at 53% draw rates.
Validation	Leave-one-game-out, skill vs base rate	Red-team: add hold-out-by-player and walk-forward (Norway's 6-player field won't transfer to Sinquefield).
Clocks	python-chess `node.clock()`, per-side	Old parser swapped sides by parity — would've poisoned the time-trouble signal (the supposed edge).
Data	Norway classical R1–8, Armageddon excluded	Red-team: pool many 2700+ events across years; 24 games is nowhere near enough.
Opening	Skip first 12 plies	Red-team: add explicit out-of-book / novelty-ply gating; early evals are prep noise.

What the 7-agent red-team found (condensed)

Calibration lens: within-game autocorrelation means effective N = *games*, not positions; need game-level bootstrap CIs, skill scores, eval-stratified reliability.

Chess lens: missing tablebases (ground truth ≤7 men), opposite-bishop/fortress draws where eval lies, king-safety/initiative, opening-book awareness. *Don't ship without tablebase + fortress draw-bias.*

ML lens: no calibration layer; logistic has no interaction capacity; need monotonicity + ordinal structure.

Latency lens: train-deep/serve-shallow = silent decalibration; pin the engine; wire the latency probe's measured window into the depth choice.

Data lens: clock parser bug (fixed); Norway-6 roster won't transfer; need multi-event, multi-year volume + a FIDE-ID player map.

Adversary lens: maybe build the microstructure/speed side first; benchmark vs sf_wdl; test market efficiency on captured data (← did that tonight).

How this could be wrong (honest)

24 games. The single biggest threat. The bins are noisy; the +8.9% skill could shrink or vanish with proper data.

The efficiency "gaps" may be sf_wdl's miscalibration, broadcast/clock-reconstruction error, or the market correctly pricing Armageddon — not true edge.

Even a *correct* model loses if we're the slow side of the move feed (untested until live) or if incentive pools don't cover adverse selection.

Norway ≠ Sinquefield: different roster, scoring, time control. Tonight's model may not transfer.

Recommendations (in priority order)

1. Get real data volume — pool many 2700+ events (multiple years, with clocks). This is the gating dependency for trusting *anything* here.

2. Re-run the efficiency test with the calibrated model (not raw sf_wdl) + proper Armageddon handling — to cleanly separate "market lag" from "sf_wdl error."

3. Run the live latency probe on Round 9 (already deployed, waiting) to learn if we even win the speed race.

4. Build the incentive-capture P&L model — the business case the red-team says we keep deferring.

5. Only then: upgrade the model (ordinal monotone GBT + calibration layer, tablebases, fortress detection).

Bottom line: the approach isn't obviously wrong — in fact the data killed the "just use the engine" shortcut. But it's unproven on 24 games, and the red-team is right that speed + incentive economics deserve equal billing. Worth continuing, eyes open.