32 document-extraction pipelines — OCR engines paired with LLMs, and vision-language models going direct — enter a single-elimination bracket. Pick your winners, then watch each matchup's real F1 score reveal, calibrated to match the actual distribution from a benchmark of 35 open-model pipelines on a real high-stakes bureaucratic extraction task (EU AI Act "high risk" category).
Here's what the real benchmark found, and how this bracket's 32 seeds stack up against it.
Only 4 of the 35 real pipeline configurations tested in the paper crossed F1 > 0.5. Most were open VLMs used directly; one OCR+LLM combo matched them.
Roughly three-quarters of all 35 configurations scored F1 < 0.25 — closer to unusable than "close enough." Scale this bracket recreates the same ratio: 24 of 32 seeds.
Vision-language models reading the page directly generally beat OCR-then-LLM pipelines — but the best OCR+LLM combo matched the top VLM, so the OCR step isn't doomed, it's just usually the weak link.
Bigger models helped, but not proportionally — some huge models underperformed smaller ones. Input quality (how well OCR preserves document structure) mattered as much as model capability.
Plain English: throwing a strong LLM at messy OCR text mostly doesn't save you — garbage structure in, garbage extraction out. Vision-language models that skip OCR entirely tend to do better, but even the best open models struggled zero-shot on this real, high-stakes government-style form extraction task. Only a handful of the 35 tested pipelines would be safe to actually deploy. Read the full paper: arXiv:2608.18289.