New SOB Benchmark Exposes 17-26 Point LLM Value Gap
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- SOB (Structured Output Benchmark) evaluates LLM structured output across text, image, and audio modalities using seven metrics, with hard schemas (arrays of objects, depth ≥3) dominating 61% of text, 88% of image, and 98% of audio records; the final leaderboard is weighted by schema complexity (1.0/2.0/3.0 for easy/medium/hard).
- Gemini-3.1-Pro tops Value Accuracy at 82.0%, with three open-weight models — GLM-5.1 (80.6%), GLM-4.7 (80.4%), and Qwen3.5-35B (80.1%) — trailing inside a 2-point band and beating several closed-source frontier releases on the metric the source flags as what production systems actually care about.
- Every model tested clears 95%+ on JSON Pass but drops 17 to 26 points on Value Accuracy; GLM-5.1 has the tightest gap (16.9 pp) and the highest Value Accuracy, while Schematron-8B lands the lowest Value Accuracy at 73.1% — a 25.6-point fall from its 98.7% JSON pass rate.
- GPT-5.4 and Gemini-3.1-Pro are the two most expensive models tested at $15.00 and $12.00 per 1M input/output tokens; running Qwen3.5-35B, GLM-4.7, or Gemini-2.5-Flash costs 6-11x less per correct field while losing at most ~2 percentage points of Value Accuracy.
- Modality rankings diverge sharply per model — GPT-5.4 ranks 5th on text but 13th on images, Schematron-8B ranks 26th on text but 15th on images, and Gemma-4-31B ranks 18th on text but 1st on images, with audio (~7,300-token transcripts with overlapping speakers) the hardest modality overall.
- The benchmark protocol ran all models at temperature 0.0 with a 2,048-token output cap and reasoning/thinking disabled where providers allowed it; GPT-5/5-Mini ran in 'minimal' reasoning and GPT-5.5 in 'none,' giving those locked models a small reasoning advantage that several non-reasoning models still beat on Value Accuracy.
Why it matters: For engineering teams wiring LLMs into data pipelines, SOB's headline finding inverts the cost calculus: the most expensive frontier models lead Value Accuracy by roughly 2 points, but open-weight models like Qwen3.5-35B and GLM-4.7 cost 6-11x less per correct field on the same API tier. The 17-26 point gap between schema validity and value grounding means existing JSON-compliance benchmarks have been telling teams their extraction pipelines are reliable when they may silently hallucinate one in every five leaf values.
Ask SkimNews
