Q1 2026 LLM Evaluation Study

Three assumptions drive most current decisions about applying LLMs in production: that evaluating LLM outputs requires an LLM judge, that pro-tier models earn their premium, and that a single frontier model can dominate the field. We tested all three across 30 evaluations and more than 20,000 individual tests in Q1 2026. The data falsified all three.

Of the 30 evaluations, exactly one used LLM-as-judge as the primary scoring method. The other 29 used deterministic or near-deterministic methods — structural validation, golden-set comparison, embedding similarity, fact and keyword coverage, and behavioral checks. Cheap, reproducible methods carry 73% of the primary scoring in this suite.

Across the suite, lite-tier and default-tier models won the majority of tasks outright. Gemini Flash Lite tied for first on action extraction (97.6% across 4,140 tests) and won entity extraction (F1 0.811). Within-provider tier inversions occurred on multiple evals: Gemini Pro scored 16 percentage points worse than Gemini Flash Lite on relevance checking; Claude Haiku beat Claude Opus by 33% on knowledge-graph utilization. Five different models won top rank on different evals; no model placed in the top three on every eval.

This paper documents the methodology lexicon, the per-model specialty profiles, the counter-intuitive results, the prompt × model interaction effects, and the recommendations for production teams designing AI systems in 2026 and beyond.

@techreport{hood2026q1llm,
  author = {Hood, David},
  title  = {Q1 2026 LLM Evaluation Study: Methodology and Findings},
  institution = {42 Robots AI},
  year   = {2026},
  url    = {https://42robots.ai/papers/q1-2026-llm-evaluation/}
}