The AA Omniscience benchmark of AI knowledge and hallucination was created by Artificial Analysis, an independent, venture-backed benchmarking company based in San Francisco. Originally launched in November 2025, this comprehensive benchmark covers 6,000 questions across six major domains (business, law, health, software, science and the humanities). Artificial Analysis published its quarterly update on October 1st 2026. Artificial Analysis also has a broader evaluation suite that evaluates models for their intelligence, speed and cost per task. It covers text-to-image models as well as traditional text LLMs; the top-scoring ones in this category were GPT Image 1.5 and Nano Banana 2. A total of 198 AI models are currently evaluated. The benchmark specifically evaluates hallucination rates, checking whether a model guesses when it lacks the information it needs to answer accurately. The benchmark distinguishes accuracy (answering questions correctly) from hallucination rate. The benchmark rewards models that abstain rather than guess.
Back when it launched, the most accurate models were Grok 4 (39% accuracy), along with GPT-5 (39%), Gemini 2.5 Pro (37%) and Claude Opus 4.1 (36%). However, the models with the lowest hallucination rates were very different. Among the models covered in that quarterly analysis, the best (lowest hallucination rate) was Claude Haiku 4.5 (26%), ahead of Claude Sonnet 4.5 (48%) and Claude Opus 4.1 (48%). By comparison, GPT-5.1 had a hallucination rate of 51%, Grok 4 scored 64% and DeepSeek v3.1 Terminus scored 74%. The models vary by domain, with Claude doing better on software engineering and law, GPT on business, and Grok on health and science. Larger models score higher on accuracy, but not always on reliability.
Moving on to October 2026, on the overall index, Claude Opus 5.5 and Claude Sonnet 5.5 are now the leaders, slightly ahead of GPT-6 Astra, Gemini 4 Argon and GPT-6.1 Sol. Claude Sonnet 5.5 is only marginally behind Claude Opus 5.5, but Sonnet uses a lot more tokens per task, seven times as many as GPT-6 Astra. It was interesting to see that the latest Gemini model, Gemini 4 Argon, which was released at the end of September, is highly competitive, level with GPT-6 Astra. Also, Gemini 4 Argon had a low hallucination rate on questions that it does not answer correctly (just 15%, compared with 51% for GPT-6 Astra). It achieves this by being much more willing to admit that it does not know the answer, even though it has lower factual accuracy than GPT-6 Astra.
The Artificial Analysis benchmark now extends to AI agents, testing tasks such as producing spreadsheets, memos and presentations. Agent benchmarks here include automating workflows (AutomationBench), knowledge work (AA-Briefcase) and autonomous computer tasks (Terminal-Bench). On these, Gemini 4 Argon came top of the AutomationBench table, while Claude Sonnet 5.5 was top of the leaderboard on AA-Briefcase and Terminal-Bench.
Artificial Analysis has an unusually broad set of benchmarks. There are many rivals, including Humanity’s Last Exam, ARC-AGI, Stanford HELM, METR, LMArena (formerly Chatbot Arena) and LiveBench. Although there are differences, the various benchmarks tend to be quite well correlated. LiveBench, for example, had a correlation of 0.91 with Chatbot Arena, while another comparison found a correlation of 0.93 between Artificial Analysis and an aggregated LLM ranking. However, a 2026 study found that agreement between benchmarks was far from perfect and could be sensitive to evaluation methodology.
There is no perfect way to evaluate LLMs, but these various organisations perform a valuable role by producing snapshots of model performance against a range of criteria. While individual models may score a little higher or lower in any particular study, there are clear trends to be seen. Perhaps the broader lesson is that there is increasingly no single “best” AI model. Accuracy, hallucination rate, agentic capability, speed and cost are distinct dimensions, and the model that leads on one may be rather ordinary on another.
The latest Artificial Analysis results from October 2026 have some interesting implications, including the re-emergence of Gemini as a high performer. What is also interesting is that, even with the latest models, LLMs still get around one in four answers wrong (at best). Claude Opus 5.5 and GPT-6 Astra both got 23% of answers wrong while, for example, Qwen 3.8 Max was wrong 38% of the time. With all the breathless news about the latest LLM capabilities, just bear in mind that LLMs still answer factual questions wrongly on a regular basis.
