#benchmarks

All articles tagged with "benchmarks"

Anthropic releases a version of its vaunted Mythos model to developers

AI & Robotics

Anthropic Releases Claude Fable 5: A Guardrailed Version of Its Mythos Model for Developers

Key Takeaways

Anthropic released Claude Fable 5, a guardrailed version of its Mythos AI model.
Fable 5 is the most capable public model from Anthropic, excelling in coding, analytics, and vision.
It scored 80.3% on SWE-Bench-Pro, far ahead of GPT-5.5 (58.6%) and Gemini 3.1 Pro (54.2%).
The model refuses cybersecurity and biology queries, routing them to a less capable model.

DT Editorial Team·Jun 10, 2026·via fastcompany.com

More in AI & Robotics→

AI & Robotics

Benchmark Finds AI Systems Often Answer Correctly but Cite the Wrong Evidence

Key Takeaways

CiteVQA measures both answer correctness and citation correctness in long documents.
A correct answer with a wrong citation receives no credit under the benchmark’s strict metric.
Gemini-3.1-Pro-Preview led the test with 76, while GPT-5.4 dropped sharply when citation accuracy was required.
Researchers say weak attribution makes many systems risky for regulated domains.

DT Editorial Team·May 25, 2026·via the-decoder.com

More in AI & Robotics→

AI & Robotics

Mathematicians Build a Tougher AI Test by Including Problems With No Valid Answer

Key Takeaways

SOOHAK contains 439 original math tasks, including 99 deliberately unsolvable ones.
Gemini 3 Pro led the challenge set at 30%, with GPT-5 variants at 26%.
Even the best models scored below 50% at recognizing unsolvable problems.

DT Editorial Team·May 18, 2026·via the-decoder.com

More in AI & Robotics→

AI & Robotics

A new exploit benchmark shows frontier AI models edging deeper into offensive security

Key Takeaways

Carnegie Mellon researchers built a benchmark that measures progress from vulnerability work to full code execution in V8.
Claude Mythos Preview outperformed GPT-5.5 in both assisted and fully autonomous modes.
Mythos' stronger performance came with a much higher reported test cost, complicating the competitive picture.

DT Editorial Team·May 16, 2026·via the-decoder.com

More in AI & Robotics→

AI & Robotics

New Benchmark Shows Why Better-Looking AI Video Still Fails at Basic World Logic

Key Takeaways

WorldReasonBench evaluates whether AI video models continue scenes in physically and logically plausible ways.
Commercial systems outperformed open-source models on the benchmark’s core reasoning metric.
Researchers found that even visually strong videos often fail simple tests involving motion, interaction, or cause and effect.

DT Editorial Team·May 16, 2026·via the-decoder.com

AI & Robotics

DT Editorial Team·Apr 12, 2026·via the-decoder.com

AI & Robotics

DT Editorial Team·Apr 12, 2026·via the-decoder.com

AI & Robotics

DT Editorial Team·Apr 7, 2026·via arstechnica.com

News

DT Editorial Team·Mar 18, 2026·4 min read·via 9to5mac.com

News

DT Editorial Team·Mar 7, 2026·3 min read·via 9to5mac.com

News

DT Editorial Team·Feb 17, 2026·5 min read·via techcrunch.com

#benchmarks

Anthropic Releases Claude Fable 5: A Guardrailed Version of Its Mythos Model for Developers

Benchmark Finds AI Systems Often Answer Correctly but Cite the Wrong Evidence

Mathematicians Build a Tougher AI Test by Including Problems With No Valid Answer

A new exploit benchmark shows frontier AI models edging deeper into offensive security

New Benchmark Shows Why Better-Looking AI Video Still Fails at Basic World Logic

Anthropic Releases Claude Fable 5: A Guardrailed Version of Its Mythos Model for Developers

Benchmark Finds AI Systems Often Answer Correctly but Cite the Wrong Evidence

Mathematicians Build a Tougher AI Test by Including Problems With No Valid Answer

A new exploit benchmark shows frontier AI models edging deeper into offensive security

New Benchmark Shows Why Better-Looking AI Video Still Fails at Basic World Logic

AI agent “skills” show limited gains once testing looks more like the real world

AI Models Still Prefer Guessing Over Asking for Help, Benchmark Finds

Google AI Overviews Clears 90% Accuracy in Benchmark, but Scale Turns Errors Into a Major Problem

MacBook Neo Matches Cloud Servers in Database Tests

iPhone 17e A19 Chip Shows Strong Benchmark Gains

Anthropic Releases Claude Sonnet 4.6 with Record Benchmarks and Million-Token Context