# benchmarks

Moritz Hardt says the AI benchmarking ecosystem is broken but potentially repairable.
Benchmarks can define research priorities as well as measure technical performance.
Better evaluation should clarify what a score measures and where it fails.

DT Editorial Team·Oct 17, 2026·via quantamagazine.org

Authors Reply to Benchmark Limitations in Clinical AI Comparison

Nature Medicine has published a reply to a comment questioning the use of limited benchmarks in comparing general-purpose and clinical AI.
The reply argues that selected benchmarks still reveal meaningful differences between model types.
The exchange highlights the importance of context-specific evaluation and transparent reporting in clinical AI.

DT Editorial Team·Sep 8, 2026·via nature.com

Benchmark split over GPT-6 Astra highlights a messier AI race

Epoch AI ranks GPT-6 Astra first overall, while Artificial Analysis rates it level with its predecessor.
The model’s strongest reported signal is efficiency, including lower compute use and standout ARC-AGI-3 performance.
Astra also shows mixed results, with reported regressions on some long-context, banking, and coding-related measures.

DT Editorial Team·Sep 4, 2026·via the-decoder.com

Google pilots double-blind AI benchmark testing for Gemini

Google Deepmind says it is launching a double-blind evaluation of a proprietary frontier AI model.
The pilot uses a Gemini Flash Lite model and confidential benchmarks with external partners including the Singapore AI Safety Institute.
Google says cryptographic protections keep test prompts private from the model provider and model weights private from evaluators.

DT Editorial Team·Aug 28, 2026·via the-decoder.com

Frontier Radar #4: China has caught up, so what's left of the Western AI lead?

Chinese AI models narrow the gap with top US systems

The report says Chinese models now score near the top on broad evaluations, not just narrow tests.
Named systems include Kimi K3, Qwen3.8-Max, and GLM-5.3.
The analysis argues that temporary model-performance leads are becoming hard to defend commercially.
Open and lower-cost Chinese models are increasing pressure on Western AI business models.

DT Editorial Team·Aug 20, 2026·via the-decoder.com

Grok Gets Cursor-Driven Upgrade, Claims to Be Competitive With Top Models

Grok 4.6 Launches with Cursor Integration, Claims Frontier-Level Performance

Grok 4.6 released, claims frontier intelligence in coding and knowledge work benchmarks.
Matches OpenAI's GPT-5.6 Sol on Artificial Analysis Intelligence Index.
Partnership with Cursor, including possible acquisition, boosted Grok's performance.
Grok 4.6 first available in Cursor, with Grok Build and Grok Bot.

DT Editorial Team·Aug 13, 2026·via gizmodo.com

Microsoft's MAI Code 1.1 Flash Lags Behind DeepSeek on Price and Performance

Microsoft released MAI Code 1.1 Flash for GitHub Copilot, claiming 25% token efficiency and lower cost.
Benchmarks show it trails DeepSeek V4 Flash on Terminal Bench 2.1 (62.9% vs 82.7%).
Pricing is higher than DeepSeek: $0.20 input vs $0.14, $1.20 output vs $0.28.
Microsoft's proprietary strategy conflicts with its open AI advocacy.

DT Editorial Team·Aug 12, 2026·via the-decoder.com

METR introduces a new metric to calculate exactly when AI agents become more expensive than humans

METR’s new cost metric tests when AI agents stop being a bargain

METR has introduced the “expenditure horizon” to compare the cost of AI agents and human labor on the same task.
The framework combines labor, inference, and experimental compute into a single economic measure.
In a NanoGPT speedrun case study, early results suggested human contributors can still be more cost-effective on harder improvements.
The metric is designed to test practical value, not just whether an AI system can complete a benchmark.

DT Editorial Team·Jul 27, 2026·via the-decoder.com

CEO-Bench Tests Whether AI Can Run a Startup for 500 Days

CEO-Bench measures whether AI agents can manage a startup over 500 simulated days.
Most models went bankrupt or ended below their starting cash balance.
A simple rule-based heuristic outperformed nearly all of the AI systems tested.

DT Editorial Team·Jun 28, 2026·via the-decoder.com

Anthropic releases a version of its vaunted Mythos model to developers

Anthropic Releases Claude Fable 5: A Guardrailed Version of Its Mythos Model for Developers

Anthropic released Claude Fable 5, a guardrailed version of its Mythos AI model.
Fable 5 is the most capable public model from Anthropic, excelling in coding, analytics, and vision.
It scored 80.3% on SWE-Bench-Pro, far ahead of GPT-5.5 (58.6%) and Gemini 3.1 Pro (54.2%).
The model refuses cybersecurity and biology queries, routing them to a less capable model.

DT Editorial Team·Jun 10, 2026·via fastcompany.com

Benchmark Finds AI Systems Often Answer Correctly but Cite the Wrong Evidence

CiteVQA measures both answer correctness and citation correctness in long documents.
A correct answer with a wrong citation receives no credit under the benchmark’s strict metric.
Gemini-3.1-Pro-Preview led the test with 76, while GPT-5.4 dropped sharply when citation accuracy was required.
Researchers say weak attribution makes many systems risky for regulated domains.

DT Editorial Team·May 25, 2026·via the-decoder.com

Mathematicians Build a Tougher AI Test by Including Problems With No Valid Answer

SOOHAK contains 439 original math tasks, including 99 deliberately unsolvable ones.
Gemini 3 Pro led the challenge set at 30%, with GPT-5 variants at 26%.
Even the best models scored below 50% at recognizing unsolvable problems.

DT Editorial Team·May 18, 2026·via the-decoder.com

A new exploit benchmark shows frontier AI models edging deeper into offensive security

Carnegie Mellon researchers built a benchmark that measures progress from vulnerability work to full code execution in V8.
Claude Mythos Preview outperformed GPT-5.5 in both assisted and fully autonomous modes.
Mythos' stronger performance came with a much higher reported test cost, complicating the competitive picture.

DT Editorial Team·May 16, 2026·via the-decoder.com

New Benchmark Shows Why Better-Looking AI Video Still Fails at Basic World Logic

WorldReasonBench evaluates whether AI video models continue scenes in physically and logically plausible ways.
Commercial systems outperformed open-source models on the benchmark’s core reasoning metric.
Researchers found that even visually strong videos often fail simple tests involving motion, interaction, or cause and effect.

DT Editorial Team·May 16, 2026·via the-decoder.com

Agent skills look great in benchmarks but fall apart under realistic conditions, researchers find

AI agent “skills” show limited gains once testing looks more like the real world

Researchers tested 34,198 real-world skills from open-source repositories.
The study argues existing benchmarks overstate gains by handing agents highly task-specific instructions.
In more realistic conditions, skill-driven improvements shrink sharply and can even hurt weaker models.

DT Editorial Team·Apr 12, 2026·via the-decoder.com

AI Models Still Prefer Guessing Over Asking for Help, Benchmark Finds

ProactiveBench tests whether multimodal AI models ask for missing visual information.
Across 22 models, average performance fell from 79.8 percent in standard settings to 17.5 percent on the proactive benchmark.
The findings suggest current systems are better at guessing than recognizing when they need help.

DT Editorial Team·Apr 12, 2026·via the-decoder.com

Google AI Overviews Clears 90% Accuracy in Benchmark, but Scale Turns Errors Into a Major Problem

A benchmark found Google AI Overviews answered factual questions correctly about 90% of the time.
The reported result improved from an earlier score around 85% after a model update.
Google challenged the benchmark’s relevance, arguing it does not reflect typical search behavior.

DT Editorial Team·Apr 7, 2026·via arstechnica.com

MacBook Neo rivalis cloud servers in database workload test - 9to5Mac

News

MacBook Neo Matches Cloud Servers in Database Tests

DuckDB benchmarks show the 512GB MacBook Neo matching mid-tier cloud server performance on analytical database workloads
Apple's unified memory architecture eliminates hypervisor and network-storage overhead that clouds impose
One-time hardware cost competes economically with annual cloud billing for offline batch workloads

DT Editorial Team·Mar 18, 2026·4 min read·via 9to5mac.com

iPhone 17e A19 Chip Shows Strong Benchmark Gains

A19 chip shows strong single-core and multi-core gains over A17
Performance positions iPhone 17e competitively against premium Android devices
A19 supports Apple Intelligence features at a more affordable price point

DT Editorial Team·Mar 7, 2026·3 min read·via 9to5mac.com

Anthropic Releases Claude Sonnet 4.6 with Record Benchmarks and Million-Token Context