Scale AI benchmarked top models (Claude, Gemini, OpenAI) on real multi-file software engineering tasks using actual production-grade code, finding they solved only 20-30% of tasks.

Anthropic's Claude model was the first agentic AI system capable of productive work like software coding.

On SWEBench Pro, Claude Mythos achieved a 78% score, while Opus previously scored 53% and GPT 5.4 scored 57.7%.

DeepSWE is a new benchmark for coding agents that measures their ability to handle real software engineering work across 91 active open-source repositories, using short, realistic prompts.