DeepSWE is a new benchmark for coding agents that measures their ability to handle real software engineering work across 91 active open-source repositories, using short, realistic prompts.

Programmers are now using AI agents to write code, often deploying teams of sub-agents for tasks like writing, testing, and error correction.

Scale AI benchmarked top models (Claude, Gemini, OpenAI) on real multi-file software engineering tasks using actual production-grade code, finding they solved only 20-30% of tasks.