Cognition announced the release of FrontierCode 1.1, a private benchmark designed to test whether AI coding agents can produce changes that open‑source maintainers would actually merge.
According to Cognition’s official X post, FrontierCode 1.1 is the first benchmark to measure end‑to‑end code quality and mergeability, asking “would the maintainer actually merge this PR?”.
Community member @ArtificialAnlys, who runs a leaderboard for the benchmark, said the dataset was built with open‑source maintainers who spent more than 40 hours per task. Each of the 150 tasks requires an agent to modify a real repository, after which both unit tests and qualitative checks evaluate whether the change meets maintainer expectations.
The primary set, FrontierCode 1.1 Main, contains 100 tasks and is run using the Stirrup open‑source reference harness to enable apples‑to‑apples comparisons across models. All tasks remain private to avoid contamination. Scoring uses blocking and non‑blocking criteria: any failure on a blocking check yields a zero score, while passing all blocking checks results in a score between 70 % and 100 %.
The benchmark also assesses behaviors such as excessive changes, large diffs, modifications outside the relevant scope, and the addition of unhelpful tests, reflecting a more holistic view of success beyond merely passing unit tests.
Sources
Fan coverage from Devin Central — not an official Cognition announcement. Devin is a trademark of Cognition.