Microsoft research shows AI coding agents trust SemVer labels over code diffs

A Microsoft study found that Claude Sonnet 5.5 and other AI coding agents skip checking dependency updates when the change is labeled a patch, even though the underlying code is a breaking change.

Microsoft published research on November 9, 2026 showing that AI coding agents' decisions about whether to verify a dependency upgrade are swayed by the SemVer label attached to it rather than by the actual code change. The study, written by Principal Developer Advocate Waldek Mastykarz, is detailed in Do AI coding agents care about SemVer?.

Key findings from the study

The first experiment ran about 1,000 runs across four models, Claude Sonnet 5.5, Claude Opus 5.5, GPT-6.1 Sol and GPT-6.1 Terra, testing identical breaking-change upgrades labeled with different SemVer versions.
Claude Sonnet 5.5 checked the code in only 2 of 10 runs when the update was labeled a patch release, versus 10 of 10 runs when the identical change was labeled a major release.
A second experiment covered 540 runs across six models; Claude Haiku 5.5 merged patch-labeled breaking updates without checking the code 37% of the time, versus 7% for major-labeled updates.
Across the full study, non-breaking updates were merged correctly in 268 of 270 runs, while breaking updates were handled correctly in only 13 of 270 runs.
Four of those 13 correct-looking merges still left the main branch broken, two by Claude Haiku 5.5 and two by GPT-6 Luna, all mislabeled as patch or minor releases.

What it means for people who build with AI

The findings give library maintainers and teams running agentic dependency-update workflows concrete evidence that today's coding agents can be swayed by a version label rather than by the substance of a change. For teams that let agents auto-merge dependency bumps, the research points to designing CI gates and auto-merge policies that force a code-diff check regardless of how an upstream package labels its release, since the label itself proved an unreliable signal across every model tested.

Sources

Do AI coding agents care about SemVer? (Microsoft, November 9, 2026, official)

Written with AI assistance, strictly from the sources listed above. Before it went live, automated checks verified its dates, sources and every language edition.