Ai2 releases XYBench, a benchmark for AI assistants solving the wrong problem

Allen Institute for AI published XYBench, over 8,000 questions testing whether AI assistants solve users' stated requests or their real underlying goals.

The Allen Institute for AI (Ai2) released XYBench on November 13, 2026, a benchmark of more than 8,000 real-world questions designed to measure whether AI assistants solve a user's literally stated request or their actual underlying goal, the classic XY problem. Details appeared in XYBench: Helping with the problem behind the question.

Inside XYBench

The dataset includes 8,115 questions: 1,557 from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 from WikiHow.
XYBench tests whether a model solves the user's stated request (Y) or their actual underlying goal (X), the classic XY problem.
The evaluated models are Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro.
Across these models, the stated request is addressed correctly 75 to 86 percent of the time.
An unprompted solution addressing the real underlying goal appears only 35 to 41 percent of the time.
In multiple-choice format, the same models pick the goal-focused answer 71 to 84 percent of the time.

What it means for people building with AI

The gap between stated-request accuracy and unprompted goal-focused accuracy quantifies a known failure mode in coding assistants and agents: answering the literal question instead of the real problem behind it. Because the same models pick the goal-focused answer far more often in multiple-choice form than they produce it unprompted, the results point to a recognition-versus-production gap rather than a lack of knowledge. That distinction matters directly for anyone building or evaluating developer-facing AI tools, where a technically correct answer to the wrong question can still leave a user stuck.

Built by a multi-institution team

XYBench was built by the Allen Institute for AI together with Carnegie Mellon University, NVIDIA, and Johns Hopkins University. The paper describing the benchmark won the Best Resource Paper Award at EMNLP 2026.

Sources

XYBench: Helping with the problem behind the question (Allen Institute for AI / Hugging Face, November 13, 2026, official)

Written with AI assistance, strictly from the sources listed above. Before it went live, automated checks verified its dates, sources and every language edition.