According to ChainCatcher, Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim have released a preprint paper, XYEval: Agents say yes to bad advice, introducing a meta-evaluation framework called XYEval for testing how agents respond to misleading user advice.
Extending the XY problem to agent settings
The paper extends the “XY problem” to agent scenarios. In this setup, the user asks about solution X — the approach they have been trying — rather than the real problem Y.
XYEval keeps the original task environment and the verifiable answer intact, and only appends user advice in the instruction that appears reasonable but is misleading. The paper says the misleading advice is generated after the generator sees the full task, and it is added as a confident suggestion rather than a hard requirement, so the original task remains solvable.
Six benchmarks and five models
The evaluation covers six benchmarks — τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity’s Last Exam, and MCP-Atlas — and five models. In τ2-bench, the misleading advice was generated using rules written by human experts.
Paper reports a relative drop of as much as 46.7%
According to the paper, performance fell sharply after tasks were turned into XY variants, with the largest relative decline reaching 46.7%.
On τ2-bench, performance dropped more when users said they would only accept a better solution after first receiving a detailed explanation. The paper also says that simple system instructions encouraging models to recognize the XY problem can only partially reduce the issue. Related code has been released in the GitHub repository google-deepmind/xyeval.