DeepStress: Stress-Testing Deep Search Agents
Abstract
A dynamic stress-testing framework shows that search agents often answer confidently despite unreliable retrieved evidence and may rely on memorized knowledge when documents are factually wrong.
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. Such evidence is rare in benchmark corpora, and resulting failures are hard to reproduce. We propose DeepStress, a dynamic stress-testing framework that subjects agents to degraded evidence at controlled rates. It replaces the retrieval module with a synthetic environment where documents are generated dynamically for each query, under independent control of their trustworthiness, relevance, and factuality. To characterize behavior under stress, we go beyond binary accuracy with a five-way classification of final-answer outcomes and two new reliability-aware metrics, RAS and CoPRAS, which credit cautious behavior and account for computational cost. Evaluating 12 systems on questions from HotpotQA and BrowseComp-Plus, we find that several open-weight agents answer confidently when sources contain obvious credibility red flags, and that many HotpotQA questions are answered from parametric knowledge even when all returned documents are factually wrong.
Community
Hi everyone! This is the first paper of my PhD, which I presented at the Workshop on Agent Behavior earlier this year at COLM! In this paper, we look at how we can use a simulated environment to test our search agents in "difficult" situations and learn more about how they behave in such cases 😃
The GitHub repo should be up soon(ish), feel free to reach out
Degradation curves are the easy part. What I need to know is whether the failure stays traceable — agent cites the poisoned doc and I can grep the log, or it quietly folds the wrong number into a confident answer and I hear about it from a customer.
In prod, retrieval is rarely off-topic. It's on-topic and stale: an old API doc that still ranks first because everyone linked to it in 2022, a wiki page two major versions back. Worth splitting "correct-looking but out of date" from "irrelevant" as separate stress axes, since the fixes diverge — relevance failures get better reranking, staleness needs date metadata and a model that actually checks it.
If the sweep only reports accuracy at each degradation rate, the number I can't read off it is how often a wrong answer arrives wearing a plausible citation. That's the one that costs me a weekend.
We chose the keep the different dimensions pretty simple and coarse to make the results very clear & easy to interpret while focusing on the idea of "Stress-Testing". However, it would indeed be super interesting to have more realistic and finegrained dimensions to play with and it's definitly something we'll look into!
Here are some recent papers I saw that could introduce more realistic noise in web/search agents: AgentHeal (https://arxiv.org/abs/2610.39234v1), AgentNoiseBench (https://arxiv.org/abs/2602.11348) and DRNoise (https://arxiv.org/abs/2607.17291)
Thanks for your comment!
Controlling the trustworthiness, relevance and factuality of evidence separately is a smart way to stress an agent, and crediting cautious behavior in the metric makes me happy. Do agents get more cautious when the evidence is bad, or just more confidently wrong?
Depends on the agent and the question asked!
If you look at the overall results (Figure 1), you'll see that some agents (Search-R1 for example) are indeed confidently wrong (they get fooled by the bad evidence), however, some other agents (those based on GPT models for instance) are indeed more caution (they abstain more).
But yeah, the question asked is also super important. If the question is related to relatively known entities (such as questions from HotpotQA, which are based on Wikipedia pages), then you wont see much abstention at all. However, for more niche questions (such as those from BrowseComp-Plus) then you start seeing some agents abstaining a lot more.
I think agents using smaller models & trained against a correctness reward will answer confidently wrong more often. We're currently doing more experiments to prove that!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge (2026)
- Parameters vs. Context: TRACE Fine-Tuning for Robust Retrieval-Augmented Generation (2026)
- CORE: Coarse-to-Fine Search with Context Management for Deep Research (2026)
- Reasoning-Search-Augmented LLMs: An Operational Taxonomy, Benchmark Analysis and Controlled Evidence Synthesis (2026)
- Bridging Static and Agentic RAG for Taiwanese Historical Question Answering (2026)
- Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents (2026)
- QREF: Improving DeepSearch Agents by Aligning Search with Reasoning via Query Refinement and Evidence Filtering (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2607.13920 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper