Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback

What's the real bottleneck in your search agent? Often it's the retriever, and you don't need to retrain your agent to fix it.

🗞️ Most existing agentic search approaches (like Search-R1) primarily optimize the reasoning agent while treating the retrieval model as a frozen black-box component. This design implicitly assumes that a sufficiently capable reasoning model can compensate for retrieval failures through improved query reformulation alone. We challenge this assumption by arguing that sub-optimal retrieval can be a bottleneck in agentic search performance. There has been some attempt (Agentic-R, CoSearch) to address this issue by jointly optimizing retrievers and reasoning agents. In practice, however, these methods are difficult to apply in settings where the reasoning model cannot be further trained, the retriever is externally provided, or gold-passage supervision is unavailable.

♦️ To address this, we propose Critic-R, a framework that closes the feedback loop between the reasoning agent and the retriever, at both inference and training time. Instead of blindly accepting whatever the retriever returns, Critic-R uses a separate critic model that reads the agent's introspective reasoning trace after it consumes the retrieved documents, and decides whether that evidence is actually sufficient to support the next reasoning step.

This verification signal powers two complementary mechanisms:
🔹 Critic-R-Zero (inference-time): when the critic finds the evidence insufficient, it rewrites the retrieval query and instruction based on reasoning agent's own introspective feedback and tries again, until the agent is satisfied or a refinement budget runs out. No gradient updates anywhere, the agent is untouched, and it works on top of any retriever, including those from Agentic-R or CoSearch.
🔹 Critic-Embed (training-time): to amortize the cost of refinement, we turn Critic-R-Zero's own trajectories into supervision. Documents that satisfy the agent become positives; documents rejected during failed refinement become hard intra-trajectory negatives. The retriever is fine-tuned with this signal, with no gold-passage annotations required.

Across HotpotQA, 2Wiki, MuSiQue, and Bamboogle:
✅ Critic-R-Zero has +12.4% relative improvement at inference time alone
✅ Critic-Embed gives +7.5% improvement when only the retriever is replaced, beating both off-the-shelf and co-trained retrievers

One interesting finding is that removing the agent's introspective feedback when collecting training data makes the retriever consistently worse. The agent's own sense of what's missing isn't a minor input to the critic, it's the primary supervisory signal Critic-Embed inherits.

Check out the paper for more details