TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
Abstract
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
Community
TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning
š” Overview
We propose TurnSight, a turn-level hindsight self-distillation framework designed for Tool-Integrated Reasoning (TIR). The key idea is to derive fine-grained, state-aligned supervision directly from the tool execution outcomes encountered by the agent itself. By aggregating token-level hindsight evidence into coherent interaction-level signals, TurnSight enables reinforcement learning to distinguish between useful, redundant, and harmful tool-use decisions across long-horizon trajectories.
š„ Key Insights
- We identify two key requirements for credit assignment in multi-turn TIR: supervision should be aligned with on-policy execution states and coherent at the interaction-turn level.
- We propose TurnSight, which derives execution-conditioned, multi-horizon hindsight assessments from on-policy tool execution outcomes and uses them to modulate RL advantages without changing their optimization direction.
- Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate the effectiveness, robustness, and generalization ability of TurnSight across diverse multi-turn TIR scenarios.
š§āØ All the code, datasets and model checkpoints of TurnSight are fully open-sourced:
Github: https://github.com/quchangle1/TurnSight
Datasets & Models: https://huggingface.co/collections/ChangleQu/turnsight
Get this paper in your agent:
hf papers read 2608.04007 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper