Junxiao Liu
Publications
RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
R3S: Refining and Recovering Reinforcement Signals for Multilingual Understanding and Reasoning
Large reasoning models often default to English reasoning when processing non-English questions, yet their performance drops substantially when reasoning in the question language. Even with the same reasoning language, semantically equivalent English and non-English questions still exhibit a clear performance gap. Together, these phenomena reveal two distinct bottlenecks: target-language question understanding and target-language reasoning. Existing methods typically optimize only one of these capabilities. However, simply combining them may not be sufficient to optimize both effectively, as answer correctness alone cannot distinguish failures in question understanding from those in reasoning. We propose R3S, a reinforcement learning framework that disentangles the optimization of the two capabilities. R3S refines translation rewards derived from downstream reasoning accuracy through English-solvability filtering and recovers target-language RLVR signals using self-generated English hints. Together, these designs require neither external model feedback nor external multilingual training data. Experiments across three backbone models and five languages show that R3S improves language-consistent accuracy over the target-language RLVR baseline on MMATH by an average of 10.3 percentage points, while maintaining near-perfect language consistency. Consistent gains on MMLU-ProX further demonstrate its generalization beyond math problems.