Xiangchen Wang
Publications
From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to follow a route-level instruction by executing its constituent steps from egocentric visual observations. Existing VLM-based navigators typically supervise both capabilities through next-action prediction alone, making progress-tracking errors difficult to distinguish from execution errors. When an agent deviates from the route, a corrective action label may recover the next movement but does not indicate whether the agent selected the wrong sub-instruction or failed to execute the correct one. Consequently, the agent may continue making decisions from an erroneous progress state. To resolve this ambiguity, we propose \textbf{Route2Step}, a framework that decouples semantic progress tracking from action generation through an explicit step-level interface. The Instruction Analysis Module ($\mathcal{M}_{\mathrm{IA}}$) predicts this state from the global instruction and visual history. Conditioned on the predicted state and recent observations, the Action Generation Module ($\mathcal{M}_{\mathrm{AG}}$) generates local action chunks. To supervise the progress state without manual temporal labels, E-SPA, a step-alignment procedure, associates sub-instructions with their corresponding portions of route-level demonstrations. These alignments enable state supervision for incorrect progress estimates, while direct action supervision is reserved for rollout groups that repeatedly fail under the correct active sub-instruction. On R2R-CE, Route2Step improves SR from 48.1\% to 55.3\% and SPL from 43.3\% to 48.2\%, using 190K state-level corrective samples while requiring only 11.5K directly action-supervised states. Experiments in real-world indoor and outdoor environments further demonstrate the practical applicability of Route2Step. The project page is: https://sisyphus-hxy.github.io/Route2Step/.
\textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation
Vision-and-Language Navigation (VLN) requires agents to interpret natural language instructions and act coherently in visually rich environments. However, most existing methods rely on reactive state-action mappings without explicitly modeling how actions causally transform subsequent visual observations. Lacking such vision-action causality, agents cannot anticipate the visual changes induced by its own actions, leading to unstable behaviors, weak generalization, and cumulative error along trajectory. To address these issues, we introduce \textsc{NaVIDA} (\textbf{Nav}igation with \textbf{I}nverse \textbf{D}ynamics \textbf{A}ugmentation), a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution. \textsc{NaVIDA} augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions. To structure this supervision and extend the effective planning range, \textsc{NaVIDA} employs hierarchical probabilistic action chunking (HPAC), which organizes trajectories into multi-step chunks and provides discriminative, longer-range visual-change cues. To further curb error accumulation and stabilize behavior at inference, an entropy-guided mechanism adaptively sets the execution horizon of action chunks. Extensive experiments show that \textsc{NaVIDA} achieves superior navigation performance compared to state-of-the-art methods with fewer parameters (3B vs. 8B). Real-world robot evaluations further validate the practical feasibility and effectiveness of our approach. Code and data will be available upon acceptance.
$\mathcal{P}^3$: Toward Versatile Embodied Agents
Embodied agents have demonstrated promising capabilities in interacting with physical environments. Yet, versatile embodied agents face three core bottlenecks: dynamic environmental perception, open tool access, and complex multi-task planning. Prior methods depend entirely on tool feedback to track scene changes and task progress, leading to poor real-time adaptability, error accumulation, and limited tool compatibility; multi-task scheduling is also understudied due to the difficulty of handling task dependencies and conflicting priorities. To address these limitations, we propose $\mathcal P^3$, a unified framework integrating real-time perception and dynamic scheduling, which perceives task-relevant information actively from the environment, plugs and utilizes tools without feedback requirements, and plans multi-task execution by prioritizing urgent tasks and dynamically adjusting task order based on dependencies. We additionally build the Active Task Perception (ATP) benchmark to quantitatively measure VLMs' capacity for active scene understanding and task proposal. Evaluations on the ATP benchmark verify that multiple VLMs can detect and propose active tasks, and comprehensive real-world robot experiments prove our method bridges the gap between benchmarks and practical deployment, yielding transferable general-purpose embodied agents. Code and data are available at https://github.com/fz-zsl/P3.