Shunqi Mao
Publications
Step-Level Preference Learning for Generative Agents in Social Simulations
Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.
Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Response
Large language models (LLMs) have substantially improved the planning capabilities of embodied agents, enabling their deployment in dynamic and safety-critical environments. However, these settings expose a critical limitation: inference latency. Delayed LLM responses can weaken real-time responsiveness and misalign agent reasoning with rapidly changing environmental states. This paper systematically studies the impact of inference latency on LLM-based embodied agents in dynamic environments. We introduce an FPS-based Time Conversion Mechanism (TCM) that maps inference time to elapsed simulation time, allowing computational delays to directly affect environmental evolution and agent outcomes. We instantiate this protocol in HAZARD and introduce Response Latency (RL) and Latency-to-Action Ratio (LAR) to evaluate agent responsiveness. Building on this framework, we propose the Rapid-Reflex Async-Reflect Agent (RRARA), which integrates rapid reflexive actions with asynchronous LLM reflection to mitigate latency-induced errors. We further introduce an LLM-based PrePlanner that generates cached object-centric subgoals, reducing repeated LLM calls while retaining the model's high-level reasoning capability. Experiments show that accounting for inference latency substantially changes embodied-agent performance and that RRARA achieves a stronger balance between decision quality and responsiveness.