2606.13192v1 Jun 11, 2026 cs.AI

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

Yuyu Zhang
Yuyu Zhang
Citations: 106
h-index: 4
Zhou Fang
Zhou Fang
Citations: 9
h-index: 2
Ruichao Mao
Ruichao Mao
Citations: 2
h-index: 1
Maji Huang
Maji Huang
Citations: 2
h-index: 1
Hai Rao
Hai Rao
Citations: 0
h-index: 0
Teng Guo
Teng Guo
Citations: 227
h-index: 10
Hao Yang
Hao Yang
Citations: 27
h-index: 2
Yaping Li
Yaping Li
Citations: 29
h-index: 1
Shaohua Peng
Shaohua Peng
Citations: 0
h-index: 0
Xiaoyu Lin
Xiaoyu Lin
Citations: 67
h-index: 3
Shuoyang Liu
Shuoyang Liu
Citations: 2
h-index: 1
Xuepeng Li
Xuepeng Li
Citations: 31
h-index: 3

User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.

0 Citations
0 Influential
5 Altmetric
25.0 Score
Original PDF

No Analysis Report Yet

This paper hasn't been analyzed by Gemini yet.

Log in to request an AI analysis.

댓글

댓글을 작성하려면 로그인하세요.

아직 댓글이 없습니다. 첫 번째 댓글을 남겨보세요!