Haoyi Zhu 朱皓怡

Ph.D student in Computer Science

Hi, there!

I am a third-year Ph.D. student in Computer Science at University of Science and Technology of China (USTC) advised by Prof. Tong He, Prof. Wanli Ouyang and Prof. Xiaogang Wang. I earned my B.S. degree in Artificial Intelligence Honor Class at Shanghai Jiao Tong University (SJTU), advised by Prof. Cewu Lu. I also have had the privilege of working with Dr. Hao-Shu Fang and Dr. Jim Fan.

Research for fun and truth. My current research interests focus on World Model, Embodied AI and Spatial Intelligence. The ultimate goal of my life is to discover myself, find the truth, and change the world! Feel free to follow me on and for latest research announcements and updates!

In my personal life, I am passionate (but amateur) about football, music, literature, philosophy, traditional Chinese painting, and modern Chinese poems!

“The philosophers have only interpreted the world, in various ways. The point, however, is to change it.”

✨ News ✨

Oct. 2025: Aether has won 🎉 Outstanding Paper Award 🎉 and has presented an Oral Presentation at ICCV 2025 RIWM workshop!
Sep. 2025: OmniWorld has been released! It is a multi-domain and multi-modal dataset for 4D world modeling. Check it out today!
Sep. 2025: WinT3R has been announced! It is a feed-forward reconstruction model capable of online prediction of precise camera poses and high-quality point maps. Check it out!
Jul. 2025: $π^{3}$ has been announced! $π^{3}$ is a novel feed-forward neural network that revolutionizes visual geometry reconstruction by eliminating the need for a fixed reference view. Paper, code, and demo are all open access. Check it out today!
Jun. 2025: Aether and VQ-VLA are accepted by ICCV 2025!
Jun. 2025: DeepVerse, an auto-regressive 4D world model, has been released!
Feb. 2025: SPA has been accepted by ICLR 2025 and Tra-MoE has been accepted by CVPR 2025!
Oct. 2024: SPA has been announced! SPA is a novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI. Paper, code, and pre-trained models are all open-sourced! Check it out!
Sep. 2024: PointCloudMatters is accepted by NeurIPS D&B 2024! We prove that explicit representation like point cloud can significantly enhance the performance and generalization ability of robot learning policies. Codes are open-sourced!
Oct. 2023: PonderV2 and UniPAD has been announced! PonderV2 is a universal pre-training paradigm for 3D vision, paving the way for 3D foundation model.
Jul. 2023: RH20T has been announced! RH20T is a large-scale, open-source, real-world robotic dataset.
Nov. 2022: MineDojo has won 🎉 Outstanding Paper Award 🎉 at NeurIPS announcement!
Nov. 2022: AlphaPose paper is accepted by TPAMI! AlphaPose is an accurate multi-person pose estimator, which has received more than 8.3K stars on Github.
Jun. 2022: MineDojo has been announced! MineDojo is a new framework for building generally capable agents with internet-scale knowledge in Minecraft.

Interests

World Model
Embodied AI
Spatial Intelligence

Education

B.S. in Artificial Intelligence Honor Class, 2019 - 2023
Shanghai Jiao Tong University
Ph.D. in Computer Science, 2023 - Present
USTC

Featured

Research Highlights

Aether

A geometric-aware unified world model, capable of 4D reconstruction, action-conditioned prediction, and visual planning.

SPA

A novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI.

AlphaPose

An accurate real-time multi-person pose estimator and tracker.

MineDojo

Building open-ended agents with internet-scale knowledge in Minecraft.

Publications

Visit my Google Scholar page for a comprehensive listing!

OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

arXiv preprint, 2025.

... (other authors), Haoyi Zhu 朱皓怡, ... (19 authors)

OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling

WinT3R: Window-Based Streaming Reconstruction With Camera Token Pool

arXiv preprint, 2025.

Zizun Li, Jianjun Zhou, Yifan Wang, Haoyi Guo, Wenzheng Chang, Yang Zhou, Haoyi Zhu 朱皓怡, Junyi Chen, Chunhua Shen, Tong He

WinT3R: Window-Based Streaming Reconstruction With Camera Token Pool

π^{3}

: Scalable Permutation-Equivariant Visual Geometry Learning

arXiv preprint, 2025.

Yifan Wang, Jianjun Zhou, Haoyi Zhu 朱皓怡, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, Tong He

$$\pi^3$: Scalable Permutation-Equivariant Visual Geometry Learning$

VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers

Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Yating Wang, Haoyi Zhu 朱皓怡, Mingyu Liu, Jiange Yang, Hao-Shu Fang, Tong He

VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers

DeepVerse: 4D Autoregressive Video Generation as a World Model

arXiv preprint, 2025.

Junyi Chen, Haoyi Zhu 朱皓怡, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, Tong He

DeepVerse: 4D Autoregressive Video Generation as a World Model

CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

arXiv preprint, 2025.

Jiange Yang, Yansong Shi, Haoyi Zhu 朱皓怡, Mingyu Liu, Kaijing Ma, Yating Wang, Gangshan Wu, Tong He, Limin Wang

CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

Aether: Geometric-Aware Unified World Modeling

💫✨ Outstanding Paper Award ✨💫 at 1st ICCV workshop on Reliable and Interactive World Models (RIWM), 2025 & International Conference on Computer Vision (ICCV), 2025.

Haoyi Zhu 朱皓怡, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, Tong He

Aether: Geometric-Aware Unified World Modeling

Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

Jiange Yang, Haoyi Zhu 朱皓怡, Yating Wang, Gangshan Wu, Tong He, Limin Wang

Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning

SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

The Thirteenth International Conference on Learning Representations (ICLR), 2025.

Haoyi Zhu 朱皓怡, Honghui Yang, Yating Wang, Jiange Yang, Limin Wang, Tong He

SPA: 3D Spatial-Awareness Enables Effective Embodied Representation

Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning

Neural Information Processing Systems (NeurIPS) Dataset & Benchmark, 2024.

Haoyi Zhu 朱皓怡, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, Tong He

Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning

PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm

IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2025..

Haoyi Zhu 朱皓怡, Honghui Yang, Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong He, Hengshuang Zhao, Chunhua Shen, Yu Qiao, Wanli Ouyang

PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm

UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu 朱皓怡, Tong He, Shixiang Tang, Hengshuang Zhao, Qibo Qiu, Binbin Lin, Xiaofei He, Wanli Ouyang

UniPAD: A Universal Pre-training Paradigm for Autonomous Driving

RH20T: A Robotic Dataset for Learning Diverse Skills in One-Shot

IEEE International Conference on Robotics and Automation (ICRA), 2024.

Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang, Haoyi Zhu 朱皓怡, Cewu Lu

RH20T: A Robotic Dataset for Learning Diverse Skills in One-Shot

AlphaTracker: a multi-animal tracking and behavioral analysis tool

Frontiers in Behavioral Neuroscience, 2023.

Zexin Chen, Ruihan Zhang, Hao-Shu Fang, Yu E. Zhang, Aneesh Bal, Haowen Zhou, Rachel R. Rock, Nancy Padilla-Coreano, Laurel R. Keyes, Haoyi Zhu 朱皓怡, Yong-Lu Li, Takaki Komiyama, Kay M. Tye, Cewu Lu

AlphaTracker: a multi-animal tracking and behavioral analysis tool

AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu 朱皓怡, Yuliang Xiu, Yong-Lu Li, Cewu Lu

X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360 Insufficient RGB-D Views

IEEE Winter Conference on Applications of Computer Vision (WACV), 2023.

Haoyi Zhu 朱皓怡, Hao-Shu Fang, Cewu Lu

X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360 Insufficient RGB-D Views

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

💫✨ Outstanding Paper Award ✨💫. Neural Information Processing Systems (NeurIPS) Dataset & Benchmark, 2022.

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu 朱皓怡, Andrew Tang, De-An Huang, Yuke Zhu, Anima Anandkumar

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Unsupervised Multi-Task Learning for 3D Subtomogram Image Alignment, Clustering and Segmentation

IEEE International Conference on Image Processing (ICIP), 2022.

Haoyi Zhu 朱皓怡, Chuting Wang, Yuanxin Wang, Zhaoxin Fan, Mostofa Rafid Uddin, Xin Gao, Jing Zhang, Xiangrui Zeng, Min Xu

Unsupervised Multi-Task Learning for 3D Subtomogram Image Alignment, Clustering and Segmentation

Experience

Ph.D in Computer Science

USTC

Sep 2023 – Present Shanghai, China

Ph.D. at USTC, majoring in Computer Science..
Doctoral advisor: Prof. Tong He, Prof. Wanli Ouyang and Prof. Xiaogang Wang.
Won the National Scholarship and the First Price Academic Scholarship at USTC.

Research Intern

Shanghai AI Lab

Sep 2023 – Present Shanghai, China

Conducting AI research on 3D vision, world model and Embodied AI.
Published several first-author papers at T-PAMI, NeurIPS, ICLR, ICCV, etc.
Aether won the Outstanding Paper Award and presented an Oral Presentation at ICCV 2025 RIWM workshop.

B.S. in Artificial Intelligence

MVIG, SJTU

Sep 2019 – Jun 2023 Shanghai, China

Co-author of AlphaPose, an accurate multi-person pose estimator. Paper accpted by TPAMI.
Proposed X-NeRF, paper accepted by WACV 2023.
Major in the first AI Honor class.
Research on computer vision and robotics at MVIG advised by Dr. Hao-shu Fang and Prof. Cewu Lu.

Remote Research Intern

Jim Team, NVIDIA AI Lab and Caltech

Feb 2022 – Feb 2023 Shanghai, China

Co-developed MineDojo for open-ended agent learning in Minecraft advised by Dr. Jim Fan and Prof. Anima Anandkumar.
Paper anounced as Outstanding Paper Award at NeurIPS Dataset & Benchmark 2022.