TongUI Logo

Dynamic JSON Badge Dynamic JSON Badge Dynamic JSON Badge Dynamic JSON Badge

Training Vision-Language-Action(VLA) Model for GUI & Computer Use tasks by watching online tutorials. Fully open-sourced dataset, model and training pipeline. Cost efficient solution for GUI task data generation.

针对图形操作界面任务设计的VLA模型和智能体框架。

📑 Paper | 🤗 HuggingFace Collections (Models & Datasets) | 🤖 ModelScope Collections (Models & Datasets) | 🤗 Spaces Demo | 🌐 Webpage

TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials
Bofei Zhang*, Zirui Shang*, Zhi Gao*, Wang Zhang, Rui Xie, Xiaojian Ma, Yuan Tao, Xinxiao Wu, Song-Chun Zhu, Qing Li✉

TongUI

• Release all experiments/evaluation scripts [WIP].
• [2025年10月23日] Release TongUI-Absolute — annotated dataset with absolute coordinate labels.
• [2025年07月10日] Release crawler code and intermediate crawler data🤗. Please feel free to process your own SFT dataset!
• [2025年06月16日] Submit evaluation to UI-Vision! Checkout result here and how to reproduce here
• [2025年05月27日] Release TongUI-32B model and Training Details.
• [2025年05月06日] Release TongUI-7B model and GUI-Net-1M dataset.
• [2025年04月21日] Release 🔧 Training pipeline.
• [2025年04月17日] Release TongUI-3B model.

Key findings

• Training with this cost-efficient dataset gives SOTA👑 performance on Multiple GUI benchmarks!
• Training with 1M version of dataset make the performance scale up🚀!

Results on ScreenSpot; † means the results are re-produced. We report results on six splits of ScreenSpot and the average scores. The best method is marked in bold. 1M means the dataset is 1M version.

Results on Mind2Web. We report results on three types of tasks: cross-task, cross-website, and cross-domain. Elem. Acc means whether the element is selected correctly, OP. F1 denotes the F1 score for the predicted action, and Step SR counts successful steps. 1M means the dataset is 1M version.

For other experiments, please refer to our paper.

We use uv to manage the dependencies.

uv sync --all-groups

To using conda and pip to install the dependencies.

conda create -n tongui python=3.12
conda activate tongui
pip install -e .

To execute any script by uv, you can use the following command.

uv run <script_name>.py

Just replace uv with python if you are using conda or pip to install the dependencies.

python <script_name>.py

We host an online Gradio Demo on Hugging Face Spaces. Please feel free to try it. We also open source the code for this demo. Feel free to run it locally.

git clone https://huggingface.co/spaces/Bofeee5675/TongUI
cd TongUI
uv run app.py

You can programatically call the TongUI API by using the following code.

uv run examples/api.py

You can serve the model by vLLM.

uv run vllm serve Bofeee5675/TongUI-3B --port 8000 --served-model-name tongui-3b --limit-mm-per-prompt image=3

Then, you can use openai compatible API to call the model. Checkout examples/call_vllm.py for more details.

uv run examples/call_vllm.py

Checkout examples/inference.py for local inference.

uv run examples/inference.py

For detailed information about model training, including hyperparameters, data preprocessing, and training configurations, please refer to our Training Documentation.

For comprehensive experimental results, ablation studies, and evaluation details, please check our Experiments Documentation.

Star History Chart

We thank the following projects for their wonderful works.

• We adopt experiments, data preprocessing pipeline from ShowUI
• We train our model by using LLaMA-Factory
• Thanks for Qwen2.5-VL series model and UI-TARS for their great work.

If you find this work useful in your research, please consider citing:

@article{zhang2025tongui,
 title={TongUI: Building Generalized GUI Agents by Learning from Multimodal Web Tutorials},
 author={Zhang, Bofei and Shang, Zirui and Gao, Zhi and Zhang, Wang and Xie, Rui and Ma, Xiaojian and Yuan, Tao and Wu, Xinxiao and Zhu, Song-Chun and Li, Qing},
 journal={arXiv preprint arXiv:2504.12679},
 year={2025}
}