VISTA-Bench motivation

Do VLMs really understand visualized text as well as pure text?
A systematic benchmark spanning multimodal perception → reasoning → unimodal knowledge.

License Dataset Tasks Eval

Figure 1: Motivation of VISTA-Bench.

We introduce VISTA-Bench, a systematic benchmark spanning multimodal perception, reasoning, and unimodal understanding. It evaluates visualized text understanding by contrasting pure-text and visualized-text (VT) questions under controlled rendering conditions.

• Size: 1,500 instances
• Composition: predominantly multiple-choice questions (MCQ), with a small portion of open-ended queries
• Task taxonomy (4 primary categories):
  • Unimodal Knowledge: 500
  • Multimodal Knowledge: 400
  • Multimodal Perception: 300
  • Multimodal Reasoning: 300

VISTA-Bench ability dimensions

Figure 2: Ability dimensions and task taxonomy of VISTA-Bench.

VISTA-Bench uni-knowledge case

VISTA-Bench multi-reasoning case

Figure 3: A representative multimodal perception case under the visualized-text interface.

VISTA-Bench/
├─ assets/figures/ # figures used in this README
├─ images/ # original images (for multimodal instances)
├─ questions/ # rendered question/option images (VT setting)
├─ VLMEvalKit/ # evaluation toolkit
├─ VISTA-Bench.tsv # dataset index (currently identical to the VT variant)
└─ VISTA-Bench-VT.tsv # dataset index (currently identical; kept for compatibility)

Note: VISTA-Bench.tsv and VISTA-Bench-VT.tsv are currently identical; we keep both filenames for compatibility and will refine the organization later.

• images/: original images used by multimodal instances
• questions/: rendered question/option images for the visualized-text (VT) setting
• *.tsv: dataset metadata and file paths used for evaluation

We evaluate VISTA-Bench with VLMEvalKit/.
Before running evaluation, we recommend converting the TSV file(s) into a normalized format with absolute image paths.

Helper script:

• VISTA-Bench/VLMEvalKit/utils/convert_data_file.py

What it does:

• converts the TSV encoding to UTF-8
• normalizes path separators (\ → /)
• renames options_A/B/C/D → A/B/C/D when needed
• converts image_path and question_image_path into a bracket-style multi-path string

Example:

python VISTA-Bench/VLMEvalKit/utils/convert_data_file.py \
 --in VISTA-Bench/VISTA-Bench.tsv \
 --out VISTA-Bench/VISTA-Bench_norm.tsv \
 --image-prefix /ABS/PATH/TO/VISTA-Bench \

• --in: input TSV path (the original dataset TSV to be converted)
• --out: output TSV path (the converted/normalized TSV produced by this script)
• --image-prefix: the dataset root directory where images/, questions/, and the TSV files are located (used to resolve relative paths)

After conversion, rename the TSV column header image to image-1 to avoid an AssertionError in some VLMEvalKit setups.

Pure-text:

python /VISTA-Bench/VLMEvalKit/run.py \
 --data VISTA-Bench_norm \
 --model llava_v1.5_7b \
 --verbose

Visualized-text (VT):

python /VISTA-Bench/VLMEvalKit/run.py \
 --data VISTA-Bench-VT \
 --model llava_v1.5_7b \
 --verbose

• --data: the dataset name corresponding to your converted TSV (e.g., VISTA-Bench_norm). The VT split (VISTA-Bench-VT) should also be converted and can be run directly using this name because it is registered in VLMEvalKit/vlmeval/dataset/image_mm_mcq.py via DATASET_URL/MD5.
• --model: the model name defined in VLMEvalKit/vlmeval/config.py (make sure the corresponding weights are available in your environment).
• --verbose: print detailed logs during evaluation.
• Outputs: the final report includes overall results and l1-categories breakdown.

• Qing'an Liu: 2223884741@mail.dlut.edu.cn
• Juntong Feng: 2253762636@mail.dlut.edu.cn

To be added.

To be added.