[arXiv] [Project Page]
Min-Jung Kim*, Jeongho Kim*, Hoiyeong Jin, Junha Hyung, Jaegul Choo
*Equal Contribution
ours_grid_in-the-wild_1.mp4
- Release data augmentation code (2025εΉ΄12ζ29ζ₯)
- Release training code (2025εΉ΄12ζ26ζ₯)
- Release inference code (2025εΉ΄12ζ19ζ₯)
- Release model weights (2025εΉ΄12ζ19ζ₯)
TL;DR: Given a video and a target camera trajectory, InfCam generates a video that faithfully follows the specified camera path without depth prior.
conda create -n infcam python=3.12
conda activate infcam
# for inference only
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install cupy-cuda12x
pip install transformers==4.46.2
pip install sentencepiece
pip install controlnet-aux==0.0.7
pip install imageio
pip install imageio[ffmpeg]
pip install safetensors
pip install einops
pip install protobuf
pip install modelscope
pip install ftfy
pip install lpips
pip install lightning
pip install pandas
pip install matplotlib
pip install wandb
pip install ffmpeg-python
pip install numpy
pip install opencv-python
# for data processing
conda install -c conda-forge ffmpeg
# for training
pip install deepspeed
Hardware: 1x NVIDIA H100 80GB GPUs.
Memory Usage: > 50 GB (48G of memory for UniDepth and 28 GB for the InfCam pipeline.)
Step 1: Download the pretrained checkpoints
(1) Pre-trained Wan2.1 model
python download_wan2.1.py
(2) Pre-trained UniDepth model
Download the pre-trained weights from huggingface and place it in models/unidepth-v2-vitl14.
cd models
git clone https://huggingface.co/lpiccinelli/unidepth-v2-vitl14(3) Pre-trained InfCam checkpoint
Download the pre-trained InfCam weights from huggingface and place it in models/InfCam.
cd models
git clone https://huggingface.co/emjay73/InfCamStep 2: Test the example videos
bash run_inference.sh
or
for CAM in {1..10}; do CUDA_VISIBLE_DEVICES=0 python inference_infcam.py \ --cam_type ${CAM} \ --ckpt_path "models/InfCam/step35000.ckpt" \ --camera_extrinsics_path "./sample_data/cameras/camera_extrinsics_10types.json" \ --output_dir "./results/sample_data" \ --dataset_path "./sample_data" \ --metadata_file_name "metadata.csv" \ --num_frames 81 --width 832 --height 480 \ --num_inference_steps 20 \ --zoom_factor 1.0 \ --k_from_unidepth \ --seed ${SEED} done
Step 3: Test your own videos
If you want to test your own videos, you need to prepare your test data following the structure of the sample_data folder. This includes N mp4 videos, each with at least 81 frames, and a metadata.csv file that stores their paths and corresponding captions. You can refer to the 'caption branch for metadata.csv extraction.
We provide several preset camera types, as shown in the table below. These follow the ReCamMaster presets, but the starting point of each trajectory differs from that of the initial frame.
| cam_type | Trajectory |
|---|---|
| 1 | Pan Right |
| 2 | Pan Left |
| 3 | Tilt Up |
| 4 | Tilt Down |
| 5 | Zoom In |
| 6 | Zoom Out |
| 7 | Translate Up (with rotation) |
| 8 | Translate Down (with rotation) |
| 9 | Arc Left (with rotation) |
| 10 | Arc Right (with rotation) |
Hardware: 4x NVIDIA H100 80GB GPUs.
Memory Usage(low resolution, B=8, F=41 H=320 W=544): Approximately 52GB of VRAM per GPU during training.
Memory Usage(high resolution, B=2, F=81 H=480 W=832): Approximately 56GB of VRAM per GPU during training.
Step1. Prepare Dataset
-
Download MultiCamVideo-Dataset
-
Augment MultiCamVideo-Dataset
python preproc/step1_aug_trajectory.py --path_mcv "path to MultiCamVideo-Dataset/train" --path_augmcv DATA/AugMCV
python preproc/step2_aug_focallength.py --path_data DATA/AugMCV
python preproc/step3_split_train_test.py --path_mcv "path to MultiCamVideo-Dataset/train" --path_augmcv DATA/AugMCV
If you would like to see an example of the training set with augmentation already applied, you can download a subset from Hugging Face: AugMCV.
mkdir DATA
cd DATA
# download train data subset
git clone https://huggingface.co/datasets/emjay73/AugMCV
cd AugMCV
tar -xvzf AugMCV.tar.gz
The training data should follow the directory structure shown below:
InfCam
βββ DATA
βββ AugMCV
βββ train
β βββ f18_aperture10
β β βββ scene1_0
β β βββ cameras
β β β βββ camera_extrinsics.json
β β β βββ video_mapping.json
β β βββ videos
β β βββ cam01.mp4
β β βββ cam01.mp4.[config].pth
β β ...
β β βββ cam10.mp4
β β βββ cam10.mp4.[config].pth
β βββ f18_aperture10_aug
β ...
β βββ f50_aperture2.4
βββ metadata_augmcv_train.csv
Step2. Run train
bash run_train.sh
We build upon the following repositories and thank the authors for their incredible work:
ReCamMaster: Re-capture in-the-wild videos with novel camera trajectories, and release a multi-camera synchronized video dataset rendered with Unreal Engine 5.
WAN2.1: A comprehensive and open suite of video foundation models.
UniDepthV2: Monocular metric depth estimation.
Please leave us a star π and cite our paper if you find our work helpful.
bibtex
@article{kim2025infcam,
title={Infinite-Homography as Robust Conditioning for Camera-Controlled Video Generation},
author={Kim, Min-Jung and Kim, Jeongho and Jin, Hoiyeong and Hyung, Junha and Choo, Jaegul},
journal={arXiv preprint arXiv:2512.17040},
year={2025}
}