Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting

Mesh-adsorbed Gaussian Splatting (MaGS) is a novel framework for reconstructing and simulating dynamic 3D objects.
It combines the rendering flexibility of 3D Gaussians with the spatial coherence of meshes, enabling both high-quality rendering and realistic deformation in dynamic scenes.

📄 More details: Project Page | Paper (arXiv)

Prerequisites
Before installation, please ensure that:

• CUDA Toolkit is installed and matches your GPU & driver version.
• PyTorch is installed with CUDA support.

# Clone the main repository
git clone https://github.com/wcwac/MaGS.git
# Clone required submodules
git clone --recursive https://gitlab.inria.fr/bkerbl/simple-knn.git submodules/simple-knn
git clone --recursive https://github.com/graphdeco-inria/diff-gaussian-rasterization.git submodules/diff-gaussian-rasterization
# Install dependencies
pip install -r requirements.txt
pip install -e submodules/simple-knn
pip install -e submodules/diff-gaussian-rasterization

1. Download the dataset

   • Original dataset: D-NeRF (official)
   • Fixed version: Deformable-3D-Gaussians

2. Download the meshes archive

   • Get D-NeRF_meshes.7z from the release page.

3. Organize and extract

   • Put both the image data and D-NeRF_meshes.7z in the same directory.

   • Extract both archives. You should get a structure like:

     bouncingballs/
     ├── train/
     │ ├── r_000.png
     │ └── r_001.png
     ├── train_meshes/
     │ ├── r_000.ply
     │ └── r_001.ply
     ├── test/
     │ ├── r_000.png
     │ └── r_001.png
     └── test_meshes/
      ├── r_000.ply
      └── r_001.ply
     

The .ply files let you reproduce the paper’s results immediately. Scripts for generating meshes from scratch (and for additional scenes) will be released soon.

The steps are similar to D-NeRF, with just a few differences:

1. Download the dataset

   • Dataset: DG-Mesh

2. Download the meshes archive

   • Get DG-Mesh_meshes.7z from the release page.

3. Organize and extract

   • Same procedure as in the D-NeRF case — place both archives in the same directory and extract.

# Example: D-NeRF Jumping Jacks
python main.py config/3dgs.yaml,config/dnerf/jumpingjacks.yaml

If you find this work useful in your research, please consider citing:

@article{mags2024,
 title={MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting},
 author={Shaojie Ma and Yawei Luo and Wei Yang and Yi Yang},
 year={2024},
 eprint={2406.01593},
 archivePrefix={arXiv},
 primaryClass={cs.CV},
 url={https://arxiv.org/abs/2406.01593}
}