🎬 NetfLips

License Python

📋 Overview

✨ Key Features

🎯 Keywords

🎥 Demo

🌐 Demo Link

🏗️ Architecture

1️⃣ Unit Extraction

2️⃣ Unit Translation

3️⃣ AV Generation

📊 Dataset

🚀 Getting Started

Prerequisites

# 1. 레포지토리 클론
git clone https://github.com/Prometheus-AI-3team/NetfLips.git
cd NetfLips
# 2. 서브모듈(fairseq) update
git submodule init
git submodule update
# 2. Conda 기본 환경 생성
conda env create -f environment.yml
conda activate unit2a
# 3. Pip 다운그레이드 (메타데이터 에러 방지)
pip install "pip<24.1"
# 4. PyTorch 설치 (CUDA 11.7 기준)
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

Installation

# 5. 나머지 라이브러리 설치
pip install -r requirements.txt
# 6. Fairseq 설치
cd av2av-main/fairseq
pip install -e .

💻 Usage

Checkpoints

End-to-End Inference

PYTHONPATH=fairseq python inference.py \
 --in-vid-path /path/to/input.mp4 \
 --out-vid-path /path/to/output.mp4 \
 --src-lang en --tgt-lang ko \
 --av2unit-path /path/to/mavhubert_large_noise.pt \
 --utut-path /path/to/utut_sts_ft.pt \
 --unit2av-path /path/to/unit_av_renderer_withKO.pt \

Training & Inference

각 모듈의 학습 및 추론(av2unit, unit2unit, unit2av)은 해당하는 모듈의 README.md를 참고해주세요.

📁 Project Structure

NetfLips/
├── av2unit/ # Audio-Visual to Unit Extraction
│ ├── avhubert/ # Feature extraction using AV-HuBERT
│ └── inference.py # Unit extraction inference script
├── unit2unit/ # Unit to Unit Translation
│ ├── utut_pretrain/ # Pre-training modules
│ ├── utut_finetune/ # Fine-tuning modules
│ └── inference.py # Translation inference script
├── unit2av/ # Unit to Audio-Visual Generation
│ ├── model.py # Unit2AV model definition
│ ├── train_unit2a.py # Training script for Unit2Audio
│ └── inference_unit2av.py # Inference scripts
├── fairseq/ # Fairseq Toolkit (Submodule)
├── scripts/ # Utility Scripts for Data Preparation
├── inference_av2av.py # Main End-to-End Inference Script
├── environment.yml # Conda Environment Configuration
└── requirements.txt # Python Dependencies

🔬 Methodology

Data Preprocessing

Model Training

Audio-Visual Generation

🛠️ Technical Details

Base Model

Fine-tuning Strategy

👥 Team Members From Prometheus(AI club)

📝 Citation

@misc{netflips2024,
 title={NetfLips: Unit-based Audiovisual Translation for Korean},
 author={장지수, 유지혜, 신규철, 이가연},
 year={2024}
}

References

License

Acknowledgments