Using human detection confidence as a simple yet effective reward model to improve human motion in video generation.

Introducing FreqWarm, a plug-and-play frequency warm-up curriculum that improves high-dimensional latent diffusion by increasing early-stage exposure to high-frequency signals.

Simplified image tokenization using diffusion

Pure text-only LLMs can use off-the-shelf multimodal embedding models to do various multimodal tasks!

Using flow to improve motion in video generation

State-of-the-Art Video (+Audio) Generation Model

State-of-the-Art open-source LLM with multimodal capabilities

Self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos.

SOTA instance-conditioned diffusion model for image generation.

Introducing a new task of generating instructions for the task you want to solve with illustrations, and a LLM + Diffusion model based solution.

Image animation FTW again! SOTA video editing results by animating the first frame with motion conditioning.

SOTA unsupervised video segmentation using CutLER.

A simple and effective approach to high-quality video generation by learning to animate high quality images.

One embedding space for 6 different modalities, enables zero-shot recognition on all modalities!

Scaling up MAE pre-pretraining, followed by weakly supervised pretraining, leads to strong representations.

Discovering objects using DINO features, and learning an unsupervised detection + segmentation model

Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text. We propose HierVL, a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. As training data, we take videos accompanied by timestamped text descriptions of human actions, together with a high-level text summary of the activity throughout the long video (as are available in Ego4D). We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. While the clip-level constraints use the step-by-step descriptions to capture what is happening in that instant, the video-level constraints use the summary text to capture why it is happening, i.e., the broader context for the activity and the intent of the actor. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA results on tasks requiring long-term video modeling. HierVL successfully transfers to multiple challenging downstream tasks (in EPIC-KITCHENS-100, Charades-Ego, HowTo100M) in both zero-shot and fine-tuned settings.

Leveraging LLMs to auto-annotate videos for representation learning.

Single self-supervised representation for images and videos.

A single model for images, video and single-view 3D.

The largest egocentric video dataset.

Leverages image classification data to build an object detector

SOTA video segmentation using Mask2Former.

Single architecture state-of-the-art in instance, semantic and panoptic segmentation.

First Transformer based detection architecture for 3D data.

An autoregressive video transformer architecture for action anticipation in videos.

WyPR can detect and segment objects in a 3D scene without needing any spatial labels at all!

SOTA 3D detection/segmentation results by learning contrastive representations on 3D data

Self-supervised representations for physical reasoning.

Forward prediction for PHYRE benchmark.

A dataset to evaluate temporal reasoning in video models.

A dataset to evaluate temporal reasoning in video models.

Distilling representations from image models to video models.

Among the first applications of Transformers to model videos. SOTA results: close 2nd at AVA Challenge, CVPR'18.

Human keypoint tracking approach that ranked first in ICCV 2017 PoseTrack keypoint tracking challenge!

Among the first applications of attention for contemporary video/action understanding.

Aggregating visual features for action recognition.

Learning how humans interact with their environment by watching TV.

A single embedding space, good for both generating and understanding 3D models