Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat: add TwelveLabs video understanding agent (Pegasus + Marengo)#197

Open
mohit-twelvelabs wants to merge 2 commits into
video-db:main from
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: add TwelveLabs video understanding agent (Pegasus + Marengo) #197
mohit-twelvelabs wants to merge 2 commits into
video-db:main from
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

@mohit-twelvelabs mohit-twelvelabs commented Jun 24, 2026
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

What this adds

A new opt-in video_understanding agent that analyzes a VideoDB video with TwelveLabs Pegasus for true multimodal understanding — grounded in the actual footage (scenes, objects, on-screen actions) and audio, not just the spoken-word transcript. It complements the existing summarize_video agent, which reasons over the transcript only.

It also adds a small TwelveLabsTool wrapper (director/tools/twelvelabs_tool.py) exposing:

  • Pegasus video analysis (analyze_video)
  • Marengo multimodal embeddings (get_text_embedding, 512-dim) — same embedding space as TwelveLabs video embeddings, useful for text-to-video retrieval.

The agent resolves a downloadable URL from the VideoDB stream and lets TwelveLabs fetch it server-side, mirroring how the existing download/generation agents work.

Why it helps

Director already mixes multiple GenAI APIs per task. Pegasus gives the reasoning engine a way to watch a video — answering questions about visual content that a transcript can't (e.g. "what products appear?", "describe the b-roll"), which is a capability gap today.

Opt-in / non-breaking

  • Gated entirely on the TWELVELABS_API_KEY env var. When it's unset the agent simply reports an error; no defaults change and no existing agent is touched.
  • Wired into handler.py and requirements.txt the same way existing tools/agents are; added to .env.sample and the README feature list.

How it was tested

  • Added backend/tests/test_twelvelabs_tool.py: no-network unit tests (SDK mocked) for the analyze/embed wiring, plus a live Marengo embedding test gated on TWELVELABS_API_KEY (skipped when unset).
  • pytest → 5 passed (including a live Marengo call returning a 512-dim vector against the real API).
  • ruff check and black --check clean on the changed files.
  • Verified director.handler imports and the agent registers.
  • Pinned twelvelabs==1.2.8 (signatures introspected against the installed SDK).

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Summary by CodeRabbit

  • New Features
    • Added multimodal video understanding so the assistant can analyze videos (visual + audio) using TwelveLabs Pegasus.
  • Documentation
    • Updated the README to highlight the new multimodal video understanding capability.
    • Added a TwelveLabs API key placeholder to the sample environment configuration.
  • Tests
    • Added automated tests for the TwelveLabs wrapper, including validation, SDK call wiring, and an optional live embedding test when credentials are available.

Adds an opt-in video_understanding agent that analyzes VideoDB videos
with TwelveLabs Pegasus for true multimodal understanding (visual +
audio), complementing the transcript-only summarize_video agent. Also
adds a TwelveLabsTool wrapper exposing Pegasus analysis and Marengo
multimodal embeddings, gated on TWELVELABS_API_KEY so existing behaviour
is unchanged when the key is unset.

coderabbitai Bot commented Jun 24, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

i️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5e79a79-48a0-4a4b-9a78-d585dc653a06

📥 Commits

Reviewing files that changed from the base of the PR and between a46d957 and e4f5926.

📒 Files selected for processing (2)
  • backend/director/agents/video_understanding.py
  • backend/director/tools/twelvelabs_tool.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • backend/director/agents/video_understanding.py
  • backend/director/tools/twelvelabs_tool.py

📝 Walkthrough

Walkthrough

Adds a TwelveLabs wrapper for video analysis and embeddings, introduces a video understanding agent that resolves a VideoDB video and sends it to TwelveLabs, and wires the agent into chat handling with setup and documentation updates.

Changes

TwelveLabs Video Understanding Feature

Layer / File(s) Summary
TwelveLabs tool and tests
backend/director/tools/twelvelabs_tool.py, backend/requirements.txt, backend/tests/test_twelvelabs_tool.py
Defines the TwelveLabs model constants and parameter schema, implements TwelveLabsTool with API-key validation, analyze_video(), and get_text_embedding(), pins twelvelabs==1.2.8, and adds tests for config defaults and SDK call wiring.
Video understanding agent
backend/director/agents/video_understanding.py
Defines the agent parameters and class metadata, then implements run() to validate the API key, fetch a downloadable video URL via VideoDBTool, call TwelveLabsTool.analyze_video(), publish status updates, and return analysis data or an error.
Handler and setup wiring
backend/director/handler.py, backend/.env.sample, README.md
Imports and registers VideoUnderstandingAgent in ChatHandler, adds the TwelveLabs env placeholder, and updates the README key features text.

Sequence Diagram(s)

sequenceDiagram
 participant ChatHandler
 participant VideoUnderstandingAgent
 participant VideoDBTool
 participant TwelveLabsTool
 ChatHandler->>VideoUnderstandingAgent: run(collection_id, video_id, prompt, twelvelabs_config)
 VideoUnderstandingAgent->>VideoUnderstandingAgent: validate TWELVELABS_API_KEY
 VideoUnderstandingAgent->>VideoDBTool: get_video(collection_id, video_id)
 VideoDBTool-->>VideoUnderstandingAgent: download_url
 VideoUnderstandingAgent->>TwelveLabsTool: analyze_video(download_url, prompt, twelvelabs_config)
 TwelveLabsTool-->>VideoUnderstandingAgent: analysis
 VideoUnderstandingAgent-->>ChatHandler: AgentResponse(data={"analysis": ...})
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • 0xrohitgarg
  • ashish-spext
  • ankit-v2-3

Poem

🐇 A bunny peeks at video trails,
With Pegasus listening through visual veils,
The agent hops, the analysis gleams,
And TwelveLabs turns frames into dreams.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding a TwelveLabs video understanding agent and related tool support.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (6)
backend/director/tools/twelvelabs_tool.py (4)

93-93: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use explicit Optional type hint for optional parameters.

Same as the analyze_video method—PEP 484 recommends explicit | None.

♻️ Proposed fix
- def get_text_embedding(self, text: str, config: dict = None) -> list:
+ def get_text_embedding(self, text: str, config: dict | None = None) -> list:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/director/tools/twelvelabs_tool.py` at line 93, Update the type hint
for the config parameter in the get_text_embedding method to use explicit
Optional typing. Change the parameter declaration from config: dict = None to
config: Optional[dict] = None, following PEP 484 conventions. Ensure that
Optional is imported from the typing module at the top of the file if it is not
already imported.

Source: Linters/SAST tools


63-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use explicit Optional type hint for optional parameters.

PEP 484 recommends config: dict | None = None over the implicit config: dict = None pattern.

♻️ Proposed fix
 def analyze_video(
 self,
 video_url: str,
 prompt: str,
- config: dict = None,
+ config: dict | None = None,
 ) -> str:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/director/tools/twelvelabs_tool.py` at line 63, The config parameter
in twelvelabs_tool.py uses an implicit None type hint (config: dict = None)
which is not explicitly clear according to PEP 484. Change this to use explicit
Optional type hints by either using the union syntax (config: dict | None = None
for Python 3.10+) or importing Optional from typing and using (config:
Optional[dict] = None). Apply this same fix to any other parameters in the file
that follow the same implicit None pattern.

Source: Linters/SAST tools


87-90: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Chain exception context when re-raising.

Python best practice recommends using raise ... from e to preserve the exception chain and make debugging easier. This helps distinguish between errors in the SDK call versus errors in exception handling itself.

♻️ Proposed fix
 except Exception as e:
- raise Exception(
- f"Error analyzing video with TwelveLabs: {type(e).__name__}: {str(e)}"
- )
+ raise Exception(
+ f"Error analyzing video with TwelveLabs: {type(e).__name__}: {e}"
+ ) from e
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/director/tools/twelvelabs_tool.py` around lines 87 - 90, In the
exception handling block where the Exception is raised after catching the
TwelveLabs SDK error, use the `raise ... from e` syntax instead of just `raise
Exception(...)` to preserve the exception chain. This means changing the raise
statement to include `from e` at the end, which will maintain the original
exception context and make debugging easier by showing the full traceback chain
between the original error and the new exception being raised.

Source: Linters/SAST tools


111-114: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Chain exception context when re-raising.

Same best practice as the analyze_video method—preserve the exception chain with from e.

♻️ Proposed fix
 except Exception as e:
- raise Exception(
- f"Error creating embedding with TwelveLabs: {type(e).__name__}: {str(e)}"
- )
+ raise Exception(
+ f"Error creating embedding with TwelveLabs: {type(e).__name__}: {e}"
+ ) from e
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/director/tools/twelvelabs_tool.py` around lines 111 - 114, Preserve
the exception chain in the exception handler block where the Exception is
re-raised with the error message containing type(e).__name__ and str(e). Add
`from e` to the raise statement to maintain the original exception context,
which improves debugging by showing the complete exception chain rather than
just the error message details.

Source: Linters/SAST tools

backend/director/agents/video_understanding.py (1)

68-68: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use explicit Optional type hint for optional parameters.

Consistent with the recommendation for TwelveLabsTool, use twelvelabs_config: dict | None = None for PEP 484 compliance.

♻️ Proposed fix
 def run(
 self,
 collection_id: str,
 video_id: str,
 prompt: str,
- twelvelabs_config: dict = None,
+ twelvelabs_config: dict | None = None,
 *args,
 **kwargs,
 ) -> AgentResponse:
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/director/agents/video_understanding.py` at line 68, The type hint for
the `twelvelabs_config` parameter is missing the explicit `Optional` annotation
for the None type. Update the parameter declaration to use the union syntax
`dict | None` instead of just `dict` to properly indicate that the parameter can
accept either a dictionary or None value, ensuring PEP 484 compliance with the
rest of the codebase.

Source: Linters/SAST tools

backend/tests/test_twelvelabs_tool.py (1)

21-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider using a more specific exception type.

While the current implementation raises a generic Exception, consider introducing a custom exception type (e.g., TwelveLabsConfigError) in the future for better error handling and more precise test assertions.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/test_twelvelabs_tool.py` around lines 21 - 23, The
test_requires_api_key function in test_twelvelabs_tool.py is catching a generic
Exception type instead of a specific custom exception. Create a custom exception
class named TwelveLabsConfigError to handle TwelveLabs configuration errors,
update the TwelveLabsTool constructor to raise this specific exception when the
api_key parameter is empty, and then update the pytest.raises() call in
test_requires_api_key to expect TwelveLabsConfigError instead of the generic
Exception type for more precise error handling.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/director/agents/video_understanding.py`:
- Around line 107-113: Replace the unsafe bracket notation for accessing the
download_url key in the download_response dictionary with the safer .get()
method. Change download_response["download_url"] to
download_response.get("download_url") to prevent potential KeyError exceptions
if the key is missing from the response, ensuring consistency with the defensive
programming pattern already used in other files like voice_replacement.py and
download.py when handling videodb responses.
---
Nitpick comments:
In `@backend/director/agents/video_understanding.py`:
- Line 68: The type hint for the `twelvelabs_config` parameter is missing the
explicit `Optional` annotation for the None type. Update the parameter
declaration to use the union syntax `dict | None` instead of just `dict` to
properly indicate that the parameter can accept either a dictionary or None
value, ensuring PEP 484 compliance with the rest of the codebase.
In `@backend/director/tools/twelvelabs_tool.py`:
- Line 93: Update the type hint for the config parameter in the
get_text_embedding method to use explicit Optional typing. Change the parameter
declaration from config: dict = None to config: Optional[dict] = None, following
PEP 484 conventions. Ensure that Optional is imported from the typing module at
the top of the file if it is not already imported.
- Line 63: The config parameter in twelvelabs_tool.py uses an implicit None type
hint (config: dict = None) which is not explicitly clear according to PEP 484.
Change this to use explicit Optional type hints by either using the union syntax
(config: dict | None = None for Python 3.10+) or importing Optional from typing
and using (config: Optional[dict] = None). Apply this same fix to any other
parameters in the file that follow the same implicit None pattern.
- Around line 87-90: In the exception handling block where the Exception is
raised after catching the TwelveLabs SDK error, use the `raise ... from e`
syntax instead of just `raise Exception(...)` to preserve the exception chain.
This means changing the raise statement to include `from e` at the end, which
will maintain the original exception context and make debugging easier by
showing the full traceback chain between the original error and the new
exception being raised.
- Around line 111-114: Preserve the exception chain in the exception handler
block where the Exception is re-raised with the error message containing
type(e).__name__ and str(e). Add `from e` to the raise statement to maintain the
original exception context, which improves debugging by showing the complete
exception chain rather than just the error message details.
In `@backend/tests/test_twelvelabs_tool.py`:
- Around line 21-23: The test_requires_api_key function in
test_twelvelabs_tool.py is catching a generic Exception type instead of a
specific custom exception. Create a custom exception class named
TwelveLabsConfigError to handle TwelveLabs configuration errors, update the
TwelveLabsTool constructor to raise this specific exception when the api_key
parameter is empty, and then update the pytest.raises() call in
test_requires_api_key to expect TwelveLabsConfigError instead of the generic
Exception type for more precise error handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

i️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bdd2599e-8f93-4551-b5e7-15308f65f2eb

📥 Commits

Reviewing files that changed from the base of the PR and between 70e0b3d and a46d957.

📒 Files selected for processing (7)
  • README.md
  • backend/.env.sample
  • backend/director/agents/video_understanding.py
  • backend/director/handler.py
  • backend/director/tools/twelvelabs_tool.py
  • backend/requirements.txt
  • backend/tests/test_twelvelabs_tool.py

Comment thread backend/director/agents/video_understanding.py Outdated
- Use .get("download_url") with a None guard instead of bracket access,
 matching the defensive pattern in download.py and voice_replacement.py
- Add explicit Optional[dict] type hints for optional config params (PEP 484)
- Chain re-raised exceptions with 'from e' to preserve the original traceback

Copy link
Copy Markdown
Author

Thanks for the review @coderabbitai! Addressed the feedback in e4f5926:

Actionable

  • video_understanding.py: replaced download_response["download_url"] with .get("download_url") plus a guard, matching download.py / voice_replacement.py.

Nitpicks (applied)

  • Explicit Optional[dict] type hints on analyze_video, get_text_embedding (twelvelabs_tool.py) and run's twelvelabs_config (video_understanding.py). Used Optional[...] rather than X | None to stay consistent with the neighbouring agents and the project's requires-python = ">=3.6".
  • Chained re-raised exceptions with ... from e in both twelvelabs_tool.py handlers to preserve the original traceback.

Declined

  • Custom TwelveLabsConfigError exception type: leaving as the generic Exception to match the rest of twelvelabs_tool.py and the agents in this repo (none define per-tool exception classes). Happy to introduce one if the maintainers want a project-wide convention.

All 5 unit tests pass locally, including the live Marengo embedding test.

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@mohit-twelvelabs Thanks for the thorough follow-up — the guard around download_url, the Optional[...] annotations, and the exception chaining all sound good to me. The rationale for skipping a custom TwelveLabsConfigError in this PR also makes sense given the current repo conventions.

Great to hear all 5 tests are passing locally as well. 🐇✨


✏️ Learnings added
Learnt from: mohit-twelvelabs
URL: https://github.com/video-db/Director/pull/197
Timestamp: 2026年06月24日T22:58:56.552Z
Learning: In this repository's Python code under `backend/`, prefer `Optional[...]` type hints over `X | None` for consistency with neighboring agents and compatibility with the project's `requires-python = ">=3.6"` setting.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


If you found this review helpful, would you consider giving us a shout-out on X?

Thank you for using CodeRabbit!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /