-
Notifications
You must be signed in to change notification settings - Fork 231
feat: add TwelveLabs video understanding agent (Pegasus + Marengo)#197
feat: add TwelveLabs video understanding agent (Pegasus + Marengo) #197mohit-twelvelabs wants to merge 2 commits into
Conversation
Adds an opt-in video_understanding agent that analyzes VideoDB videos with TwelveLabs Pegasus for true multimodal understanding (visual + audio), complementing the transcript-only summarize_video agent. Also adds a TwelveLabsTool wrapper exposing Pegasus analysis and Marengo multimodal embeddings, gated on TWELVELABS_API_KEY so existing behaviour is unchanged when the key is unset.
|
No actionable comments were generated in the recent review. 🎉 i️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds a TwelveLabs wrapper for video analysis and embeddings, introduces a video understanding agent that resolves a VideoDB video and sends it to TwelveLabs, and wires the agent into chat handling with setup and documentation updates. ChangesTwelveLabs Video Understanding Feature
Sequence Diagram(s)sequenceDiagram
participant ChatHandler
participant VideoUnderstandingAgent
participant VideoDBTool
participant TwelveLabsTool
ChatHandler->>VideoUnderstandingAgent: run(collection_id, video_id, prompt, twelvelabs_config)
VideoUnderstandingAgent->>VideoUnderstandingAgent: validate TWELVELABS_API_KEY
VideoUnderstandingAgent->>VideoDBTool: get_video(collection_id, video_id)
VideoDBTool-->>VideoUnderstandingAgent: download_url
VideoUnderstandingAgent->>TwelveLabsTool: analyze_video(download_url, prompt, twelvelabs_config)
TwelveLabsTool-->>VideoUnderstandingAgent: analysis
VideoUnderstandingAgent-->>ChatHandler: AgentResponse(data={"analysis": ...})
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🧹 Nitpick comments (6)
backend/director/tools/twelvelabs_tool.py (4)
93-93: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse explicit
Optionaltype hint for optional parameters.Same as the
analyze_videomethod—PEP 484 recommends explicit| None.♻️ Proposed fix
- def get_text_embedding(self, text: str, config: dict = None) -> list: + def get_text_embedding(self, text: str, config: dict | None = None) -> list:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/director/tools/twelvelabs_tool.py` at line 93, Update the type hint for the config parameter in the get_text_embedding method to use explicit Optional typing. Change the parameter declaration from config: dict = None to config: Optional[dict] = None, following PEP 484 conventions. Ensure that Optional is imported from the typing module at the top of the file if it is not already imported.Source: Linters/SAST tools
63-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse explicit
Optionaltype hint for optional parameters.PEP 484 recommends
config: dict | None = Noneover the implicitconfig: dict = Nonepattern.♻️ Proposed fix
def analyze_video( self, video_url: str, prompt: str, - config: dict = None, + config: dict | None = None, ) -> str:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/director/tools/twelvelabs_tool.py` at line 63, The config parameter in twelvelabs_tool.py uses an implicit None type hint (config: dict = None) which is not explicitly clear according to PEP 484. Change this to use explicit Optional type hints by either using the union syntax (config: dict | None = None for Python 3.10+) or importing Optional from typing and using (config: Optional[dict] = None). Apply this same fix to any other parameters in the file that follow the same implicit None pattern.Source: Linters/SAST tools
87-90: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winChain exception context when re-raising.
Python best practice recommends using
raise ... from eto preserve the exception chain and make debugging easier. This helps distinguish between errors in the SDK call versus errors in exception handling itself.♻️ Proposed fix
except Exception as e: - raise Exception( - f"Error analyzing video with TwelveLabs: {type(e).__name__}: {str(e)}" - ) + raise Exception( + f"Error analyzing video with TwelveLabs: {type(e).__name__}: {e}" + ) from e🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/director/tools/twelvelabs_tool.py` around lines 87 - 90, In the exception handling block where the Exception is raised after catching the TwelveLabs SDK error, use the `raise ... from e` syntax instead of just `raise Exception(...)` to preserve the exception chain. This means changing the raise statement to include `from e` at the end, which will maintain the original exception context and make debugging easier by showing the full traceback chain between the original error and the new exception being raised.Source: Linters/SAST tools
111-114: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winChain exception context when re-raising.
Same best practice as the
analyze_videomethod—preserve the exception chain withfrom e.♻️ Proposed fix
except Exception as e: - raise Exception( - f"Error creating embedding with TwelveLabs: {type(e).__name__}: {str(e)}" - ) + raise Exception( + f"Error creating embedding with TwelveLabs: {type(e).__name__}: {e}" + ) from e🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/director/tools/twelvelabs_tool.py` around lines 111 - 114, Preserve the exception chain in the exception handler block where the Exception is re-raised with the error message containing type(e).__name__ and str(e). Add `from e` to the raise statement to maintain the original exception context, which improves debugging by showing the complete exception chain rather than just the error message details.Source: Linters/SAST tools
backend/director/agents/video_understanding.py (1)
68-68: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse explicit
Optionaltype hint for optional parameters.Consistent with the recommendation for
TwelveLabsTool, usetwelvelabs_config: dict | None = Nonefor PEP 484 compliance.♻️ Proposed fix
def run( self, collection_id: str, video_id: str, prompt: str, - twelvelabs_config: dict = None, + twelvelabs_config: dict | None = None, *args, **kwargs, ) -> AgentResponse:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/director/agents/video_understanding.py` at line 68, The type hint for the `twelvelabs_config` parameter is missing the explicit `Optional` annotation for the None type. Update the parameter declaration to use the union syntax `dict | None` instead of just `dict` to properly indicate that the parameter can accept either a dictionary or None value, ensuring PEP 484 compliance with the rest of the codebase.Source: Linters/SAST tools
backend/tests/test_twelvelabs_tool.py (1)
21-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider using a more specific exception type.
While the current implementation raises a generic
Exception, consider introducing a custom exception type (e.g.,TwelveLabsConfigError) in the future for better error handling and more precise test assertions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/tests/test_twelvelabs_tool.py` around lines 21 - 23, The test_requires_api_key function in test_twelvelabs_tool.py is catching a generic Exception type instead of a specific custom exception. Create a custom exception class named TwelveLabsConfigError to handle TwelveLabs configuration errors, update the TwelveLabsTool constructor to raise this specific exception when the api_key parameter is empty, and then update the pytest.raises() call in test_requires_api_key to expect TwelveLabsConfigError instead of the generic Exception type for more precise error handling.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/director/agents/video_understanding.py`:
- Around line 107-113: Replace the unsafe bracket notation for accessing the
download_url key in the download_response dictionary with the safer .get()
method. Change download_response["download_url"] to
download_response.get("download_url") to prevent potential KeyError exceptions
if the key is missing from the response, ensuring consistency with the defensive
programming pattern already used in other files like voice_replacement.py and
download.py when handling videodb responses.
---
Nitpick comments:
In `@backend/director/agents/video_understanding.py`:
- Line 68: The type hint for the `twelvelabs_config` parameter is missing the
explicit `Optional` annotation for the None type. Update the parameter
declaration to use the union syntax `dict | None` instead of just `dict` to
properly indicate that the parameter can accept either a dictionary or None
value, ensuring PEP 484 compliance with the rest of the codebase.
In `@backend/director/tools/twelvelabs_tool.py`:
- Line 93: Update the type hint for the config parameter in the
get_text_embedding method to use explicit Optional typing. Change the parameter
declaration from config: dict = None to config: Optional[dict] = None, following
PEP 484 conventions. Ensure that Optional is imported from the typing module at
the top of the file if it is not already imported.
- Line 63: The config parameter in twelvelabs_tool.py uses an implicit None type
hint (config: dict = None) which is not explicitly clear according to PEP 484.
Change this to use explicit Optional type hints by either using the union syntax
(config: dict | None = None for Python 3.10+) or importing Optional from typing
and using (config: Optional[dict] = None). Apply this same fix to any other
parameters in the file that follow the same implicit None pattern.
- Around line 87-90: In the exception handling block where the Exception is
raised after catching the TwelveLabs SDK error, use the `raise ... from e`
syntax instead of just `raise Exception(...)` to preserve the exception chain.
This means changing the raise statement to include `from e` at the end, which
will maintain the original exception context and make debugging easier by
showing the full traceback chain between the original error and the new
exception being raised.
- Around line 111-114: Preserve the exception chain in the exception handler
block where the Exception is re-raised with the error message containing
type(e).__name__ and str(e). Add `from e` to the raise statement to maintain the
original exception context, which improves debugging by showing the complete
exception chain rather than just the error message details.
In `@backend/tests/test_twelvelabs_tool.py`:
- Around line 21-23: The test_requires_api_key function in
test_twelvelabs_tool.py is catching a generic Exception type instead of a
specific custom exception. Create a custom exception class named
TwelveLabsConfigError to handle TwelveLabs configuration errors, update the
TwelveLabsTool constructor to raise this specific exception when the api_key
parameter is empty, and then update the pytest.raises() call in
test_requires_api_key to expect TwelveLabsConfigError instead of the generic
Exception type for more precise error handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
i️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bdd2599e-8f93-4551-b5e7-15308f65f2eb
📒 Files selected for processing (7)
README.mdbackend/.env.samplebackend/director/agents/video_understanding.pybackend/director/handler.pybackend/director/tools/twelvelabs_tool.pybackend/requirements.txtbackend/tests/test_twelvelabs_tool.py
- Use .get("download_url") with a None guard instead of bracket access,
matching the defensive pattern in download.py and voice_replacement.py
- Add explicit Optional[dict] type hints for optional config params (PEP 484)
- Chain re-raised exceptions with 'from e' to preserve the original traceback
mohit-twelvelabs
commented
Jun 24, 2026
Thanks for the review @coderabbitai! Addressed the feedback in e4f5926:
Actionable
video_understanding.py: replaceddownload_response["download_url"]with.get("download_url")plus a guard, matchingdownload.py/voice_replacement.py.
Nitpicks (applied)
- Explicit
Optional[dict]type hints onanalyze_video,get_text_embedding(twelvelabs_tool.py) andrun'stwelvelabs_config(video_understanding.py). UsedOptional[...]rather thanX | Noneto stay consistent with the neighbouring agents and the project'srequires-python = ">=3.6". - Chained re-raised exceptions with
... from ein bothtwelvelabs_tool.pyhandlers to preserve the original traceback.
Declined
- Custom
TwelveLabsConfigErrorexception type: leaving as the genericExceptionto match the rest oftwelvelabs_tool.pyand the agents in this repo (none define per-tool exception classes). Happy to introduce one if the maintainers want a project-wide convention.
All 5 unit tests pass locally, including the live Marengo embedding test.
Tip
For best results, initiate chat on the files or code changes.
@mohit-twelvelabs Thanks for the thorough follow-up — the guard around download_url, the Optional[...] annotations, and the exception chaining all sound good to me. The rationale for skipping a custom TwelveLabsConfigError in this PR also makes sense given the current repo conventions.
Great to hear all 5 tests are passing locally as well. 🐇✨
✏️ Learnings added
Learnt from: mohit-twelvelabs
URL: https://github.com/video-db/Director/pull/197
Timestamp: 2026年06月24日T22:58:56.552Z
Learning: In this repository's Python code under `backend/`, prefer `Optional[...]` type hints over `X | None` for consistency with neighboring agents and compatibility with the project's `requires-python = ">=3.6"` setting.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
If you found this review helpful, would you consider giving us a shout-out on X?
Thank you for using CodeRabbit!
Uh oh!
There was an error while loading. Please reload this page.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
A new opt-in
video_understandingagent that analyzes a VideoDB video with TwelveLabs Pegasus for true multimodal understanding — grounded in the actual footage (scenes, objects, on-screen actions) and audio, not just the spoken-word transcript. It complements the existingsummarize_videoagent, which reasons over the transcript only.It also adds a small
TwelveLabsToolwrapper (director/tools/twelvelabs_tool.py) exposing:analyze_video)get_text_embedding, 512-dim) — same embedding space as TwelveLabs video embeddings, useful for text-to-video retrieval.The agent resolves a downloadable URL from the VideoDB stream and lets TwelveLabs fetch it server-side, mirroring how the existing
download/generation agents work.Why it helps
Director already mixes multiple GenAI APIs per task. Pegasus gives the reasoning engine a way to watch a video — answering questions about visual content that a transcript can't (e.g. "what products appear?", "describe the b-roll"), which is a capability gap today.
Opt-in / non-breaking
TWELVELABS_API_KEYenv var. When it's unset the agent simply reports an error; no defaults change and no existing agent is touched.handler.pyandrequirements.txtthe same way existing tools/agents are; added to.env.sampleand the README feature list.How it was tested
backend/tests/test_twelvelabs_tool.py: no-network unit tests (SDK mocked) for the analyze/embed wiring, plus a live Marengo embedding test gated onTWELVELABS_API_KEY(skipped when unset).pytest→ 5 passed (including a live Marengo call returning a 512-dim vector against the real API).ruff checkandblack --checkclean on the changed files.director.handlerimports and the agent registers.twelvelabs==1.2.8(signatures introspected against the installed SDK).You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Summary by CodeRabbit