-
Notifications
You must be signed in to change notification settings - Fork 0
[AI SDK] Removes input setting from v1 and adds test cases to v2 #46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AI SDK] Removes input setting from v1 and adds test cases to v2 #46
Conversation
- [AI SDK] Removes input setting from v1 and adds test cases to v2 #46 Graphite 👈 (View in Graphite)
main
This stack of pull requests is managed by Graphite. Learn more about stacking.
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. 📝 WalkthroughSummary by CodeRabbitRelease v6.19.0
WalkthroughRemoves trace.input recording for AI SDK v1, bumps package version to v6.19.0, adds a large AI SDK v2 test suite for Vercel/Maxim, and introduces an end-to-end City Prediction test script. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant setupLogging as wrapper.ts:setupLogging
participant Logger as MaximLogger
Note over Client,setupLogging: Before: setupLogging extracted\nuser message and called trace.input
Client->>setupLogging: call setupLogging(promptMessages, providerOptions)
note right of setupLogging: Extract user message removed\nNo trace.input() invocation
setupLogging->>Logger: return { maximMetadata, trace, session, span, promptMessages }
Logger-->>Client: metadata + trace objects (no recorded user input)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
af9bc12 to
77440b3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR removes automatic input setting from AI SDK v1 wrapper and adds comprehensive test coverage for v2 specification. The changes prevent v1 wrapper from automatically extracting and logging user input from messages, deferring this responsibility to the caller, while ensuring v2 functionality is thoroughly tested.
- Removed automatic trace input extraction logic from v1 wrapper
- Added comprehensive test suite for AI SDK v2 specification covering multiple providers and scenarios
- Updated version to 6.19.0 with changelog entry
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/lib/logger/vercel/wrapper.ts | Removes automatic input extraction from user messages in v1 wrapper |
| src/lib/tests/vercel/agent_testing.ts | Adds manual testing script demonstrating proper trace input usage |
| src/lib/logger/vercel/vercelLoggerV2.test.ts | Adds comprehensive test suite for v2 specification covering text generation, streaming, objects, tools, sessions, multi-modal, and error handling |
| package.json | Bumps version to 6.19.0 |
| README.md | Documents the changes in changelog |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
@coderabbitai
coderabbitai
bot
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
package.json (1)
23-54: Pin all dependencies (no carets allowed).Per coding guidelines, remove all leading ^ to avoid auto-updates. This ensures reproducible builds and CI stability.
Apply diffs (excerpted):
- "@ai-sdk/anthropic": "^2.0.9", + "@ai-sdk/anthropic": "2.0.9", - "@ai-sdk/google": "^2.0.23", + "@ai-sdk/google": "2.0.23", - "@ai-sdk/openai": "^2.0.23", + "@ai-sdk/openai": "2.0.23", - "@langchain/anthropic": "^0.3.21", + "@langchain/anthropic": "0.3.21", - "@langchain/community": "^0.3.46", + "@langchain/community": "0.3.46", - "@langchain/core": "^0.3.0", + "@langchain/core": "0.3.0", - "@langchain/langgraph": "^0.3.1", + "@langchain/langgraph": "0.3.1", - "@langchain/openai": "^0.5.12", + "@langchain/openai": "0.5.12", - "@openai/agents": "^0.1.9", + "@openai/agents": "0.1.9", - "@types/jest": "^29.5.14", + "@types/jest": "29.5.14", - "ai": "^5.0.72", + "ai": "5.0.72", - "ai-sdk-provider-v1": "npm:@ai-sdk/provider@^1.0.0", + "ai-sdk-provider-v1": "npm:@ai-sdk/provider@1.0.0", - "ai-sdk-provider-v2": "npm:@ai-sdk/provider@^2.0.0", + "ai-sdk-provider-v2": "npm:@ai-sdk/provider@2.0.0", - "dotenv": "^16.5.0", + "dotenv": "16.5.0", - "langchain": "^0.3.28", + "langchain": "0.3.28", - "rimraf": "^6.0.1", + "rimraf": "6.0.1", - "ts-jest": "^29.3.4", + "ts-jest": "29.3.4", - "ts-node": "^10.9.2", + "ts-node": "10.9.2", - "typedoc": "^0.28.5", + "typedoc": "0.28.5", - "typedoc-plugin-frontmatter": "^1.3.0", + "typedoc-plugin-frontmatter": "1.3.0", - "typedoc-plugin-markdown": "^4.6.4", + "typedoc-plugin-markdown": "4.6.4", - "zod": "^3.25.76" + "zod": "3.25.76"- "axios": "^1.11.0", + "axios": "1.11.0", - "axios-retry": "^4.5.0" + "axios-retry": "4.5.0"- "expo-crypto": "^13.0.0", + "expo-crypto": "13.0.0",As per coding guidelines.
Also applies to: 55-59, 105-107
🧹 Nitpick comments (4)
README.md (1)
526-529: Changelog entry OK; add a short privacy note.Clarify that v1 wrapper no longer auto-logs user prompts; apps must call trace.input() explicitly if they want inputs captured.
### v6.19.0 - fix: Removes trace input setting for AI SDK v1 - adds test cases for AI SDK v2 wrapper + - Note: AI SDK v1 models no longer auto-populate trace input. If you want to capture user prompts, call `trace.input(...)` explicitly.src/lib/tests/vercel/agent_testing.ts (2)
15-33: Prefer raiseExceptions during tests.Enable raiseExceptions for faster failure in CI.
- const maxim = new Maxim({ baseUrl, apiKey, debug: true }); + const maxim = new Maxim({ baseUrl, apiKey, debug: true, raiseExceptions: true });
103-115: Also log the final output to the trace.Optional: set trace.output(formattedOutput) before ending the trace to complete the I/O pair.
- const formattedOutput = await formatOutput(model, structuredData, trace); + const formattedOutput = await formatOutput(model, structuredData, trace); + trace.output(formattedOutput);src/lib/logger/vercel/vercelLoggerV2.test.ts (1)
23-30: Env var message consistency.Use the same name everywhere: MAXIM_LOG_REPO_ID.
- throw new Error("MAXIM_BASE_URL, MAXIM_API_KEY & LOG_REPO_ID environment variables are required"); + throw new Error("MAXIM_BASE_URL, MAXIM_API_KEY & MAXIM_LOG_REPO_ID environment variables are required");
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
README.md(1 hunks)package.json(1 hunks)src/lib/logger/vercel/vercelLoggerV2.test.ts(1 hunks)src/lib/logger/vercel/wrapper.ts(0 hunks)src/lib/tests/vercel/agent_testing.ts(1 hunks)
💤 Files with no reviewable changes (1)
- src/lib/logger/vercel/wrapper.ts
🧰 Additional context used
📓 Path-based instructions (1)
package.json
⚙️ CodeRabbit configuration file
do not allow any carats in package.json, we never want any auto updates for any patch versions of any packages in package.json
Files:
package.json
🧬 Code graph analysis (2)
src/lib/tests/vercel/agent_testing.ts (3)
src/lib/maxim.ts (2)
Maxim(149-1302)logger(1183-1214)src/lib/logger/components/trace.ts (1)
Trace(65-456)src/lib/logger/vercel/wrapper.ts (1)
wrapMaximAISDKModel(28-37)
src/lib/logger/vercel/vercelLoggerV2.test.ts (2)
src/lib/maxim.ts (2)
Maxim(149-1302)logger(1183-1214)src/lib/logger/vercel/wrapper.ts (1)
wrapMaximAISDKModel(28-37)
🪛 Biome (2.1.2)
src/lib/logger/vercel/vercelLoggerV2.test.ts
[error] 694-694: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.
The declaration is defined in this switch clause:
Safe fix: Wrap the declaration in a block.
(lint/correctness/noSwitchDeclarations)
[error] 695-695: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.
The declaration is defined in this switch clause:
Safe fix: Wrap the declaration in a block.
(lint/correctness/noSwitchDeclarations)
🔇 Additional comments (2)
package.json (1)
4-4: Version bump looks good.6.19.0 aligns with README changelog and new V2 tests.
src/lib/logger/vercel/vercelLoggerV2.test.ts (1)
36-56: Good V2 path smoke test setup.Wrapping the model and tagging with providerOptions is correct and aligns with wrapper semantics.
Please ensure OPENAI_API_KEY/ANTHROPIC_API_KEY are present in CI for these tests to run against live providers.
77440b3 to
9e747d3
Compare
@coderabbitai
coderabbitai
bot
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
♻️ Duplicate comments (2)
src/lib/logger/vercel/vercelLoggerV2.test.ts (1)
685-701: Wrap case-local declarations in 'median' to satisfy noSwitchDeclarations.Biome flags const declarations in switch without a block; scope them with braces.
- case "median": - const sorted = [...data].sort((a, b) => a - b); - const mid = Math.floor(sorted.length / 2); - return { - result: sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid], - type: "median", - dataPoints: data.length - }; + case "median": { + const sorted = [...data].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return { + result: sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid], + type: "median", + dataPoints: data.length, + }; + }src/lib/tests/vercel/agent_testing.ts (1)
95-123: Also cleanup the Maxim instance; destructure it from initializeMaxim.Prevents agent/socket leaks and ensures flush. This was flagged earlier as well.
- const { logger } = await initializeMaxim(); + const { maxim, logger } = await initializeMaxim(); try { ... } catch (error) { console.error("Error in city prediction demo:", error); } finally { await logger.cleanup(); + await maxim.cleanup(); }
🧹 Nitpick comments (2)
src/lib/logger/vercel/vercelLoggerV2.test.ts (2)
682-711: Enum has "mode" but switch lacks a handler. Add a mode case.Prevents default/error path for a declared option.
switch (analysisType) { case "mean": return { result: data.reduce((a, b) => a + b, 0) / data.length, type: "mean", dataPoints: data.length }; - case "median": { + case "median": { const sorted = [...data].sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2); return { result: sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid], type: "median", dataPoints: data.length }; } + case "mode": { + const counts = new Map<number, number>(); + for (const n of data) counts.set(n, (counts.get(n) ?? 0) + 1); + let mode = data[0]; + let freq = 0; + for (const [k, v] of counts) { + if (v > freq) { + mode = k; + freq = v; + } + } + return { result: mode, type: "mode", dataPoints: data.length }; + } case "range": return { result: Math.max(...data) - Math.min(...data), type: "range", min: Math.min(...data), max: Math.max(...data), dataPoints: data.length };
492-516: Prefer rejects assertion over try/catch + fail in error test.Cleaner and avoids false positives.
- try { - // Intentionally cause an error with invalid parameters - await generateText({ - model: model, - maxOutputTokens: -1, // Invalid token count - prompt: "This should cause an error.", - providerOptions: { - maxim: { - traceName: "V2 Error Handling Test", - generationName: "Invalid Parameters", - generationTags: { - test_type: "error_handling", - specification: "v2", - }, - } as MaximVercelProviderMetadata, - }, - }); - - // If we reach here, the test should fail - fail("Expected an error to be thrown"); - } catch (error) { - console.log("Expected error caught in V2 error handling test:", error); - expect(error).toBeDefined(); - } + await expect( + generateText({ + model: model, + maxOutputTokens: -1, + prompt: "This should cause an error.", + providerOptions: { + maxim: { + traceName: "V2 Error Handling Test", + generationName: "Invalid Parameters", + generationTags: { test_type: "error_handling", specification: "v2" }, + } as MaximVercelProviderMetadata, + }, + }) + ).rejects.toThrow();
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
README.md(1 hunks)package.json(1 hunks)src/lib/logger/vercel/vercelLoggerV2.test.ts(1 hunks)src/lib/logger/vercel/wrapper.ts(0 hunks)src/lib/tests/vercel/agent_testing.ts(1 hunks)
💤 Files with no reviewable changes (1)
- src/lib/logger/vercel/wrapper.ts
✅ Files skipped from review due to trivial changes (1)
- README.md
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
🧬 Code graph analysis (2)
src/lib/tests/vercel/agent_testing.ts (3)
src/lib/maxim.ts (2)
Maxim(149-1302)logger(1183-1214)src/lib/logger/components/trace.ts (1)
Trace(65-456)src/lib/logger/vercel/wrapper.ts (1)
wrapMaximAISDKModel(28-37)
src/lib/logger/vercel/vercelLoggerV2.test.ts (2)
src/lib/maxim.ts (2)
Maxim(149-1302)logger(1183-1214)src/lib/logger/vercel/wrapper.ts (1)
wrapMaximAISDKModel(28-37)
🪛 Biome (2.1.2)
src/lib/logger/vercel/vercelLoggerV2.test.ts
[error] 693-693: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.
The declaration is defined in this switch clause:
Safe fix: Wrap the declaration in a block.
(lint/correctness/noSwitchDeclarations)
[error] 694-694: Other switch clauses can erroneously access this declaration.
Wrap the declaration in a block to restrict its access to the switch clause.
The declaration is defined in this switch clause:
Safe fix: Wrap the declaration in a block.
(lint/correctness/noSwitchDeclarations)
🔇 Additional comments (1)
src/lib/logger/vercel/vercelLoggerV2.test.ts (1)
235-243: Verify trace existence when passing only a generated traceId.You set traceId = uuid() but don’t create a corresponding trace here. Confirm the wrapper/server creates or upserts traces by ID; otherwise, create a trace via logger.trace(...) and reuse its id.
No description provided.