-
Notifications
You must be signed in to change notification settings - Fork 2.3k
feat(vertexai): add google_vertex_ai_online_evaluator resource#17944
feat(vertexai): add google_vertex_ai_online_evaluator resource #17944guvenenb wants to merge 3 commits into
Conversation
Googlers: For automatic test runs see go/terraform-auto-test-runs.
@roaks3, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.
You can help make sure that review is quick by doing a self-review and by running impacted tests locally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be great to create a simple reasoning engine here as well so the example is complete.
Also - did you test if it works and online evals are properly configured (in pantheon) ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I test with angelica's test project, it worked. I will do reasoning engine thing also
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be great to be explicit about metric types at least. If it's too much work let's just leave a todo and start working on it in a separate PR.
Image
Also, I haven't verified ai output but those are the predefined metric names available in pantheon, so would be good to include them in the comment so that users know what they can use.
final_response_quality_v1
tool_use_quality_v1
hallucination_v1
safety_v1
multi_turn_task_success_v1
multi_turn_tool_use_quality_v1
multi_turn_trajectory_quality_v1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's include other 6 predefined metrics you can find in pantheon to make the example better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you also show how to create custom code or llm based metric (available in pantheon) ?
I actually am not sure if your yaml allows this because custom metrics are a separate resource (projects/.../locations/.../metrics/...)
It might make sense to create a separate PR for that so it's easier to review and we can at least deliver something this week, even if it doesn't include code and llm metrics
- Rename location parameter to region for consistency. - Update format description in name property to use region. - Add timeouts (insert/update: 20m, delete: 60m) to OnlineEvaluator. - Specify OpAsync type for async operations. - Clean up comments in custom flatten template. TAG=agy CONV=5308268d-2e95-4a0c-8ce0-cdac27b8c645
- Use custom flattener for name to extract short ID. - Update self_link to use full path template with short name. - Update basic example to use region instead of location. TAG=agy CONV=5308268d-2e95-4a0c-8ce0-cdac27b8c645
Release Note Template for Downstream PRs (will be copied)
See Write release notes for guidance.