-
Notifications
You must be signed in to change notification settings - Fork 320
Added interface for reporting metrics #2887
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for this contribution @cccs-cat001! I left a few comments mostly around API design.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better to consolidate all information into a single message. This is because the four lines are not guaranteed to appear consecutively in the logs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Would it make sense to define and maintain a set of Micrometer metrics and increment them here?
- Would it make sense to fire a
PolarisEventhere?
(I don't really know what an IRC server is supposed to do with these scan or commit metrics 😅)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what those are 😅 my intentions were to have the default behaviour remain the same, and adding that may change the default behaviour
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😄 - no worries, we can look into that later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Iceberg already has a MetricsReporter and a LoggingMetricsReporter. I suppose you created new ones because you wanted to pass the warehouse and namespace names as well? In this case would it be possible to extend Iceberg's MetricsReporter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So that was my original plan! But the iceberg MetricsReporter is intended for client-side operations, so implementing it won't suffice for the /metrics endpoint reporting.
It would be ideal to change the name of it so that it doesn't get confused, but I wasn't sure of a better name 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Would something like this suit your needs?
public interface PolarisMetricsReporter extends MetricsReporter { void report(String catalog, TableIdentifier table, MetricsReport report); }
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the iceberg implementation is definitely not what we want to extend/implement here - https://iceberg.apache.org/docs/1.10.0/docs/metrics-reporting
it's a client side thing. but I can change the name of this interface to PolarisMetricsReporter to have that separation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- I would rather rename
prefixtowarehouseorcatalogNamesinceprefixis a REST-specific notion.- But you should to call
prefixParser.prefixToCatalogName(realmContext, prefix)to get the catalog name from the prefix.
- But you should to call
- For a richer API, I would use
TableIdentifierrather than 2 strings (namespace + table). It's very easy to get aTableIdentifier:Namespace ns = decodeNamespace(namespace); TableIdentifier tableIdentifier = TableIdentifier.of(ns, RESTUtil.decodeString(table));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot: it's probably better to pass MetricsReport here, not ReportMetricsRequest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are rather useless as they are basically testing SmallRye Config.
What would be more meaningful is a real test that invokes the IRC metrics endpoint and verifies that metrics were collected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we need a distinct logging reporter. The default reporter could log the metrics, then users could simply adjust their log levels and decide whether they want the metrics to appear in the logs or not (by default, they wouldn't be logged). WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your interest in Polaris and contributing, @cccs-cat001 !
I'm not sure how this PR relates to the /metrics endpoint in the Iceberg REST API, though 🤔 Could you clarify, please?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
config.type() is fixed in runtime... Can this bean be @ApplicationScoped?
I'm not sure how this PR relates to the
/metricsendpoint in the Iceberg REST API, though 🤔 Could you clarify, please?
If that can help: I was also puzzled initially, but we are actually talking about this IRC endpoint:
polaris/spec/iceberg-rest-catalog-open-api.yaml
Line 1288 in f7b5646
What changes were proposed in this pull request?
I'm adding an interface to be able to code for the
/metricsendpoint in the Iceberg REST API Spec.Why are the changes needed?
The metrics endpoint in Polaris is currently a no-op endpoint. In our organization we use it for auditing who is reading/writing tables, as this endpoint is called after every read/write with Iceberg. I added a default no-op implementation so that the behaviour remains the same, and I added a logging implementation to show how it could be used.
Does this PR introduce any user-facing change?
it adds an optional configuration for telling the metrics endpoint how it should behave. the default behaviour is no change, however if the user sets the
polaris.metrics.reporting.typetologgingit will log the MetricsReport to INFO. There is also the ability to implement the interface for your own needs if necessary.How was this patch tested?
Tests were added. it was also tested in a staging environment.
CHANGELOG.md