Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Added interface for reporting metrics #2887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cccs-cat001 wants to merge 2 commits into apache:main
base: main
Choose a base branch
Loading
from cccs-cat001:main

Conversation

@cccs-cat001
Copy link

@cccs-cat001 cccs-cat001 commented Oct 24, 2025

What changes were proposed in this pull request?

I'm adding an interface to be able to code for the /metrics endpoint in the Iceberg REST API Spec.

Why are the changes needed?

The metrics endpoint in Polaris is currently a no-op endpoint. In our organization we use it for auditing who is reading/writing tables, as this endpoint is called after every read/write with Iceberg. I added a default no-op implementation so that the behaviour remains the same, and I added a logging implementation to show how it could be used.

Does this PR introduce any user-facing change?

it adds an optional configuration for telling the metrics endpoint how it should behave. the default behaviour is no change, however if the user sets the polaris.metrics.reporting.type to logging it will log the MetricsReport to INFO. There is also the ability to implement the interface for your own needs if necessary.

How was this patch tested?

Tests were added. it was also tested in a staging environment.

CHANGELOG.md

Copy link
Contributor

@adutra adutra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this contribution @cccs-cat001! I left a few comments mostly around API design.

@Override
public void reportMetric(
String prefix, String namespace, String table, ReportMetricsRequest reportMetricsRequest) {
LOGGER.info(prefix);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to consolidate all information into a single message. This is because the four lines are not guaranteed to appear consecutively in the logs.

@Override
public void reportMetric(
String prefix, String namespace, String table, ReportMetricsRequest reportMetricsRequest) {
// Do Nothing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Would it make sense to define and maintain a set of Micrometer metrics and increment them here?
  2. Would it make sense to fire a PolarisEvent here?

(I don't really know what an IRC server is supposed to do with these scan or commit metrics 😅)

Copy link
Author

@cccs-cat001 cccs-cat001 Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what those are 😅 my intentions were to have the default behaviour remain the same, and adding that may change the default behaviour

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😄 - no worries, we can look into that later.


import org.apache.iceberg.rest.requests.ReportMetricsRequest;

public interface MetricsReporter {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iceberg already has a MetricsReporter and a LoggingMetricsReporter. I suppose you created new ones because you wanted to pass the warehouse and namespace names as well? In this case would it be possible to extend Iceberg's MetricsReporter?

Copy link
Author

@cccs-cat001 cccs-cat001 Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that was my original plan! But the iceberg MetricsReporter is intended for client-side operations, so implementing it won't suffice for the /metrics endpoint reporting.

It would be ideal to change the name of it so that it doesn't get confused, but I wasn't sure of a better name 😅

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Would something like this suit your needs?

public interface PolarisMetricsReporter extends MetricsReporter {
 void report(String catalog, TableIdentifier table, MetricsReport report);
}

Copy link
Author

@cccs-cat001 cccs-cat001 Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the iceberg implementation is definitely not what we want to extend/implement here - https://iceberg.apache.org/docs/1.10.0/docs/metrics-reporting

it's a client side thing. but I can change the name of this interface to PolarisMetricsReporter to have that separation.


public interface MetricsReporter {
public void reportMetric(
String prefix, String namespace, String table, ReportMetricsRequest reportMetricsRequest);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I would rather rename prefix to warehouse or catalogName since prefix is a REST-specific notion.
    1. But you should to call prefixParser.prefixToCatalogName(realmContext, prefix) to get the catalog name from the prefix.
  2. For a richer API, I would use TableIdentifier rather than 2 strings (namespace + table). It's very easy to get a TableIdentifier:
    Namespace ns = decodeNamespace(namespace);
    TableIdentifier tableIdentifier = TableIdentifier.of(ns, RESTUtil.decodeString(table));

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot: it's probably better to pass MetricsReport here, not ReportMetricsRequest.


@Test
@TestProfile(MetricsReportingConfiguration.DefaultProfile.class)
void testDefault() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are rather useless as they are basically testing SmallRye Config.

What would be more meaningful is a real test that invokes the IRC metrics endpoint and verifies that metrics were collected.

private static final Logger LOGGER = LoggerFactory.getLogger(LoggingMetricsReporter.class);

@Override
public void reportMetric(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need a distinct logging reporter. The default reporter could log the metrics, then users could simply adjust their log levels and decide whether they want the metrics to appear in the logs or not (by default, they wouldn't be logged). WDYT?

Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your interest in Polaris and contributing, @cccs-cat001 !

I'm not sure how this PR relates to the /metrics endpoint in the Iceberg REST API, though 🤔 Could you clarify, please?

@RequestScoped
public MetricsReporter metricsReporter(
MetricsReportingConfiguration config, @Any Instance<MetricsReporter> reporters) {
return reporters.select(Identifier.Literal.of(config.type())).get();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

config.type() is fixed in runtime... Can this bean be @ApplicationScoped?

Copy link
Contributor

adutra commented Oct 24, 2025

I'm not sure how this PR relates to the /metrics endpoint in the Iceberg REST API, though 🤔 Could you clarify, please?

If that can help: I was also puzzled initially, but we are actually talking about this IRC endpoint:

/v1/{prefix}/namespaces/{namespace}/tables/{table}/metrics:
dimas-b and cccs-cat001 reacted with thumbs up emoji

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adutra adutra adutra left review comments

@dimas-b dimas-b dimas-b left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /