CLA assistant check
All committers have signed the CLA.

The general problem with this I think is cardinality: if users use data that is different on pretty much every request (e.g. the whole URL), then it will balloon the amount of data tracked in memory and returned in metrics responses.

Imagine it's configured with orig_uri and the app has URLs like /user/{user_id}/page/{page_id} or something, every one of those URLs would be uniquely tracked for metrics, meaning infinite cardinality. Very dangerous for performance.

One way to work around that would be to use vars in sites to only track specific path prefixes, doing something like vars /user* subpath "user-path" and vars /admin* subpath "admin-path" and such, then you could have a label subpath {vars.subpath} so it's controlled and you only have cardinality of 3 in that case (those two, plus "" for any other paths). Obviously, that can be expanded as needed for tracking more areas of an app.

Also the other problem is that since metrics are configured globally, if you do take that vars approach, you don't necessarily know that the vars will be set in all server routes, so you almost always end up with empty label values in some case.

So all that said, I think we need to be careful about what we recommend, the config in your PR description is absolutely dangerous as-is, so I think you should edit it (don't recommend remote or orig_uri at the very least, unbound cardinality). Also, our docs would certainly need an explainer on the risks of cardinality when it comes to labels and have certain recommendations to keep it in check.

Disclaimer: I'm saying this as someone who has literally never worked with metrics or prometheus etc, but I've absorbed some of this risk from talking a bunch with @hairyhenderson about this stuff.