[BUG] Profiles failing to load through email URL or search #194

nour commented 2024年02月18日 21:50:50 +01:00

What happened

Using the URL format https://keyoxide.org/name@domain.tld appears to no longer correctly resolve to the corresponding profile.

The following error page is displayed:

Something went wrong while viewing the profile
The following error was reported:
No public profile/keys could be found
Please see the documentation for help.

Using https://keyoxide.org/name%40domain.tld instead works.

Proposed solutions

Fix profile link parsing for the URL format https://keyoxide.org/ <email> or update the docs if this is deprecated and no longer supported.

Edit: It does not look like https://keyoxide.org/name%40domain.tld consistently works, and going to https://keyoxide.org instead and manually searching by email also fails.

### What happened Using the URL format https://keyoxide.org/name@domain.tld appears to no longer correctly resolve to the corresponding profile. The following error page is displayed: ``` Something went wrong while viewing the profile The following error was reported: No public profile/keys could be found Please see the documentation for help. ``` Using https://keyoxide.org/name%40domain.tld instead works. ### Proposed solutions Fix profile link parsing for the URL format https://keyoxide.org/`<email>` or update the docs if this is deprecated and no longer supported. **Edit:** It does not look like https://keyoxide.org/name%40domain.tld consistently works, and going to https://keyoxide.org instead and manually searching by email also fails.

vladimyr commented 2024年02月18日 23:07:06 +01:00

Thanks for your report. It certainly isn't deprecated so what we are dealing here with is a bug in implementation. If you don't mind could you share a failing query to help us debug it?

Thanks for your report. It certainly isn't deprecated so what we are dealing here with is a bug in implementation. If you don't mind could you share a failing query to help us debug it?

nour commented 2024年02月18日 23:52:45 +01:00

You can try with my profile: https://keyoxide.org/nour@agha.dev, otherwise accessible at https://keyoxide.org/C1859ED057E714DC.

To make sure this wasn't only occurring with my profile, I tried profile links of other users using email and encountered the same issue, apart from different browsers and devices.

Thank you for confirming this and looking into it!

You can try with my profile: https://keyoxide.org/nour@agha.dev, otherwise accessible at https://keyoxide.org/C1859ED057E714DC. To make sure this wasn't only occurring with my profile, I tried profile links of other users using email and encountered the same issue, apart from different browsers and devices. Thank you for confirming this and looking into it!

vladimyr commented 2024年02月19日 00:45:59 +01:00

Thanks for providing reproduction details but I've failed to reproduce it on my end 😕 Your profile loaded just fine using an email-like search query (inside multiple browsers) which makes me think that maybe there is something off with your network setup 🤔

Thanks for providing reproduction details but I've failed to reproduce it on my end 😕 Your profile loaded just fine using an email-like search query (inside multiple browsers) which makes me think that maybe there is something off with your network setup 🤔

tyy commented 2024年02月19日 01:53:46 +01:00

This is reproducable, but it is very inconsistent. I don't remember where I saw this but I believe it was thought to be a HKP ratelimiting issue before, which would explain the inconsistency.

Screenshot_20240218-175152.png

This is reproducable, but it is very inconsistent. I don't remember where I saw this but I believe it was thought to be a HKP ratelimiting issue before, which would explain the inconsistency. 

nour commented 2024年02月19日 02:19:17 +01:00

Thanks for providing reproduction details but I've failed to reproduce it on my end 😕 Your profile loaded just fine using an email-like search query (inside multiple browsers) which makes me think that maybe there is something off with your network setup 🤔

That is strange.. I had also made sure before to try different networks for extra measure, as well as changing DNS providers. Now it's suddenly working fine for me too.

@tyy 's post might explain it. I've also noticed something off now; after the email URL wasn't working correctly (which was verified on my Mastodon profile), I changed the Keyoxide link on my Mastodon profile (https://fosstodon.org/@Nour) to the fingerprint URL I provided earlier and it verified and displayed a checkmark with no issues. Now when I tried to change the Keyoxide link in Mastodon back to the email URL, it fails to verify. I tried refreshing/updating the link on Mastodon several times and changing back and forth between fingerprint URL and email URL, but the same issue - fingerprint URL verifies instantly, but email URL doesn't. It's as if Mastodon might be having issues accessing the email URL.

I can no longer reproduce it in my browser, but if I do I'll try digging into the console to check for any errors.

> Thanks for providing reproduction details but I've failed to reproduce it on my end 😕 Your profile loaded just fine using an email-like search query (inside multiple browsers) which makes me think that maybe there is something off with your network setup 🤔 That is strange.. I had also made sure before to try different networks for extra measure, as well as changing DNS providers. Now it's suddenly working fine for me too. @tyy 's post might explain it. I've also noticed something off now; after the email URL wasn't working correctly (which was verified on my Mastodon profile), I changed the Keyoxide link on my Mastodon profile (https://fosstodon.org/@Nour) to the fingerprint URL I provided earlier and it verified and displayed a checkmark with no issues. Now when I tried to change the Keyoxide link in Mastodon back to the email URL, it fails to verify. I tried refreshing/updating the link on Mastodon several times and changing back and forth between fingerprint URL and email URL, but the same issue - fingerprint URL verifies instantly, but email URL doesn't. It's as if Mastodon might be having issues accessing the email URL. I can no longer reproduce it in my browser, but if I do I'll try digging into the console to check for any errors.

vladimyr commented 2024年02月19日 02:57:10 +01:00

I tried refreshing/updating the link on Mastodon several times and changing back and forth between fingerprint URL and email URL, but the same issue - fingerprint URL verifies instantly, but email URL doesn't. It's as if Mastodon might be having issues accessing the email URL.

And that could be very much true. Since your key lives on keys.openpgp.org keyserver which is powered by Hagrid keyserver let me walk through what actually happens when you issue an email-like search query:

1. You land to /pks/lookup route - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L102
2. Next step is internal /pks/internal/get/(.+(?:%40|@).+)$ route (that regex just ensures that param is email-like aka contains @ character or %40 HTML escape - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L157
3. And finally you end up being redirected to /vks/v1/by-email which is part of Hagrid's custom Verifying Keyserver Interface - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L69

In contrast, when you search for the key by fingerprint after you've done the whole PKS to VKS dance, you'll land here: https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L44 and key will be served directly by nginx.

If you paid close attention you may have noticed two things:

1. when you do fingerprint-based lookup key is returned by nginx directly while email-like searches end up being served by Hagrid which is OFC much slower
2. email-like searches have way stricter rate limits (https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L61-62) than straight fingerprint-based lookups (https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L36)

And although all that applies exclusively to Hagrid (and keys.openpgp.org) as a general rule of thumb email-like searches shouldn't be used for linking to keys. To give you an analogy it is like linking to a Google (or any other search engine) search query instead of linking to the result directly. The standard and most reliable way of linking keys is through openpgp4fpr: URIs and as you can see from your own Mastodon example it really makes a difference.

Long story short: always use openpgp4fpr: URIs in one of 4 possible forms:

1. direct - openpgp4fpr:135f691534c8919295ed97c1c1859ed057e714dc
2. linked through Keyoxide - https://keyoxide.org/openpgp4fpr:135f691534c8919295ed97c1c1859ed057e714dc
3. hashed
4. redirected to using alias URL

> I tried refreshing/updating the link on Mastodon several times and changing back and forth between fingerprint URL and email URL, but the same issue - fingerprint URL verifies instantly, but email URL doesn't. It's as if Mastodon might be having issues accessing the email URL. And that could be very much true. Since your key lives on keys.openpgp.org keyserver which is powered by Hagrid keyserver let me walk through what actually happens when you issue an email-like search query: 1. You land to `/pks/lookup` route - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L102 2. Next step is internal `/pks/internal/get/(.+(?:%40|@).+)$` route (that regex just ensures that param is _email-like_ aka contains `@` character or `%40` HTML escape - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L157 3. And finally you end up being redirected to `/vks/v1/by-email` which is part of Hagrid's custom [Verifying Keyserver Interface](https://keys.openpgp.org/about/api) - https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L69 In contrast, when you _search_ for the key by fingerprint after you've done the whole PKS to VKS dance, you'll land here: https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L44 and key will be served directly by nginx. If you paid close attention you may have noticed two things: 1. when you do fingerprint-based lookup key is returned by nginx directly while email-like searches end up being served by Hagrid which is OFC much slower 2. email-like searches have way stricter rate limits (https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L61-62) than straight fingerprint-based lookups (https://gitlab.com/keys.openpgp.org/hagrid/-/blob/master/hagrid-routes.conf#L36) And although all that applies exclusively to Hagrid (and keys.openpgp.org) as a general rule of thumb email-like searches shouldn't be used for linking to keys. To give you an analogy it is like linking to a Google (or any other search engine) search query instead of linking to the result directly. The standard and most reliable way of linking keys is through openpgp4fpr: URIs and as you can see from your own Mastodon example it really makes a difference. Long story short: always use openpgp4fpr: URIs in one of 4 possible forms: 1. direct - openpgp4fpr:135f691534c8919295ed97c1c1859ed057e714dc 2. linked through Keyoxide - https://keyoxide.org/openpgp4fpr:135f691534c8919295ed97c1c1859ed057e714dc 3. hashed 4. redirected to using alias URL

nour commented 2024年02月19日 03:28:29 +01:00

Thank you for the detailed information. I did indeed feel like email is not very reliable after this and switched to using a shortened fingerprint URL which seems to strike a good balance between link length and speed. I'll have to update my claims/proofs.

I also have a custom redirect using my domain (https://keyoxide.agha.dev), so when it comes to profile linking it's not an issue overall, especially when it comes to using email. I'd always wondered if there were a difference between using email or other URIs and this clearly explains it.

Out of curiosity, would it be feasible to implement usernames that can be claimed/registered by users for their profiles, similar to how Keybase works? It would make things a lot more user friendly for many people, but if this has already been considered before and a decision was made against implementing it, I'm sure there were good reasons for it.

Thank you for the detailed information. I did indeed feel like email is not very reliable after this and switched to using a shortened fingerprint URL which seems to strike a good balance between link length and speed. I'll have to update my claims/proofs. I also have a custom redirect using my domain (https://keyoxide.agha.dev), so when it comes to profile linking it's not an issue overall, especially when it comes to using email. I'd always wondered if there were a difference between using email or other URIs and this clearly explains it. Out of curiosity, would it be feasible to implement usernames that can be claimed/registered by users for their profiles, similar to how Keybase works? It would make things a lot more user friendly for many people, but if this has already been considered before and a decision was made against implementing it, I'm sure there were good reasons for it.

tyy commented 2024年02月19日 03:48:30 +01:00

The main reason that isn't a thing already is just because keyoxide-web aims to be stateless, in that all data is fetched and parsed on the go rather than stored in a database specific to keyoxide. Something very similar to what you are talking about however is https://kxid.me/, though it isn't used very often and requires contacting yarmo directly.

The main reason that isn't a thing already is just because keyoxide-web aims to be stateless, in that all data is fetched and parsed on the go rather than stored in a database specific to keyoxide. Something very similar to what you are talking about however is https://kxid.me/, though it isn't used very often and requires contacting yarmo directly.

nour commented 2024年02月19日 04:04:21 +01:00

The main reason that isn't a thing already is just because keyoxide-web aims to be stateless, in that all data is fetched and parsed on the go rather than stored in a database specific to keyoxide. Something very similar to what you are talking about however is https://kxid.me/, though it isn't used very often and requires contacting yarmo directly.

Thanks, that's what I thought and I remember seeing this before.

> The main reason that isn't a thing already is just because keyoxide-web aims to be stateless, in that all data is fetched and parsed on the go rather than stored in a database specific to keyoxide. Something very similar to what you are talking about however is https://kxid.me/, though it isn't used very often and requires contacting yarmo directly. Thanks, that's what I thought and I remember seeing this before.

vladimyr commented 2024年02月19日 04:08:09 +01:00

Thank you for the detailed information. I did indeed feel like email is not very reliable after this and switched to using a shortened fingerprint URL which seems to strike a good balance between link length and speed.

PGP key fingerprints have length of 160 bits which when encoded using base16 (2^4) results in 40 hexa characters. Those 40 chars are typically printed as 10 groups of 4 chars. The last 4 groups containing 16 hexa characters are called key ID and they hold the last 64 bits of the key's fingerprint. Using your key as an example:

• PGP key URI: openpgp4fpr:135F691534C8919295ED97C1C1859ED057E714DC
• Key fingerprint: 135F 6915 34C8 9192 95ED 97C1 C185 9ED0 57E7 14DC (160 bits, 10x4 hexa chars)
• Key ID: C185 9ED0 57E7 14DC (64 bits, last 4x4 hexa chars)

The problem with using key ID for linking is that you increase the risk of collision. ⚠️ Back in the day people used to use even shorter identifiers:

• Short key ID: 57E7 14DC (32 bits, last 2x4 hexa chars)

and while it arguably results in short and good-looking identifiers it is dangerously unsafe --> https://evil32.com/
So my advice to you is simply don't do that. If you need a short link hide the fingerprint behind an alias because each time you trim some bits in the name of elegance you are actually making a nasty security tradeoff that could be exploited by 3rd party producing collision with your shortened identifier and thus redirecting your Keyoxide profile.

That said, however, I do have a custom redirect using my domain (https://keyoxide.agha.dev), so it's not an issue overall, especially when it comes to using email. I'd always wondered if there were a difference between using email or other URIs and this clearly explains it.

The difference is simple:

1. fingerprint --> simple key-value lookup
2. anything else --> search with potentially long execution time and possible collisions

Out of curiosity, would it be feasible to implement usernames that can be claimed/registered by users for their profiles, similar to how Keybase works? It would make things a lot more user friendly for many people, but if this has already been considered before and a decision was made against implementing it, I'm sure there were good reasons for it.

The answer to that question is both yes and no 🙃 I wholeheartedly agree that it makes the whole thing more user-friendly for a lot of folks out there but at the same time, it goes against core Keyoxide principles. However, there is always an option of using an auxiliary service that would provide alias URLs. Such service has already been implemented as PoC and might be revisited in future: https://kxid.me/

> Thank you for the detailed information. I did indeed feel like email is not very reliable after this and switched to using a shortened fingerprint URL which seems to strike a good balance between link length and speed. PGP key fingerprints have length of 160 bits which when encoded using base16 (2^4) results in 40 hexa characters. Those 40 chars are typically printed as 10 groups of 4 chars. The last 4 groups containing 16 hexa characters are called key ID and they hold the last 64 bits of the key's fingerprint. Using your key as an example: - PGP key URI: openpgp4fpr:135F691534C8919295ED97C1C1859ED057E714DC - Key fingerprint: 135F 6915 34C8 9192 95ED 97C1 C185 9ED0 57E7 14DC (160 bits, 10x4 hexa chars) - Key ID: C185 9ED0 57E7 14DC (64 bits, last 4x4 hexa chars) The problem with using key ID for linking is that you increase the risk of collision. ⚠️ Back in the day people used to use even shorter identifiers: - Short key ID: 57E7 14DC (32 bits, last 2x4 hexa chars) and while it arguably results in short and good-looking identifiers it is dangerously unsafe --> https://evil32.com/ So my advice to you is simply don't do that. If you need a short link _hide_ the fingerprint behind an alias because each time you trim some bits in the name of elegance you are actually making a nasty security tradeoff that could be exploited by 3rd party producing collision with your shortened identifier and thus redirecting your Keyoxide profile. > That said, however, I do have a custom redirect using my domain (https://keyoxide.agha.dev), so it's not an issue overall, especially when it comes to using email. I'd always wondered if there were a difference between using email or other URIs and this clearly explains it. The difference is simple: 1. fingerprint --> simple key-value lookup 2. anything else --> search with potentially long execution time and possible collisions > Out of curiosity, would it be feasible to implement usernames that can be claimed/registered by users for their profiles, similar to how Keybase works? It would make things a lot more user friendly for many people, but if this has already been considered before and a decision was made against implementing it, I'm sure there were good reasons for it. The answer to that question is both yes and no 🙃 I wholeheartedly agree that it makes the whole thing more user-friendly for a lot of folks out there but at the same time, it goes against core Keyoxide principles. However, there is always an option of using an auxiliary service that would provide alias URLs. Such service has already been implemented as PoC and might be revisited in future: https://kxid.me/

nour commented 2024年02月19日 04:32:54 +01:00

I agree with this. I wasn't aware that there was a serious possibility for a collision, so will definitely opt for the fingerprint rather than the key ID for sharing.

An alias URL is beneficial for profile sharing, but when it comes to proof/claim linking it's definitely best to rely directly on the fingerprint.

I agree with this. I wasn't aware that there was a serious possibility for a collision, so will definitely opt for the fingerprint rather than the key ID for sharing. An alias URL is beneficial for profile sharing, but when it comes to proof/claim linking it's definitely best to rely directly on the fingerprint.