Codeberg/Community
54
325
Fork
You've already forked Community
12

codeberg.org TLS server sometimes serves *.codeberg.page certificate #1368

Closed
opened 2023年12月12日 20:27:53 +01:00 by neuschaefer · 13 comments

Comment

I am debugging a strange issue, and I'm not quite sure whether it's a client or server problem:

  • On a specific machine (armv5) git fetch https://codeberg.org/... sometimes gets the server certificate for *.codeberg.page, and then the TLS client library (GnuTLS) complains about it and the connection isn't established (verbose log here)
  • DNS doesn't seem to be the issue here, the IP address is always resolved the same (either 217.197.91.145 or 2001:67c:1401:20f0::1)
  • The error seems like it would be SNI-related but SNI looks fine in the packet capture, as far as I can tell
  • The likelihood of the error is close to 100% inside a podman container, and closer to (削除) 10% (削除ここまで) 25% outside
  • (削除) I don't know whether the error is triggered by GnuTLS's randomization of the order of TLS extensions (削除ここまで) Data analysis shows that this seems to be independent of the order of TLS extensions added by GnuTLS

I'd be happy to test suggestions that might bring us closer to the root cause, because it's quite a strange issue.

### Comment I am debugging a strange issue, and I'm not quite sure whether it's a client or server problem: - On a specific machine (armv5) `git fetch https://codeberg.org/...` sometimes gets the server certificate for `*.codeberg.page`, and then the TLS client library (GnuTLS) complains about it and the connection isn't established ([verbose log here](https://codeberg.org/neuschaefer/git-host-bug/src/branch/main/fail-verbose.txt)) - DNS doesn't seem to be the issue here, the IP address is always resolved the same (either 217.197.91.145 or 2001:67c:1401:20f0::1) - The error seems like it would be SNI-related but SNI looks fine in the [packet capture](https://codeberg.org/neuschaefer/git-host-bug/src/branch/main/codeberg.pcap), as far as I can tell - The likelihood of the error is close to 100% inside a podman container, and closer to ~~10%~~ 25% outside - ~~I don't know whether the error is triggered by GnuTLS's randomization of the order of TLS extensions~~ [Data analysis](https://codeberg.org/neuschaefer/git-host-bug#data-analysis) shows that this seems to be independent of the order of TLS extensions added by GnuTLS I'd be happy to test suggestions that might bring us closer to the root cause, because it's quite a strange issue.

Could you try if you can reproduce this failure with gnutls-cli -V codeberg.org:443? If that's reproducible could you then try to reproduce it via openssl s_client -connect codeberg.org:443 (requires OpenSSL). This should hopefully give a clue where to look, either somewhere in Git or in GnuTLS or at Codeberg's HAProxy setup.

Could you try if you can reproduce this failure with `gnutls-cli -V codeberg.org:443`? If that's reproducible could you then try to reproduce it via `openssl s_client -connect codeberg.org:443` (requires OpenSSL). This should hopefully give a clue where to look, either somewhere in Git or in GnuTLS or at Codeberg's HAProxy setup.

So far I haven't been able to reproduce it with gnutls-cli or with openssl.

So far I haven't been able to reproduce it with `gnutls-cli` or with `openssl`.
Owner
Copy link

If this is reproducible reliably, can you also reproduce this on another machine with similar setup? In the end, can you somehow identify which part of the setup causes the problem?

It's not impossible that we have a bug on our end, but it would be good to know if there is a specific reason for it.

Our current HAProxy setup serves a codeberg.org certificate when the hostname matches a specific pattern - and forwards every unknown domain to codeberg.page. So I suppose that the domain is technically unknown, either due to a missing, corrupted or wrong SNI.

If this is reproducible reliably, can you also reproduce this on another machine with similar setup? In the end, can you somehow identify which part of the setup causes the problem? It's not impossible that we have a bug on our end, but it would be good to know if there is a specific reason for it. Our current HAProxy setup serves a codeberg.org certificate when the hostname matches a specific pattern - and forwards every unknown domain to codeberg.page. So I suppose that the domain is technically unknown, either due to a missing, corrupted or wrong SNI.
Member
Copy link

Yes, if the SNI is not exactly what HAProxy is expecting, your packets get proxied to pages-server because HAProxy can't terminate encryption for custom domains.

Yes, if the SNI is not exactly what HAProxy is expecting, your packets get proxied to pages-server because HAProxy can't terminate encryption for custom domains.
fnetX removed their assignment 2024年04月16日 19:35:55 +02:00
Owner
Copy link

There are currently no actionable items. I am fine with closing this issue, unless more information becomes available.

There are currently no actionable items. I am fine with closing this issue, unless more information becomes available.

There are currently no actionable items. I am fine with closing this issue, unless more information becomes available.

fair enough. if i learn something new and actionable, i'll reopen the issue. i currently have no timeline for this.

> There are currently no actionable items. I am fine with closing this issue, unless more information becomes available. fair enough. if i learn something new and actionable, i'll reopen the issue. i currently have no timeline for this.

I'm now experiencing this while trying to push docker images.

Put "https://codeberg.org/v2/neuner/x/x/blobs/uploads/jspycm3oz7w8e9wn2oxl1tihu?digest=sha256%3Ae154b5bd03645894cd6b1dca4d0db3b788c12d325f2a8b05be491aa8646b0766": tls: failed to verify certificate: x509: certificate is valid for *.codeberg.page, codeberg.page, not codeberg.org

I'll take a look at my environment, but this is pretty odd...

I'm now experiencing this while trying to push docker images. ``` Put "https://codeberg.org/v2/neuner/x/x/blobs/uploads/jspycm3oz7w8e9wn2oxl1tihu?digest=sha256%3Ae154b5bd03645894cd6b1dca4d0db3b788c12d325f2a8b05be491aa8646b0766": tls: failed to verify certificate: x509: certificate is valid for *.codeberg.page, codeberg.page, not codeberg.org ``` I'll take a look at my environment, but this is pretty odd...
Owner
Copy link

We never managed to reproduce it, but it gets reported every few months or so. It is likely an issue with our setup, but we don't know how to reproduce or which component/config to blame.

We never managed to reproduce it, but it gets reported every few months or so. It is likely an issue with our setup, but we don't know how to reproduce or which component/config to blame.

yeah, I assume it's something to do with codeberg's proxy responding incorrectly to the first request it gets.

regardless, the fix for it is to just try it again. about a minute later it resolved itself

yeah, I assume it's something to do with codeberg's proxy responding incorrectly to the first request it gets. regardless, the fix for it is to just try it again. about a minute later it resolved itself

I'm more than willing to help troubleshoot or look at configurations (I do have some experience with haproxy). I can reproduce this regularly from my end:
Also this isn't a matter of just waiting an it resolves itself, it may take 10-20 attempts to get a proper response from the git side.
If it were just a matter of me trying over and over with a git clone command that's one thing I guess, but when the clone is part of a larger installation package which is out of my control it makes for a very frustrating experience. At any rate, closing this because you can't solve it seems the wrong move and sends a message that it's unimportant. This should remain open until a fix is found.
Just my $.02. Again I'm willing to help in any way that I can.

Without knowing what the full architecture looks like, it seems that what is happening is that the haproxy is doing a round-robin load balancing to a pool of servers but the servers are specialized so that some are only doing pages and others are only doing git, but the load balancer isn't inspecting the request to determine which pool to forward the connection to, so sometimes you get a pages server and sometimes you get a git server. I'm not certain why this doesn't appear to be an issue in the browser but only on the command line, I haven't been able to reproduce this behavior in the browser.

Here are the outputs I get for both a successful attempt and failed attempt:

Failure:

openssl s_client -showcerts -connect codeberg.org:443
Connecting to 217.197.91.145
CONNECTED(00000005)
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R10
verify return:1
depth=0 CN=*.codeberg.page
verify return:1
---
Certificate chain
 0 s:CN=*.codeberg.page
 i:C=US, O=Let's Encrypt, CN=R10
 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
 v:NotBefore: Sep 13 04:06:34 2024 GMT; NotAfter: Dec 12 04:06:33 2024 GMT
-----BEGIN CERTIFICATE-----
MIIE/TCCA+WgAwIBAgISBE41DOxX9tvwZZi/byTCJ0z3MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTAwHhcNMjQwOTEzMDQwNjM0WhcNMjQxMjEyMDQwNjMzWjAaMRgwFgYDVQQD
DA8qLmNvZGViZXJnLnBhZ2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQC0JYdtNveSPU1kN/f0SjNL6/ZgWXOqCj4PrhSjxsQClFOqKvdEVMV2KjgL3A7P
LSWxdok18AlAcxmP/UNo3YbX0lBa0HxPnjqT+Ned2hDJAhlXi2j0sUemHoAK6a7q
lE3Bl4lJMGRlgnKvLRKl9E7jRLyEuGHFPQZ0bFDnxr5J7tS1DCzo+boHS86NYsBQ
0IBpq1wAMOqswzpNQC/4o0T8y1zK52B00AEbjEWTFNJJHhPiIp81PNqnZTAUUQQa
IDR1s+YsjoTDAmzC9dv1Wq+94K5J37WX0Tt4WFquaTzSR2OqaMcUZBl85LIcCqzF
y9+XtHGCUvP5fhdgR5ooZ1a7AgMBAAGjggIiMIICHjAOBgNVHQ8BAf8EBAMCBaAw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD
VR0OBBYEFEmRhcGRheHwxTaoLpeBKbgx8IVyMB8GA1UdIwQYMBaAFLu8w0el5Lyp
xsOkcgwQjaI14cjoMFcGCCsGAQUFBwEBBEswSTAiBggrBgEFBQcwAYYWaHR0cDov
L3IxMC5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYXaHR0cDovL3IxMC5pLmxlbmNy
Lm9yZy8wKQYDVR0RBCIwIIIPKi5jb2RlYmVyZy5wYWdlgg1jb2RlYmVyZy5wYWdl
MBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHYA
GZgQcQnw1lIuMIDSnj9ku4NuKMz5D1KO7t/OSj8WtMoAAAGR6cVCFwAABAMARzBF
AiEAmhjzPZi5n+OaxjhTK0pAzhMILgAE3QpzYcnvqZwVAzUCIEFyhAuZHFSJWhz9
ukfAfoB/Rz1YzIrqtMcP8YpJbNJJAHYAdv+IPwq2+5VRwmHM9Ye6NLSkzbsp3GhC
Cp/mZ0xaOnQAAAGR6cVCIgAABAMARzBFAiEAvzE/BNeFCS+sAsb3amN6dUYOJoAj
QDx+UQIbT0RDcV0CICuR8wel4gEJiRJcL8uoeCClFbztEfZqxjzGN6mPOQgwMA0G
CSqGSIb3DQEBCwUAA4IBAQAbbyGECdDcair6brWj1Fxuh8f0NTaOaQQxpfYVz57G
QScYoLxHWIaxiwFLUaJS/ojq4qW6mZGEwKnmhUBfh024jB9BsSF8A30BeWSeKE7l
/1DY5dV1KFH/NDnJB31i1DaCJAhAXVpj89GL8IXdETzsieQla0//IYoc8vCdsKpD
rktULWx30zcRja6GhP6ikNzpUx0wIQwTtIjlXFcRZj9o9soLm2jG3JH5yb+rkf24
+oyvSFhXRlF++/lz5X8Law9M5ZZVkK1UAicPLnarGp6nmDMP1eeAxXh5XUw2qIBK
FHu4COLvaXLwUqo4qh145O4acZuFimn73SYNORTemfK5
-----END CERTIFICATE-----
 1 s:C=US, O=Let's Encrypt, CN=R10
 i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
 v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
-----BEGIN CERTIFICATE-----
MIIFBTCCAu2gAwIBAgIQS6hSk/eaL6JzBkuoBI110DANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFy
Y2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMTAeFw0yNDAzMTMwMDAwMDBa
Fw0yNzAzMTIyMzU5NTlaMDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBF
bmNyeXB0MQwwCgYDVQQDEwNSMTAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDPV+XmxFQS7bRH/sknWHZGUCiMHT6I3wWd1bUYKb3dtVq/+vbOo76vACFL
YlpaPAEvxVgD9on/jhFD68G14BQHlo9vH9fnuoE5CXVlt8KvGFs3Jijno/QHK20a
/6tYvJWuQP/py1fEtVt/eA0YYbwX51TGu0mRzW4Y0YCF7qZlNrx06rxQTOr8IfM4
FpOUurDTazgGzRYSespSdcitdrLCnF2YRVxvYXvGLe48E1KGAdlX5jgc3421H5KR
mudKHMxFqHJV8LDmowfs/acbZp4/SItxhHFYyTr6717yW0QrPHTnj7JHwQdqzZq3
DZb3EoEmUVQK7GH29/Xi8orIlQ2NAgMBAAGjgfgwgfUwDgYDVR0PAQH/BAQDAgGG
MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHRMBAf8ECDAGAQH/
AgEAMB0GA1UdDgQWBBS7vMNHpeS8qcbDpHIMEI2iNeHI6DAfBgNVHSMEGDAWgBR5
tFnme7bl5AFzgAiIyBpY9umbbjAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAKG
Fmh0dHA6Ly94MS5pLmxlbmNyLm9yZy8wEwYDVR0gBAwwCjAIBgZngQwBAgEwJwYD
VR0fBCAwHjAcoBqgGIYWaHR0cDovL3gxLmMubGVuY3Iub3JnLzANBgkqhkiG9w0B
AQsFAAOCAgEAkrHnQTfreZ2B5s3iJeE6IOmQRJWjgVzPw139vaBw1bGWKCIL0vIo
zwzn1OZDjCQiHcFCktEJr59L9MhwTyAWsVrdAfYf+B9haxQnsHKNY67u4s5Lzzfd
u6PUzeetUK29v+PsPmI2cJkxp+iN3epi4hKu9ZzUPSwMqtCceb7qPVxEbpYxY1p9
1n5PJKBLBX9eb9LU6l8zSxPWV7bK3lG4XaMJgnT9x3ies7msFtpKK5bDtotij/l0
GaKeA97pb5uwD9KgWvaFXMIEt8jVTjLEvwRdvCn294GPDF08U8lAkIv7tghluaQh
1QnlE4SEN4LOECj8dsIGJXpGUk3aU3KkJz9icKy+aUgA+2cP21uh6NcDIS3XyfaZ
QjmDQ993ChII8SXWupQZVBiIpcWO4RqZk3lr7Bz5MUCwzDIA359e57SSq5CCkY0N
4B6Vulk7LktfwrdGNVI5BsC9qqxSwSKgRJeZ9wygIaehbHFHFhcBaMDKpiZlBHyz
rsnnlFXCb5s8HKn5LsUgGvB24L7sGNZP2CX7dhHov+YhD+jozLW2p9W4959Bz2Ei
RmqDtmiXLnzqTpXbI+suyCsohKRg6Un0RC47+cpiVwHiXZAW+cn8eiNIjqbVgXLx
KPpdzvvtTnOPlC7SQZSYmdunr3Bf9b77AiC/ZidstK36dRILKz7OA54=
-----END CERTIFICATE-----
---
Server certificate
subject=CN=*.codeberg.page
issuer=C=US, O=Let's Encrypt, CN=R10
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3115 bytes and written 384 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
 Protocol : TLSv1.3
 Cipher : TLS_AES_128_GCM_SHA256
 Session-ID: 466B38D82ACD1A14FCAD1C205F0F16A273BA8081CD2AF57F441F8A2FBCC5F658
 Session-ID-ctx:
 Resumption PSK: A5DAF331F6558FE553F6C4DF4B88413E815AB08892F8BF72E4931C1F737520B0
 PSK identity: None
 PSK identity hint: None
 SRP username: None
 TLS session ticket lifetime hint: 604800 (seconds)
 TLS session ticket:
 0000 - 09 03 c6 f9 57 95 da 95-e0 5c 08 74 81 f6 35 f0 ....W....\.t..5.
 0010 - 37 e7 bd cc 00 44 bf 5c-c7 25 a2 f8 b1 51 40 61 7....D.\.%...Q@a
 0020 - cf 8a d7 4a e7 9c 4d f6-1d e0 0e fb 66 34 26 db ...J..M.....f4&.
 0030 - 9d 16 73 0b a2 44 d6 3b-ed 3e b7 bf 85 b0 47 7c ..s..D.;.>....G|
 0040 - 54 a2 ec 63 50 69 de 99-c6 30 c7 c2 1a ae fe ac T..cPi...0......
 0050 - dd 61 f0 d9 06 21 64 a2-76 4e d3 95 f6 3e 8e e6 .a...!d.vN...>..
 0060 - db 93 9a 41 cc 5d a6 96-96 ...A.]...
 Start Time: 1729004588
 Timeout : 7200 (sec)
 Verify return code: 0 (ok)
 Extended master secret: no
 Max Early Data: 0
---
read R BLOCK
DONE

Success:

openssl s_client -showcerts -connect codeberg.org:443 < /dev/null
Connecting to 217.197.91.145
CONNECTED(00000005)
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=E5
verify return:1
depth=0 CN=codeberg.org
verify return:1
---
Certificate chain
 0 s:CN=codeberg.org
 i:C=US, O=Let's Encrypt, CN=E5
 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
 v:NotBefore: Oct 5 21:13:57 2024 GMT; NotAfter: Jan 3 21:13:56 2025 GMT
-----BEGIN CERTIFICATE-----
MIIDnDCCAyKgAwIBAgISAxBTn0w7udVUmebFLvqlzQTDMAoGCCqGSM49BAMDMDIx
CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF
NTAeFw0yNDEwMDUyMTEzNTdaFw0yNTAxMDMyMTEzNTZaMBcxFTATBgNVBAMTDGNv
ZGViZXJnLm9yZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABBxXbn7bGqIykrQO
7EnGmha4+IhLGI4eyGCzBjXSbXDgXEJkeQOj9naGOmA9/2whggd//YTlqzU9njjy
CpMsVrGjggIxMIICLTAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUH
AwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFeWLnKF1/QDZBrW
gg4hG2RqMckpMB8GA1UdIwQYMBaAFJ8rX888IU+dBLftKyzExnCL0tcNMFUGCCsG
AQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL2U1Lm8ubGVuY3Iub3JnMCIG
CCsGAQUFBzAChhZodHRwOi8vZTUuaS5sZW5jci5vcmcvMDoGA1UdEQQzMDGCDiou
Y29kZWJlcmcub3JnghFjb2RlYmVyZy10ZXN0Lm9yZ4IMY29kZWJlcmcub3JnMBMG
A1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHYAfVke
EuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGSXr3E2wAABAMARzBFAiAp
L7fYQp+Ai0U6FkVKFBrsazeahGmKaKTyUllue/gBYwIhAKpEJ9tUTH7UjxQvBrsG
mJhpYzdxTlbYgCkyJW9E/HFZAHYAzxFW7tUufK/zh1vZaS6b6RpxZ0qwF+ysAdJb
d87MOwgAAAGSXr3EoAAABAMARzBFAiEAjIj78E3uw6f9h+1DoCe7u3w9bvaSakL/
y4RWq4d5Y+ACICZhUga6990OCdK5hzdJ67Z5987jzUJi0KMHzuj5KWNDMAoGCCqG
SM49BAMDA2gAMGUCMQCpX/X3QNPC1khLmKZOR4VLVpvpwX4I0npBArB4vjPkYK3c
t8fL9iCbELfdLKRaSFoCMDJt2NitvLH0s3GauPYP0VW40UeyPthrSopnW5UGfJD4
Iq9w7OCi6mY7D97mhSOWnA==
-----END CERTIFICATE-----
 1 s:C=US, O=Let's Encrypt, CN=E5
 i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
 a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
 v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
-----BEGIN CERTIFICATE-----
MIIEVzCCAj+gAwIBAgIRAIOPbGPOsTmMYgZigxXJ/d4wDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjQwMzEzMDAwMDAw
WhcNMjcwMzEyMjM1OTU5WjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCRTUwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQNCzqK
a2GOtu/cX1jnxkJFVKtj9mZhSAouWXW0gQI3ULc/FnncmOyhKJdyIBwsz9V8UiBO
VHhbhBRrwJCuhezAUUE8Wod/Bk3U/mDR+mwt4X2VEIiiCFQPmRpM5uoKrNijgfgw
gfUwDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcD
ATASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBSfK1/PPCFPnQS37SssxMZw
i9LXDTAfBgNVHSMEGDAWgBR5tFnme7bl5AFzgAiIyBpY9umbbjAyBggrBgEFBQcB
AQQmMCQwIgYIKwYBBQUHMAKGFmh0dHA6Ly94MS5pLmxlbmNyLm9yZy8wEwYDVR0g
BAwwCjAIBgZngQwBAgEwJwYDVR0fBCAwHjAcoBqgGIYWaHR0cDovL3gxLmMubGVu
Y3Iub3JnLzANBgkqhkiG9w0BAQsFAAOCAgEAH3KdNEVCQdqk0LKyuNImTKdRJY1C
2uw2SJajuhqkyGPY8C+zzsufZ+mgnhnq1A2KVQOSykOEnUbx1cy637rBAihx97r+
bcwbZM6sTDIaEriR/PLk6LKs9Be0uoVxgOKDcpG9svD33J+G9Lcfv1K9luDmSTgG
6XNFIN5vfI5gs/lMPyojEMdIzK9blcl2/1vKxO8WGCcjvsQ1nJ/Pwt8LQZBfOFyV
XP8ubAp/au3dc4EKWG9MO5zcx1qT9+NXRGdVWxGvmBFRAajciMfXME1ZuGmk3/GO
koAM7ZkjZmleyokP1LGzmfJcUd9s7eeu1/9/eg5XlXd/55GtYjAM+C4DG5i7eaNq
cm2F+yxYIPt6cbbtYVNJCGfHWqHEQ4FYStUyFnv8sjyqU8ypgZaNJ9aVcWSICLOI
E1/Qv/7oKsnZCWJ926wU6RqG1OYPGOi1zuABhLw61cuPVDT28nQS/e6z95cJXq0e
K1BcaJ6fJZsmbjRgD5p3mvEf5vdQM7MCEvU0tHbsx2I5mHHJoABHb8KVBgWp/lcX
GWiWaeOyB7RP+OfDtvi2OsapxXiV7vNVs7fMlrRjY1joKaqmmycnBvAq14AEbtyL
sVfOS66B8apkeFX2NY4XPEYV4ZSCe8VHPrdrERk2wILG3T/EGmSIkCYVUMSnjmJd
VQD9F6Na/+zmXCc=
-----END CERTIFICATE-----
---
Server certificate
subject=CN=codeberg.org
issuer=C=US, O=Let's Encrypt, CN=E5
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2423 bytes and written 400 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

For curl failure:

curl -vv https://codeberg.org/ideasman42/emacs-undo-fu-session.git
* Host codeberg.org:443 was resolved.
* IPv6: (none)
* IPv4: 217.197.91.145
* Trying 217.197.91.145:443...
* Connected to codeberg.org (217.197.91.145) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.codeberg.page
* start date: Sep 13 04:06:34 2024 GMT
* expire date: Dec 12 04:06:33 2024 GMT
* subjectAltName does not match host name codeberg.org
* SSL: no alternative certificate subject name matches target host name 'codeberg.org'
* Closing connection
curl: (60) SSL: no alternative certificate subject name matches target host name 'codeberg.org'
More details here: https://curl.se/docs/sslcerts.html

Curl success:

curl -vv https://codeberg.org/ideasman42/emacs-undo-fu-session.git
* Host codeberg.org:443 was resolved.
* IPv6: (none)
* IPv4: 217.197.91.145
* Trying 217.197.91.145:443...
* Connected to codeberg.org (217.197.91.145) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=codeberg.org
* start date: Oct 5 21:13:57 2024 GMT
* expire date: Jan 3 21:13:56 2025 GMT
* subjectAltName: host "codeberg.org" matched cert's "codeberg.org"
* issuer: C=US; O=Let's Encrypt; CN=E5
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://codeberg.org/ideasman42/emacs-undo-fu-session.git
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: codeberg.org]
* [HTTP/2] [1] [:path: /ideasman42/emacs-undo-fu-session.git]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /ideasman42/emacs-undo-fu-session.git HTTP/2
> Host: codeberg.org
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< cache-control: max-age=0, private, must-revalidate, no-transform
< content-type: text/html; charset=utf-8
< set-cookie: i_like_gitea=08b76050b4428f39; Path=/; HttpOnly; Secure; SameSite=Lax; Secure; SameSite=Lax
< set-cookie: _csrf=4ur6Tuf76VvNh-Egs43ZwUmjhw06MTcyOTAwNDg3MzMyMDAwNTgxNA; Path=/; Max-Age=86400; HttpOnly; Secure; SameSite=Lax; Secure; SameSite=Lax
< date: 2024年10月15日 15:07:53 GMT
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< permissions-policy: interest-cohort=()
< x-frame-options: sameorigin
< x-content-type-options: nosniff
<
---- Content omitted ---
I'm more than willing to help troubleshoot or look at configurations (I do have some experience with haproxy). I can reproduce this regularly from my end: Also this isn't a matter of just waiting an it resolves itself, it may take 10-20 attempts to get a proper response from the git side. If it were just a matter of me trying over and over with a git clone command that's one thing I guess, but when the clone is part of a larger installation package which is out of my control it makes for a very frustrating experience. At any rate, closing this because you can't solve it seems the wrong move and sends a message that it's unimportant. This should remain open until a fix is found. Just my $.02. Again I'm willing to help in any way that I can. Without knowing what the full architecture looks like, it seems that what is happening is that the haproxy is doing a round-robin load balancing to a pool of servers but the servers are specialized so that some are only doing pages and others are only doing git, but the load balancer isn't inspecting the request to determine which pool to forward the connection to, so sometimes you get a pages server and sometimes you get a git server. I'm not certain why this doesn't appear to be an issue in the browser but only on the command line, I haven't been able to reproduce this behavior in the browser. Here are the outputs I get for both a successful attempt and failed attempt: Failure: ``` openssl s_client -showcerts -connect codeberg.org:443 Connecting to 217.197.91.145 CONNECTED(00000005) depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1 verify return:1 depth=1 C=US, O=Let's Encrypt, CN=R10 verify return:1 depth=0 CN=*.codeberg.page verify return:1 --- Certificate chain 0 s:CN=*.codeberg.page i:C=US, O=Let's Encrypt, CN=R10 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Sep 13 04:06:34 2024 GMT; NotAfter: Dec 12 04:06:33 2024 GMT -----BEGIN CERTIFICATE----- MIIE/TCCA+WgAwIBAgISBE41DOxX9tvwZZi/byTCJ0z3MA0GCSqGSIb3DQEBCwUA MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD EwNSMTAwHhcNMjQwOTEzMDQwNjM0WhcNMjQxMjEyMDQwNjMzWjAaMRgwFgYDVQQD DA8qLmNvZGViZXJnLnBhZ2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB AQC0JYdtNveSPU1kN/f0SjNL6/ZgWXOqCj4PrhSjxsQClFOqKvdEVMV2KjgL3A7P LSWxdok18AlAcxmP/UNo3YbX0lBa0HxPnjqT+Ned2hDJAhlXi2j0sUemHoAK6a7q lE3Bl4lJMGRlgnKvLRKl9E7jRLyEuGHFPQZ0bFDnxr5J7tS1DCzo+boHS86NYsBQ 0IBpq1wAMOqswzpNQC/4o0T8y1zK52B00AEbjEWTFNJJHhPiIp81PNqnZTAUUQQa IDR1s+YsjoTDAmzC9dv1Wq+94K5J37WX0Tt4WFquaTzSR2OqaMcUZBl85LIcCqzF y9+XtHGCUvP5fhdgR5ooZ1a7AgMBAAGjggIiMIICHjAOBgNVHQ8BAf8EBAMCBaAw HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYD VR0OBBYEFEmRhcGRheHwxTaoLpeBKbgx8IVyMB8GA1UdIwQYMBaAFLu8w0el5Lyp xsOkcgwQjaI14cjoMFcGCCsGAQUFBwEBBEswSTAiBggrBgEFBQcwAYYWaHR0cDov L3IxMC5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYXaHR0cDovL3IxMC5pLmxlbmNy Lm9yZy8wKQYDVR0RBCIwIIIPKi5jb2RlYmVyZy5wYWdlgg1jb2RlYmVyZy5wYWdl MBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHYA GZgQcQnw1lIuMIDSnj9ku4NuKMz5D1KO7t/OSj8WtMoAAAGR6cVCFwAABAMARzBF AiEAmhjzPZi5n+OaxjhTK0pAzhMILgAE3QpzYcnvqZwVAzUCIEFyhAuZHFSJWhz9 ukfAfoB/Rz1YzIrqtMcP8YpJbNJJAHYAdv+IPwq2+5VRwmHM9Ye6NLSkzbsp3GhC Cp/mZ0xaOnQAAAGR6cVCIgAABAMARzBFAiEAvzE/BNeFCS+sAsb3amN6dUYOJoAj QDx+UQIbT0RDcV0CICuR8wel4gEJiRJcL8uoeCClFbztEfZqxjzGN6mPOQgwMA0G CSqGSIb3DQEBCwUAA4IBAQAbbyGECdDcair6brWj1Fxuh8f0NTaOaQQxpfYVz57G QScYoLxHWIaxiwFLUaJS/ojq4qW6mZGEwKnmhUBfh024jB9BsSF8A30BeWSeKE7l /1DY5dV1KFH/NDnJB31i1DaCJAhAXVpj89GL8IXdETzsieQla0//IYoc8vCdsKpD rktULWx30zcRja6GhP6ikNzpUx0wIQwTtIjlXFcRZj9o9soLm2jG3JH5yb+rkf24 +oyvSFhXRlF++/lz5X8Law9M5ZZVkK1UAicPLnarGp6nmDMP1eeAxXh5XUw2qIBK FHu4COLvaXLwUqo4qh145O4acZuFimn73SYNORTemfK5 -----END CERTIFICATE----- 1 s:C=US, O=Let's Encrypt, CN=R10 i:C=US, O=Internet Security Research Group, CN=ISRG Root X1 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT -----BEGIN CERTIFICATE----- MIIFBTCCAu2gAwIBAgIQS6hSk/eaL6JzBkuoBI110DANBgkqhkiG9w0BAQsFADBP MQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJuZXQgU2VjdXJpdHkgUmVzZWFy Y2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBYMTAeFw0yNDAzMTMwMDAwMDBa Fw0yNzAzMTIyMzU5NTlaMDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBF bmNyeXB0MQwwCgYDVQQDEwNSMTAwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQDPV+XmxFQS7bRH/sknWHZGUCiMHT6I3wWd1bUYKb3dtVq/+vbOo76vACFL YlpaPAEvxVgD9on/jhFD68G14BQHlo9vH9fnuoE5CXVlt8KvGFs3Jijno/QHK20a /6tYvJWuQP/py1fEtVt/eA0YYbwX51TGu0mRzW4Y0YCF7qZlNrx06rxQTOr8IfM4 FpOUurDTazgGzRYSespSdcitdrLCnF2YRVxvYXvGLe48E1KGAdlX5jgc3421H5KR mudKHMxFqHJV8LDmowfs/acbZp4/SItxhHFYyTr6717yW0QrPHTnj7JHwQdqzZq3 DZb3EoEmUVQK7GH29/Xi8orIlQ2NAgMBAAGjgfgwgfUwDgYDVR0PAQH/BAQDAgGG MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATASBgNVHRMBAf8ECDAGAQH/ AgEAMB0GA1UdDgQWBBS7vMNHpeS8qcbDpHIMEI2iNeHI6DAfBgNVHSMEGDAWgBR5 tFnme7bl5AFzgAiIyBpY9umbbjAyBggrBgEFBQcBAQQmMCQwIgYIKwYBBQUHMAKG Fmh0dHA6Ly94MS5pLmxlbmNyLm9yZy8wEwYDVR0gBAwwCjAIBgZngQwBAgEwJwYD VR0fBCAwHjAcoBqgGIYWaHR0cDovL3gxLmMubGVuY3Iub3JnLzANBgkqhkiG9w0B AQsFAAOCAgEAkrHnQTfreZ2B5s3iJeE6IOmQRJWjgVzPw139vaBw1bGWKCIL0vIo zwzn1OZDjCQiHcFCktEJr59L9MhwTyAWsVrdAfYf+B9haxQnsHKNY67u4s5Lzzfd u6PUzeetUK29v+PsPmI2cJkxp+iN3epi4hKu9ZzUPSwMqtCceb7qPVxEbpYxY1p9 1n5PJKBLBX9eb9LU6l8zSxPWV7bK3lG4XaMJgnT9x3ies7msFtpKK5bDtotij/l0 GaKeA97pb5uwD9KgWvaFXMIEt8jVTjLEvwRdvCn294GPDF08U8lAkIv7tghluaQh 1QnlE4SEN4LOECj8dsIGJXpGUk3aU3KkJz9icKy+aUgA+2cP21uh6NcDIS3XyfaZ QjmDQ993ChII8SXWupQZVBiIpcWO4RqZk3lr7Bz5MUCwzDIA359e57SSq5CCkY0N 4B6Vulk7LktfwrdGNVI5BsC9qqxSwSKgRJeZ9wygIaehbHFHFhcBaMDKpiZlBHyz rsnnlFXCb5s8HKn5LsUgGvB24L7sGNZP2CX7dhHov+YhD+jozLW2p9W4959Bz2Ei RmqDtmiXLnzqTpXbI+suyCsohKRg6Un0RC47+cpiVwHiXZAW+cn8eiNIjqbVgXLx KPpdzvvtTnOPlC7SQZSYmdunr3Bf9b77AiC/ZidstK36dRILKz7OA54= -----END CERTIFICATE----- --- Server certificate subject=CN=*.codeberg.page issuer=C=US, O=Let's Encrypt, CN=R10 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 3115 bytes and written 384 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 Server public key is 2048 bit This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- --- Post-Handshake New Session Ticket arrived: SSL-Session: Protocol : TLSv1.3 Cipher : TLS_AES_128_GCM_SHA256 Session-ID: 466B38D82ACD1A14FCAD1C205F0F16A273BA8081CD2AF57F441F8A2FBCC5F658 Session-ID-ctx: Resumption PSK: A5DAF331F6558FE553F6C4DF4B88413E815AB08892F8BF72E4931C1F737520B0 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 604800 (seconds) TLS session ticket: 0000 - 09 03 c6 f9 57 95 da 95-e0 5c 08 74 81 f6 35 f0 ....W....\.t..5. 0010 - 37 e7 bd cc 00 44 bf 5c-c7 25 a2 f8 b1 51 40 61 7....D.\.%...Q@a 0020 - cf 8a d7 4a e7 9c 4d f6-1d e0 0e fb 66 34 26 db ...J..M.....f4&. 0030 - 9d 16 73 0b a2 44 d6 3b-ed 3e b7 bf 85 b0 47 7c ..s..D.;.>....G| 0040 - 54 a2 ec 63 50 69 de 99-c6 30 c7 c2 1a ae fe ac T..cPi...0...... 0050 - dd 61 f0 d9 06 21 64 a2-76 4e d3 95 f6 3e 8e e6 .a...!d.vN...>.. 0060 - db 93 9a 41 cc 5d a6 96-96 ...A.]... Start Time: 1729004588 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no Max Early Data: 0 --- read R BLOCK DONE ``` Success: ``` openssl s_client -showcerts -connect codeberg.org:443 < /dev/null Connecting to 217.197.91.145 CONNECTED(00000005) depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1 verify return:1 depth=1 C=US, O=Let's Encrypt, CN=E5 verify return:1 depth=0 CN=codeberg.org verify return:1 --- Certificate chain 0 s:CN=codeberg.org i:C=US, O=Let's Encrypt, CN=E5 a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384 v:NotBefore: Oct 5 21:13:57 2024 GMT; NotAfter: Jan 3 21:13:56 2025 GMT -----BEGIN CERTIFICATE----- MIIDnDCCAyKgAwIBAgISAxBTn0w7udVUmebFLvqlzQTDMAoGCCqGSM49BAMDMDIx CzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQDEwJF NTAeFw0yNDEwMDUyMTEzNTdaFw0yNTAxMDMyMTEzNTZaMBcxFTATBgNVBAMTDGNv ZGViZXJnLm9yZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABBxXbn7bGqIykrQO 7EnGmha4+IhLGI4eyGCzBjXSbXDgXEJkeQOj9naGOmA9/2whggd//YTlqzU9njjy CpMsVrGjggIxMIICLTAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUH AwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFeWLnKF1/QDZBrW gg4hG2RqMckpMB8GA1UdIwQYMBaAFJ8rX888IU+dBLftKyzExnCL0tcNMFUGCCsG AQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL2U1Lm8ubGVuY3Iub3JnMCIG CCsGAQUFBzAChhZodHRwOi8vZTUuaS5sZW5jci5vcmcvMDoGA1UdEQQzMDGCDiou Y29kZWJlcmcub3JnghFjb2RlYmVyZy10ZXN0Lm9yZ4IMY29kZWJlcmcub3JnMBMG A1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHYAfVke EuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGSXr3E2wAABAMARzBFAiAp L7fYQp+Ai0U6FkVKFBrsazeahGmKaKTyUllue/gBYwIhAKpEJ9tUTH7UjxQvBrsG mJhpYzdxTlbYgCkyJW9E/HFZAHYAzxFW7tUufK/zh1vZaS6b6RpxZ0qwF+ysAdJb d87MOwgAAAGSXr3EoAAABAMARzBFAiEAjIj78E3uw6f9h+1DoCe7u3w9bvaSakL/ y4RWq4d5Y+ACICZhUga6990OCdK5hzdJ67Z5987jzUJi0KMHzuj5KWNDMAoGCCqG SM49BAMDA2gAMGUCMQCpX/X3QNPC1khLmKZOR4VLVpvpwX4I0npBArB4vjPkYK3c t8fL9iCbELfdLKRaSFoCMDJt2NitvLH0s3GauPYP0VW40UeyPthrSopnW5UGfJD4 Iq9w7OCi6mY7D97mhSOWnA== -----END CERTIFICATE----- 1 s:C=US, O=Let's Encrypt, CN=E5 i:C=US, O=Internet Security Research Group, CN=ISRG Root X1 a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256 v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT -----BEGIN CERTIFICATE----- MIIEVzCCAj+gAwIBAgIRAIOPbGPOsTmMYgZigxXJ/d4wDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjQwMzEzMDAwMDAw WhcNMjcwMzEyMjM1OTU5WjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg RW5jcnlwdDELMAkGA1UEAxMCRTUwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQNCzqK a2GOtu/cX1jnxkJFVKtj9mZhSAouWXW0gQI3ULc/FnncmOyhKJdyIBwsz9V8UiBO VHhbhBRrwJCuhezAUUE8Wod/Bk3U/mDR+mwt4X2VEIiiCFQPmRpM5uoKrNijgfgw gfUwDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcD ATASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBSfK1/PPCFPnQS37SssxMZw i9LXDTAfBgNVHSMEGDAWgBR5tFnme7bl5AFzgAiIyBpY9umbbjAyBggrBgEFBQcB AQQmMCQwIgYIKwYBBQUHMAKGFmh0dHA6Ly94MS5pLmxlbmNyLm9yZy8wEwYDVR0g BAwwCjAIBgZngQwBAgEwJwYDVR0fBCAwHjAcoBqgGIYWaHR0cDovL3gxLmMubGVu Y3Iub3JnLzANBgkqhkiG9w0BAQsFAAOCAgEAH3KdNEVCQdqk0LKyuNImTKdRJY1C 2uw2SJajuhqkyGPY8C+zzsufZ+mgnhnq1A2KVQOSykOEnUbx1cy637rBAihx97r+ bcwbZM6sTDIaEriR/PLk6LKs9Be0uoVxgOKDcpG9svD33J+G9Lcfv1K9luDmSTgG 6XNFIN5vfI5gs/lMPyojEMdIzK9blcl2/1vKxO8WGCcjvsQ1nJ/Pwt8LQZBfOFyV XP8ubAp/au3dc4EKWG9MO5zcx1qT9+NXRGdVWxGvmBFRAajciMfXME1ZuGmk3/GO koAM7ZkjZmleyokP1LGzmfJcUd9s7eeu1/9/eg5XlXd/55GtYjAM+C4DG5i7eaNq cm2F+yxYIPt6cbbtYVNJCGfHWqHEQ4FYStUyFnv8sjyqU8ypgZaNJ9aVcWSICLOI E1/Qv/7oKsnZCWJ926wU6RqG1OYPGOi1zuABhLw61cuPVDT28nQS/e6z95cJXq0e K1BcaJ6fJZsmbjRgD5p3mvEf5vdQM7MCEvU0tHbsx2I5mHHJoABHb8KVBgWp/lcX GWiWaeOyB7RP+OfDtvi2OsapxXiV7vNVs7fMlrRjY1joKaqmmycnBvAq14AEbtyL sVfOS66B8apkeFX2NY4XPEYV4ZSCe8VHPrdrERk2wILG3T/EGmSIkCYVUMSnjmJd VQD9F6Na/+zmXCc= -----END CERTIFICATE----- --- Server certificate subject=CN=codeberg.org issuer=C=US, O=Let's Encrypt, CN=E5 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 2423 bytes and written 400 bytes Verification: OK --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 256 bit This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- DONE ``` For curl failure: ``` curl -vv https://codeberg.org/ideasman42/emacs-undo-fu-session.git * Host codeberg.org:443 was resolved. * IPv6: (none) * IPv4: 217.197.91.145 * Trying 217.197.91.145:443... * Connected to codeberg.org (217.197.91.145) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: CN=*.codeberg.page * start date: Sep 13 04:06:34 2024 GMT * expire date: Dec 12 04:06:33 2024 GMT * subjectAltName does not match host name codeberg.org * SSL: no alternative certificate subject name matches target host name 'codeberg.org' * Closing connection curl: (60) SSL: no alternative certificate subject name matches target host name 'codeberg.org' More details here: https://curl.se/docs/sslcerts.html ``` Curl success: ``` curl -vv https://codeberg.org/ideasman42/emacs-undo-fu-session.git * Host codeberg.org:443 was resolved. * IPv6: (none) * IPv4: 217.197.91.145 * Trying 217.197.91.145:443... * Connected to codeberg.org (217.197.91.145) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: CN=codeberg.org * start date: Oct 5 21:13:57 2024 GMT * expire date: Jan 3 21:13:56 2025 GMT * subjectAltName: host "codeberg.org" matched cert's "codeberg.org" * issuer: C=US; O=Let's Encrypt; CN=E5 * SSL certificate verify ok. * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://codeberg.org/ideasman42/emacs-undo-fu-session.git * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: codeberg.org] * [HTTP/2] [1] [:path: /ideasman42/emacs-undo-fu-session.git] * [HTTP/2] [1] [user-agent: curl/8.7.1] * [HTTP/2] [1] [accept: */*] > GET /ideasman42/emacs-undo-fu-session.git HTTP/2 > Host: codeberg.org > User-Agent: curl/8.7.1 > Accept: */* > * Request completely sent off < HTTP/2 200 < cache-control: max-age=0, private, must-revalidate, no-transform < content-type: text/html; charset=utf-8 < set-cookie: i_like_gitea=08b76050b4428f39; Path=/; HttpOnly; Secure; SameSite=Lax; Secure; SameSite=Lax < set-cookie: _csrf=4ur6Tuf76VvNh-Egs43ZwUmjhw06MTcyOTAwNDg3MzMyMDAwNTgxNA; Path=/; Max-Age=86400; HttpOnly; Secure; SameSite=Lax; Secure; SameSite=Lax < date: 2024年10月15日 15:07:53 GMT < strict-transport-security: max-age=63072000; includeSubDomains; preload < permissions-policy: interest-cohort=() < x-frame-options: sameorigin < x-content-type-options: nosniff < ---- Content omitted --- ```
Owner
Copy link

We have seen some similar output, but were never able to reproduce. Could you check one thing for us next time this happens: Is there a potential timing issue? So that requests that receive the *.codeberg.page certificate take longer (about 5 seconds or more) than successful requests?

This is something that just caught my eye in the config file (the potential timeout):

# Wait up to 5s for a SNI header & only accept TLS connections
tcp-request inspect-delay 5s
tcp-request content capture req.ssl_sni len 255
We have seen some similar output, but were never able to reproduce. Could you check one thing for us next time this happens: Is there a potential timing issue? So that requests that receive the *.codeberg.page certificate take longer (about 5 seconds or more) than successful requests? This is something that just caught my eye in the config file (the potential timeout): https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/src/commit/bef038ca91cb928e0b865ada4bc6d579b2bc857e/hosts/kampenwand/etc/haproxy/haproxy.cfg#L88-L90
Owner
Copy link

If this is true, and we intend to increase the timeout, it still means an underlying connection issue that prevents successful transmission of the SNI header for at least five seconds. It could be related to networking issues, or maybe due to high load on our system. Yesterday was a hard day for Codeberg with huge amounts of crawling going on, so it might be that our server was simply not able to process the header within five seconds. If the behaviour was changed, it might instead run into connection issues due to connection timeouts.

If this is true, and we intend to increase the timeout, it still means an underlying connection issue that prevents successful transmission of the SNI header for at least five seconds. It could be related to networking issues, or maybe due to high load on our system. Yesterday was a hard day for Codeberg with huge amounts of crawling going on, so it might be that our server was simply not able to process the header within five seconds. If the behaviour was changed, it might instead run into connection issues due to connection timeouts.

Is there a potential timing issue?

Potentially related detail: In my test case I was working on a rather slow system, where the whole TLS handshake might take a few seconds on my end.

> Is there a potential timing issue? Potentially related detail: In my test case I was working on a rather slow system, where the whole TLS handshake might take a few seconds on my end.
Sign in to join this conversation.
No Branch/Tag specified
main
No results found.
Labels
Clear labels
accessibility

Reduces accessibility and is thus a "bug" for certain user groups on Codeberg.
bug

Something is not working the way it should. Does not concern outages.
bug
infrastructure
Errors evidently caused by infrastructure malfunctions or outages
Codeberg

This issue involves Codeberg's downstream modifications and settings and/or Codeberg's structures.
contributions welcome

Please join the discussion and consider contributing a PR!
docs

No bug, but an improvement to the docs or UI description will help
duplicate

This issue or pull request already exists
enhancement

New feature
infrastructure

Involves changes to the server setups, use `bug/infrastructure` for infrastructure-related user errors.
legal

An issue directly involving legal compliance
licence / ToS

involving questions about the ToS, especially licencing compliance
please chill
we are volunteers
Please consider editing your posts and remember that there is a human on the other side. We get that you are frustrated, but it's harder for us to help you this way.
public relations

Things related to Codeberg's external communication
question

More information is needed
question
user support
This issue contains a clearly stated problem. However, it is not clear whether we have to fix anything on Codeberg's end, but we're helping them fix it and/or find the cause.
s/Forgejo

Related to Forgejo. Please also check Forgejo's issue tracker.
s/Forgejo/migration

Migration related issues in Forgejo
s/Pages

Issues related to the Codeberg Pages feature
s/Weblate

Issue is related to the Weblate instance at https://translate.codeberg.org
s/Woodpecker

Woodpecker CI related issue
security

involves improvements to the sites security
service

Add a new service to the Codeberg ecosystem (instead of implementing into Gitea)
upstream

An open issue or pull request to an upstream repository to fix this issue (partially or completely) exists (i.e. Gitea, Forgejo, etc.)
wontfix

Codeberg's current set of contributors are not planning to spend time on delegating this issue.
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
6 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Codeberg/Community#1368
Reference in a new issue
Codeberg/Community
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?