Cannot perform SSH based git operations on private repositories #2056

Juhe commented 2025年07月27日 19:08:23 +02:00

Comment

Hi,

First of all, I have been loving Codeberg for a few years now! ❤️ But unfortunately now I have to create this issue :(

While pushing few commits to my gitops repo, I noticed that I started just getting an error:

Connection closed by 217.197.84.140 port 22
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

While troubleshooting this issue as it did not go away itself after few minutes I made the following observations:

• My workstation is not the issue, my ArgoCD instance is not able to sync the repo either
• This is not my home network issue, my ArgoCD instance has completely different public IP than my workstation
• I can also run nc -vzw2 codeberg.org 22 succesfully, so there probably is no IP ban?
• I can clone my own public repositories with git clone ssh://git@codeberg.org/<user>/<repository>.git so it seems like SSH works fine for those
• When running ssh git@codeberg.org -v 5 I see that the connection is made but KEX fails, see below for detailed error

debug1: Connecting to codeberg.org [217.197.84.140] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type 12
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.9
kex_exchange_identification: Connection closed by remote host
Connection closed by 217.197.84.140 port 22

I am running out of ideas here. Could it be that I am rate limited? I believe the ArgoCD syncs the data from this private repo every 5 mins and has probably 20 subcalls for every "application". And as I am still developing I can imagine I am producing multiple hunders of git SSH operations every hour, both from my ArgoCD instance and also from my workstation.

### Comment Hi, First of all, I have been loving Codeberg for a few years now! ❤️ But unfortunately now I have to create this issue :( While pushing few commits to my gitops repo, I noticed that I started just getting an error: ``` Connection closed by 217.197.84.140 port 22 fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` While troubleshooting this issue as it did not go away itself after few minutes I made the following observations: - My workstation is not the issue, my ArgoCD instance is not able to sync the repo either - This is not my home network issue, my ArgoCD instance has completely different public IP than my workstation - I can also run `nc -vzw2 codeberg.org 22` succesfully, so there probably is no IP ban? - I can clone my own public repositories with `git clone ssh://git@codeberg.org/<user>/<repository>.git` so it seems like SSH works fine for those - When running `ssh git@codeberg.org -v 5` I see that the connection is made but KEX fails, see below for detailed error ``` debug1: Connecting to codeberg.org [217.197.84.140] port 22. debug1: Connection established. debug1: identity file /home/user/.ssh/id_rsa type 0 debug1: identity file /home/user/.ssh/id_rsa-cert type -1 debug1: identity file /home/user/.ssh/id_ecdsa type -1 debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/user/.ssh/id_ed25519 type -1 debug1: identity file /home/user/.ssh/id_ed25519-cert type -1 debug1: identity file /home/user/.ssh/id_ed25519_sk type 12 debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/user/.ssh/id_xmss type -1 debug1: identity file /home/user/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_9.9 kex_exchange_identification: Connection closed by remote host Connection closed by 217.197.84.140 port 22 ``` I am running out of ideas here. Could it be that I am rate limited? I believe the ArgoCD syncs the data from this private repo every 5 mins and has probably 20 subcalls for every "application". And as I am still developing I can imagine I am producing multiple hunders of git SSH operations every hour, both from my ArgoCD instance and also from my workstation.

allainp commented 2025年07月27日 19:21:53 +02:00

$ ssh -Tvvv git@codeberg.org
debug1: OpenSSH_10.0p2, OpenSSL 3.5.1 1 Jul 2025
debug3: Running on Linux 6.12.39_1 #1 SMP PREEMPT_DYNAMIC Thu Jul 24 09:58:03 UTC 2025 x86_64
debug3: Started with: ssh -Tvvv git@codeberg.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/allainp/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/allainp/.ssh/known_hosts2'
debug2: resolving "codeberg.org" port 22
debug3: resolve_host: lookup codeberg.org:22
debug3: channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to codeberg.org [217.197.84.140] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/allainp/.ssh/id_rsa type -1
debug1: identity file /home/allainp/.ssh/id_rsa-cert type -1
debug1: identity file /home/allainp/.ssh/id_ecdsa type -1
debug1: identity file /home/allainp/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/allainp/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/allainp/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/allainp/.ssh/id_ed25519 type 3
debug1: identity file /home/allainp/.ssh/id_ed25519-cert type -1
debug1: identity file /home/allainp/.ssh/id_ed25519_sk type -1
debug1: identity file /home/allainp/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/allainp/.ssh/id_xmss type -1
debug1: identity file /home/allainp/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_10.0
kex_exchange_identification: Connection closed by remote host
Connection closed by 217.197.84.140 port 22

```bash $ ssh -Tvvv git@codeberg.org debug1: OpenSSH_10.0p2, OpenSSL 3.5.1 1 Jul 2025 debug3: Running on Linux 6.12.39_1 #1 SMP PREEMPT_DYNAMIC Thu Jul 24 09:58:03 UTC 2025 x86_64 debug3: Started with: ssh -Tvvv git@codeberg.org debug1: Reading configuration data /etc/ssh/ssh_config debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/allainp/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/allainp/.ssh/known_hosts2' debug2: resolving "codeberg.org" port 22 debug3: resolve_host: lookup codeberg.org:22 debug3: channel_clear_timeouts: clearing debug3: ssh_connect_direct: entering debug1: Connecting to codeberg.org [217.197.84.140] port 22. debug3: set_sock_tos: set socket 3 IP_TOS 0x48 debug1: Connection established. debug1: identity file /home/allainp/.ssh/id_rsa type -1 debug1: identity file /home/allainp/.ssh/id_rsa-cert type -1 debug1: identity file /home/allainp/.ssh/id_ecdsa type -1 debug1: identity file /home/allainp/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/allainp/.ssh/id_ecdsa_sk type -1 debug1: identity file /home/allainp/.ssh/id_ecdsa_sk-cert type -1 debug1: identity file /home/allainp/.ssh/id_ed25519 type 3 debug1: identity file /home/allainp/.ssh/id_ed25519-cert type -1 debug1: identity file /home/allainp/.ssh/id_ed25519_sk type -1 debug1: identity file /home/allainp/.ssh/id_ed25519_sk-cert type -1 debug1: identity file /home/allainp/.ssh/id_xmss type -1 debug1: identity file /home/allainp/.ssh/id_xmss-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_10.0 kex_exchange_identification: Connection closed by remote host Connection closed by 217.197.84.140 port 22 ```

oliphant commented 2025年07月27日 19:25:34 +02:00

Same issue here, and it's not just private repos. I'm dead in the water.

I referenced the issue here, but same error.

forgejo/forgejo#8634 (comment)

Same issue here, and it's not just private repos. I'm dead in the water. I referenced the issue here, but same error. https://codeberg.org/forgejo/forgejo/issues/8634#issuecomment-6012814

Juhe commented 2025年07月27日 19:31:33 +02:00

@oliphant I do not think this is related to the issues that came with the forgejo release. I started experiencing this issue around 1 hour ago. That would be ~19:30 EEST.

@oliphant I do not think this is related to the issues that came with the forgejo release. I started experiencing this issue around 1 hour ago. That would be ~19:30 EEST.

oliphant commented 2025年07月27日 19:35:48 +02:00

@Juhe wrote in #2056 (comment):

@oliphant I do not think this is related to the issues that came with the forgejo release. I started experiencing this issue around 1 hour ago. That would be ~19:30 EEST.

Was just able to push an update to a repo. Give it another shot, I'd say, they might be fixing it.

@Juhe wrote in https://codeberg.org/Codeberg/Community/issues/2056#issuecomment-6013072: > @oliphant I do not think this is related to the issues that came with the forgejo release. I started experiencing this issue around 1 hour ago. That would be ~19:30 EEST. Was just able to push an update to a repo. Give it another shot, I'd say, they might be fixing it.

Juhe commented 2025年07月27日 19:38:31 +02:00

Yup, I got a successful push and also my ArgoCD was able to pull! Lets see if it stays this way.

I also checked status.codeberg.or and the SSH service shows 100% uptime. Looks like it might not be using proper metrics for the status monitoring.

Yup, I got a successful push and also my ArgoCD was able to pull! Lets see if it stays this way. I also checked [status.codeberg.or](https://status.codeberg.org/status/codeberg) and the SSH service shows 100% uptime. Looks like it might not be using proper metrics for the status monitoring.

Juhe commented 2025年07月27日 20:39:04 +02:00

Closing as the issue seems to be mostly mitigated and caused by malicious actors.

Closing as the issue seems to be mostly mitigated and caused by malicious actors.