3 second SSH latency on codeberg.org #2152

tonymet commented 2025年09月30日 05:58:57 +02:00

Comment

Codeberg SSH connections are 3-4 seconds . I believe this may be due to reverse lookup settings on openssh. The uniform 3 second latency makes me suspicious of a timeout bug.

Steps to Repeat

ssh test

time ssh git@codeberg.org

git fetch test

git remote add test ssh://git@codeberg.org/USER/1mb-club.git
git fetch test

Actual Results -- 3-4 seconds

time ssh git@codeberg.org
Connection to codeberg.org closed.
ssh git@codeberg.org 0.07s user 0.00s system 2% cpu 3.347 total

Expected Results -- 500-1000 ms should be doable.

time ssh git@github.com 
ssh git@github.com 0.07s user 0.00s system 7% cpu 0.923 total

### Comment Codeberg SSH connections are 3-4 seconds . I believe this may be due to reverse lookup settings on openssh. The uniform 3 second latency makes me suspicious of a timeout bug. ## Steps to Repeat #### ssh test ``` time ssh git@codeberg.org ``` #### git fetch test ``` git remote add test ssh://git@codeberg.org/USER/1mb-club.git git fetch test ``` ## Actual Results -- 3-4 seconds ``` time ssh git@codeberg.org Connection to codeberg.org closed. ssh git@codeberg.org 0.07s user 0.00s system 2% cpu 3.347 total ``` ## Expected Results -- 500-1000 ms should be doable. ``` time ssh git@github.com ssh git@github.com 0.07s user 0.00s system 7% cpu 0.923 total ```

tonymet commented 2025年09月30日 20:04:03 +02:00

Ran a test against ssh://git@code.forgejo.org/tonymet/test1.git and observed uniformly terrible performance

git fetch origin 0.06s user 0.01s system 2% cpu 2.726 total

Very suspiciously close results avg =2.74 , σ (stddev) = 2.8ms (tiny variance)

Run 4 completed in: 2.740137571s
Run 2 completed in: 2.744722054s
Run 3 completed in: 2.744687098s
Run 5 completed in: 2.744952366s
Run 1 completed in: 2.748852712s
sum: 13.723351801s, μ 2.74467036s, σ 2.761185ms

ssh git@code.forgejo.org similar peculiar uniformity

Averaging time for: [ssh git@code.forgejo.org] (running 5 times concurrently)
Run 3 completed in: 2.369417094s
Run 5 completed in: 2.374196812s
Run 2 completed in: 2.374138142s
Run 1 completed in: 2.373833856s
Run 4 completed in: 2.385197094s
sum: 11.876782998s, μ 2.375356599s, σ 5.239495ms

Ran a test against `ssh://git@code.forgejo.org/tonymet/test1.git` and observed uniformly terrible performance ``` git fetch origin 0.06s user 0.01s system 2% cpu 2.726 total ``` Very suspiciously close results avg =2.74 , σ (stddev) = 2.8ms (tiny variance) ``` Run 4 completed in: 2.740137571s Run 2 completed in: 2.744722054s Run 3 completed in: 2.744687098s Run 5 completed in: 2.744952366s Run 1 completed in: 2.748852712s sum: 13.723351801s, μ 2.74467036s, σ 2.761185ms ``` `ssh git@code.forgejo.org` similar peculiar uniformity ``` Averaging time for: [ssh git@code.forgejo.org] (running 5 times concurrently) Run 3 completed in: 2.369417094s Run 5 completed in: 2.374196812s Run 2 completed in: 2.374138142s Run 1 completed in: 2.373833856s Run 4 completed in: 2.385197094s sum: 11.876782998s, μ 2.375356599s, σ 5.239495ms ```

tonymet commented 2025年09月30日 20:04:43 +02:00

in other words there are fixed startup costs. maybe a reverse lookup or ident. or other connection timeout to the backend.

in other words there are fixed startup costs. maybe a reverse lookup or ident. or other connection timeout to the backend.

tonymet commented 2025年09月30日 23:37:56 +02:00

I was not able to repro on fresh forgejo-12.0.4-linux-amd64 with START_SSH_SERVER=true , but i'm guessing prod instances use openssh . I'll see if i can tinker and help repro. if someone can share non-sensitive ssh-server.conf that will help

I was not able to repro on fresh forgejo-12.0.4-linux-amd64 with `START_SSH_SERVER=true` , but i'm guessing prod instances use openssh . I'll see if i can tinker and help repro. if someone can share non-sensitive ssh-server.conf that will help

0ko commented 2025年10月02日 13:07:15 +02:00

Both code.forgejo and codeberg use external SSH server according to

• forgejo/k8s-cluster@46df2c58ba/flux/apps/forgejo-code/forgejo-code.yaml
• Codeberg-Infrastructure/build-deploy-forgejo@b211bb3a05/etc/forgejo/conf/base.ini

I tried running same tests and got 1.1±0.1s for both code.forgejo and codeberg and 1.8±0.1s for github. The performance of all three was ok at the time. So I can't repro your initial testing.

For better reproducibility:

• Are you sure you're using ed25519 consistently across all services? I can only see what you have on Codeberg.
• Are you sure you're getting authenticated on all three? When I get Permission denied this speeds up the command quite a bit for all three.

Both code.forgejo and codeberg use external SSH server according to * https://codeberg.org/forgejo/k8s-cluster/src/commit/46df2c58ba8bf8572811acce571e0c439e4a99e7/flux/apps/forgejo-code/forgejo-code.yaml * https://codeberg.org/Codeberg-Infrastructure/build-deploy-forgejo/src/commit/b211bb3a0572128f2f4f8ed867c4f16f84587795/etc/forgejo/conf/base.ini I tried running same tests and got 1.1±0.1s for both code.forgejo and codeberg and 1.8±0.1s for github. The performance of all three was ok at the time. So I can't repro your initial testing. For better reproducibility: * Are you sure you're using ed25519 consistently across all services? I can only see [what you have on Codeberg](https://codeberg.org/tonymet.keys). * Are you sure you're getting authenticated on all three? When I get Permission denied this speeds up the command quite a bit for all three.

tonymet commented 2025年10月02日 17:53:50 +02:00

thanks for the guidance it's a big help.

Are you sure you're getting authenticated on all three? When I get Permission denied this speeds up the command quite a bit for all three.

authentication is working well

Are you sure you're using ed25519 consistently across all services? I can only see what you have on Codeberg.

i'll do some better testing around this to see if there's a pattern. I have rsa on code.forgejo and ed25519 on codeberg.

thanks for the guidance it's a big help. > Are you sure you're getting authenticated on all three? When I get Permission denied this speeds up the command quite a bit for all three. authentication is working well > Are you sure you're using ed25519 consistently across all services? I can only see what you have on Codeberg. i'll do some better testing around this to see if there's a pattern. I have rsa on code.forgejo and ed25519 on codeberg.

tonymet commented 2025年10月02日 18:36:46 +02:00

I tried running same tests and got 1.1±0.1s

try running your connects with hyperfine (apt install hyperfine) which will help provide a statistical distribution e.g.

hyperfine 'git fetch origin' or hyperfine 'ssh git@codeberg.org' .

It's not just the latency that raises suspicion, but the uniformity. usually fixed startup costs are an easy win.

> I tried running same tests and got 1.1±0.1s try running your connects with hyperfine (`apt install hyperfine`) which will help provide a statistical distribution e.g. `hyperfine 'git fetch origin'` or `hyperfine 'ssh git@codeberg.org'` . It's not just the latency that raises suspicion, but the uniformity. usually fixed startup costs are an easy win.

tonymet commented 2025年10月02日 19:01:48 +02:00

Your guidance helped i was able to improve client side SSH . but there still seems to be a huge delta e.g. vs https remtes and ssh

Results

https avg = 792ms, sd = 96.6
ssh avg = 3698 ms , sd=316

❯ hyperfine -M3 -N "git fetch latest" # https
Benchmark 1: git fetch latest
 Time (mean ± σ): 792.0 ms ± 96.6 ms [User: 29.9 ms, System: 20.0 ms]
 Range (min ... max): 734.7 ms ... 903.5 ms 3 runs
❯ hyperfine -M3 -N "git fetch tonymet-cb" # ssh
Benchmark 1: git fetch tonymet-cb
 Time (mean ± σ): 3.698 s ± 0.316 s [User: 0.066 s, System: 0.013 s]
 Range (min ... max): 3.457 s ... 4.056 s 3 runs

Fixes

1. remove dir scan ssh_config

index 7f43866..ec7363e 100644
--- a/ssh/ssh_config
+++ b/ssh/ssh_config
@@ -16,7 +16,7 @@
 # list of available options, their meanings and defaults, please see the
 # ssh_config(5) man page.
-Include /etc/ssh/ssh_config.d/*.conf
+#Include /etc/ssh/ssh_config.d/*.conf

2. reduce identity & known_host lookups in ~/.ssh/config

--- a/.ssh/config
+++ b/.ssh/config
@@ -2,6 +2,10 @@ ForwardAgent yes
 ForwardX11 no
 StrictHostKeyChecking no
 IdentityFile ~/.ssh/id_rsa
+Host *
+ UserKnownHostsFile ~/.ssh/known_hosts
+ GlobalKnownHostsFile /dev/null
+
@@ -15,5 +19,7 @@ Host pi*.local
 Host *codeberg.org
 IdentityFile ~/.ssh/id_ed25519
+ IdentitiesOnly yes
 Host *forgejo.org
 IdentityFile ~/.ssh/id_ed25519
+ IdentitiesOnly yes

Your guidance helped i was able to improve client side SSH . but there still seems to be a huge delta e.g. vs https remtes and ssh ## Results https avg = 792ms, sd = 96.6 ssh avg = 3698 ms , sd=316 ``` ❯ hyperfine -M3 -N "git fetch latest" # https Benchmark 1: git fetch latest Time (mean ± σ): 792.0 ms ± 96.6 ms [User: 29.9 ms, System: 20.0 ms] Range (min ... max): 734.7 ms ... 903.5 ms 3 runs ❯ hyperfine -M3 -N "git fetch tonymet-cb" # ssh Benchmark 1: git fetch tonymet-cb Time (mean ± σ): 3.698 s ± 0.316 s [User: 0.066 s, System: 0.013 s] Range (min ... max): 3.457 s ... 4.056 s 3 runs ``` ## Fixes 1. remove dir scan ssh_config ``` index 7f43866..ec7363e 100644 --- a/ssh/ssh_config +++ b/ssh/ssh_config @@ -16,7 +16,7 @@ # list of available options, their meanings and defaults, please see the # ssh_config(5) man page. -Include /etc/ssh/ssh_config.d/*.conf +#Include /etc/ssh/ssh_config.d/*.conf ``` 2. reduce identity & known_host lookups in ~/.ssh/config ``` --- a/.ssh/config +++ b/.ssh/config @@ -2,6 +2,10 @@ ForwardAgent yes ForwardX11 no StrictHostKeyChecking no IdentityFile ~/.ssh/id_rsa +Host * + UserKnownHostsFile ~/.ssh/known_hosts + GlobalKnownHostsFile /dev/null + @@ -15,5 +19,7 @@ Host pi*.local Host *codeberg.org IdentityFile ~/.ssh/id_ed25519 + IdentitiesOnly yes Host *forgejo.org IdentityFile ~/.ssh/id_ed25519 + IdentitiesOnly yes ```

tonymet commented 2025年10月02日 23:03:07 +02:00

You were right about the redundant identities. cutting that out reduced connect times by a second.

Benchmark 1: ssh git@codeberg.org
 Time (mean ± σ): 2.931 s ± 0.113 s [User: 0.062 s, System: 0.007 s]
 Range (min ... max): 2.852 s ... 3.060 s 3 runs

I discovered the codeberg.org server is in berlin, and I'm connecting from west coast, USA about 250ms latency . 5 roundtrips add about 1250ms latency, which puts performance 500-600ms above github for comparison . I no longer have enough evidence of startup time configuration concerns.

I'll leave this open for a few days to see if anyone else in europe has latency issues, and there are good debugging tips.

You were right about the redundant identities. cutting that out reduced connect times by a second. ``` Benchmark 1: ssh git@codeberg.org Time (mean ± σ): 2.931 s ± 0.113 s [User: 0.062 s, System: 0.007 s] Range (min ... max): 2.852 s ... 3.060 s 3 runs ``` I discovered the codeberg.org server is in berlin, and I'm connecting from west coast, USA about 250ms latency . 5 roundtrips add about 1250ms latency, which puts performance 500-600ms above github for comparison . I no longer have enough evidence of startup time configuration concerns. I'll leave this open for a few days to see if anyone else in europe has latency issues, and there are good debugging tips.

tonymet commented 2025年10月02日 23:06:24 +02:00

Major discoveries:

1. https is much faster at connect (and probably transfer)
2. be careful about configured IdentityFile, IdentitiesOnly, IdentityAgent . Also UserKnownHostsFile & GlobalKnownHostsFile
3. check roundtrip latency
4. ssh likely deserves attention, but I can't diagnose the issue from here

Major discoveries: 1. https is much faster at connect (and probably transfer) 2. be careful about configured `IdentityFile`, `IdentitiesOnly`, `IdentityAgent` . Also `UserKnownHostsFile` & `GlobalKnownHostsFile` 3. check roundtrip latency 4. ssh likely deserves attention, but I can't diagnose the issue from here

tonymet commented 2025年10月09日 20:34:40 +02:00

@0ko thanks for helping debug this. I'll close this for now and reopen if I have better repro case.

@0ko thanks for helping debug this. I'll close this for now and reopen if I have better repro case.