Push mirror by ssh key fails when remote host changes IP #2067

mattdm commented 2025年08月06日 00:44:50 +02:00

Comment

My host provider migrated my system to a different host, and the IP changed. I have a Codeberg repo set up to push mirror on commit to the host. (Actual hostname replaced with example.org below.)

push failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo.
Please contact your system administrator.
Add correct host key in /home/git/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/git/.ssh/known_hosts:37
 remove with:
 ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org'
Host key for example.org has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
 - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo.
Please contact your system administrator.
Add correct host key in /home/git/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/git/.ssh/known_hosts:37
 remove with:
 ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org'
Host key for example.org has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
 - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo.
Please contact your system administrator.
Add correct host key in /home/git/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/git/.ssh/known_hosts:37
 remove with:
 ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org'
Host key for example.org has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

I can't find any way to clear this. The manual "Synchronize Now" button doesn't do it. Removing and replacing the sync config doesn't help either.

I worked around the issue by using a different hostname that resolves to the same IP, but that's not really a great solution.

### Comment My host provider migrated my system to a different host, and the IP changed. I have a Codeberg repo set up to push mirror on commit to the host. (Actual hostname replaced with `example.org` below.) ``` push failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo. Please contact your system administrator. Add correct host key in /home/git/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /home/git/.ssh/known_hosts:37 remove with: ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org' Host key for example.org has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo. Please contact your system administrator. Add correct host key in /home/git/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /home/git/.ssh/known_hosts:37 remove with: ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org' Host key for example.org has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. - @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is SHA256:oV9BA0yBGsARD4JOD8DxQ4AlGBJtF0o0rsYHmE4chOo. Please contact your system administrator. Add correct host key in /home/git/.ssh/known_hosts to get rid of this message. Offending ED25519 key in /home/git/.ssh/known_hosts:37 remove with: ssh-keygen -f '/home/git/.ssh/known_hosts' -R 'example.org' Host key for example.org has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` I can't find any way to clear this. The manual "Synchronize Now" button doesn't do it. Removing and replacing the sync config doesn't help either. I worked around the issue by using a different hostname that resolves to the same IP, but that's not really a great solution.

Gusted commented 2025年08月06日 01:04:55 +02:00

@mattdm wrote in #2067 (comment):

My host provider migrated my system to a different host, and the IP changed

The SSH host keys aren't bound to IP. It does mean that your host provider didn't migrate the SSH host keys to the new host which is quite bad.

Removing a SSH host key is quite serious, however as no other push mirror is configured to that remote address I will remove it.

@mattdm wrote in https://codeberg.org/Codeberg/Community/issues/2067#issue-2004469: > My host provider migrated my system to a different host, and the IP changed The SSH host keys aren't bound to IP. It does mean that your host provider didn't migrate the SSH host keys to the new host which is quite bad. Removing a SSH host key is quite serious, however as no other push mirror is configured to that remote address I will remove it.

Gusted commented 2025年08月06日 01:06:37 +02:00

It's removed.

It's removed.

mattdm commented 2025年08月06日 16:14:18 +02:00

Thank you! Unfortunately, it's common for budget hosting providers to do this. They just do a whole reinstall, and often when they do a hardware refresh they consolidate more small sites on a single system. Is the underlying known-hosts file shared across codeberg, or is it per repo?

Thank you! Unfortunately, it's common for budget hosting providers to do this. They just do a whole reinstall, and often when they do a hardware refresh they consolidate more small sites on a single system. Is the underlying known-hosts file shared across codeberg, or is it per repo?

Gusted commented 2025年08月06日 16:14:59 +02:00

It's shared across codeberg.

It's shared across codeberg.

mattdm commented 2025年08月06日 19:41:15 +02:00

Ah, then, the concern makes sense. I'm not familiar enough with the architecture to make a meaningful suggestion, but it seems like maybe UserKnownHostsFile could be set to a repo-specific value or something...

Anyway, thank you for your help!

Ah, then, the concern makes sense. I'm not familiar enough with the architecture to make a meaningful suggestion, but it seems like maybe UserKnownHostsFile could be set to a repo-specific value or something... Anyway, thank you for your help!

Gusted commented 2025年08月06日 19:50:02 +02:00

Forgejo tries to mitigate SSH's trust on first use security model as much as possible by sharing the SSH hostkeys between all repositories. An admin needing to remove the host key is more or less what should have happened in this situation.

Forgejo tries to mitigate SSH's trust on first use security model as much as possible by sharing the SSH hostkeys between all repositories. An admin needing to remove the host key is more or less what should have happened in this situation.