SSH client connection delay on LAN when Windows Firewall is enabled

Question

The problem appeared once I upgraded my dev computer to Win11, which was running Win10 before that.

• My local server is running Ubuntu 22.04.
• Win11 ssh clients: putty, WSL2 ssh, both experience 20 second delay when connecting to the server on LAN.
• However, the same clients connect to WAN server with no delays.
• Delay is the same for password or keyfile connections.
• Two different android clients from two different devices connect to the same server/port without any issues or delays.

What I've tried so far:

• SSH port number is irrelevant, same issue on 22 or any other.
• Enabling/Disabling VPN on both server/win11 client has no effect, same delay, while Android connects fine with VPN on both server/client or no VPN.

Initially, I suspected server config, tried all the common solutions, UseDNS is set to no, and to summarize, none of the resolutions listed here help - https://jrs-s.net/2017/07/01/slow-ssh-logins/.

• Adding a new firewall rule in windows firewall for putty, didn't help, since the connection is not blocked, just that delay.

• Also, for testing, trying to find offending rule, I simply disabled all of the active firewall rules, no change, the same 20 second delay. Disabling the firewall does make the issue go away.

• Tried editing Group Policies as well, using GPE(Group Policy Editor). Not sure I understand why inbound rules under Local Computer Policy/Computer Configuration/Windows Settings/Security Settings/Windows Defender Firewall/WDF Local Group is empty when firewall settings from windows settings show large number of rules, but anyway, adding inbound/outbound rules for putty in there didn't resolve the issue either.

• Bypassed network switch, some posts mentioned quos as a potential problem, directly connecting both comps to router, same result.

• Created Win10 VM and used ssh from admin terminal, same delay.

At this point I've run out of ideas to investigate.

If I simply disable windows defender firewall the issue goes away. That's not really an option.

No errors are logged on either side server or client. Log outputs when the delay occurs: On the client side last line before delay:

debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5

And the corresponding line form the server auth log(syslog has nothing):

GatorNas sshd[1655346]: debug1: inetd sockets after dupping: 4, 4

20 seconds later, all works, I get the password prompt, connection is established, speeds are ok etc.

I'm really puzzled, why LAN connection is delayed while connecting to the server on the inet is subsecond...

Updates:

• Capturing packets on Win11, to see LLNMR & mDNS filtering by udp 5355 & udp 5353. Nothing.
• Capturing packets on Ubuntu box, same filter as above for wireshark, no traffic.

Using filter (udp.port == 5355) || (udp.port == 5353). Any advise on how to filter out network traffic to narrow down to something useful would be helpful.

Found the reason for the delay: More packet capture revealed port ident issue on port 113. Server sends request on 113, then that's followed by 3 TCP re-transmissions, hence the delay.

Still can't figure out:

1. Why the issue only manifests on LAN
2. More importantly, opening port 113 in the windows firewall for LAN has no effect :(

Workaround: Reject all traffic on port 113 on the ssh server box.

• grawity: Have you looked at a packet capture yet? Is either the client or possibly the server waiting on some external query, or trying to resolve hostnames via LLMNR/mDNS?
• GatorSU: Not yet, not exactly my area of expertise :( I'm trying to et up wireshark to monitor client, I suppose that's where the problem is.
• grawity: Try capturing for all traffic, not just those specific examples, and take a look at other interfaces on the client as well (e.g. VPN interfaces or such).
• grawity: Also: do you have any active outbound "deny" rules on the machine?
• GatorSU: Updated OG post. It's the ident request on 113 from server that causes the delay. I can see 3 retransmissions for it until it fails and gives up. Unfortunately opening the 113 port in WF has no effect, but wireshark clearly shows the culprit. No outbound blocked ports as far as I can tell. I tried to intentionally block a port 2725 use in the tests, ssh goes through fine, even though firewall blocks it theoretically. Somehow I have a feeling nothing I do in that firewall has any effect

Answer 1

Your SSH server is configured to make an Ident request back to the connecting client, in order to learn the client's OS username (presumably for logging purposes).

Why the issue only manifests on LAN

The LAN server is the only one that has been specifically configured to make Ident queries. This is not the default (it requires some quite deliberate configuration to end up in this situation) and your WAN server doesn't do this.

I haven't seen any SSH server that would make Ident queries on its own (OpenSSH certainly doesn't), but from your logs it sounds like your sshd is run through inetd instead of being a standalone service as usual. (Inetd and xinetd are "super-server" services that wait for connections and start the actual sshd/ftpd/telnetd/etc. service on demand; this is also known as "socket activation" in systemd or launchd.)

Specifically, the server is probably running xinetd and has an ssh service configured in /etc/xinetd.conf (or somewhere in /etc/xinetd.d/ssh*). If you find that, look for a setting that enables USERID logging – if found, remove it and restart xinetd. It might be something like this:

service ssh
{
 [...]
 log_on_success += HOST USERID
 log_on_failure += HOST USERID
}

More importantly, opening port 113 in the windows firewall for LAN has no effect :(

It's very rare for any computer to run an Ident responder these days (IRC is the only place that has seen any kind of significant use for it), but normally the incoming connection to port 113 would be immediately rejected using a TCP RST packet and the server would carry on. Similarly, firewalls can be set up to block connections by explicitly rejecting them.

Windows Firewall, however, runs in "Stealth mode" by default where it ignores connection attempts to ports that have no listening socket, so the server is forced to wait for the ~30s TCP connect timeout. This happens even if you explicitly allow connections to port 113, as the "Stealth mode" feature applies to all requests on unused ports whether the rules allow that traffic or not.

So as long as nothing is listening on port 113 and the firewall is active, the request will be quietly ignored (instead of the correct TCP RST rejection packet being sent).

While stopping the server from making Ident queries in the first place would be a better option, you can also disable "Stealth" mode in Windows through Registry (it's unlikely to actually be as important as it is claimed – blocked requests will still continue to be blocked):

• Cmd:

  for %p in (Public Private Domain Standard) do (
   reg add HKLM\SOFTWARE\Policies\Microsoft\WindowsFirewall\%pProfile /v DisableStealthMode /t REG_DWORD /d 1
  )
  

• GatorSU: Thanks. I'd rather fix the server than mess with unsupported windows stealth. There's no xinetd.conf anywhere in etc. Also, I tried to check the status, systemctl status xinetd and inetd, neither is found. I didn't configure anything specific for ssh server on ubuntu during the upgrade, just changed the default port. As far as I understand ubuntu 22 uses networkmanager, no idea how's that related to those two services though. Any pointers what/where should i look for?
• GatorSU: As indicated in several sources, some linux distros dont' have inetd or xinetd, which seems to be the case with my ubuntu 22.04. Instead systemd performs similar functions using systemd-logind for logins, logs and associated management. In my authlog I can see systemd-logind entry with ssh client login username. So, that's inline. I can't however find anything in systemd-logind config that's control userid log or access to inetd or anything like that. Tried more generic research how to prevent SSH server access to ident, no dice..
• GatorSU: Workaround found - added firewall rule on ubuntu box, reject all traffic on 113. Not quite sure what else will break, but my ssh, putty and most importantly git connect fast to ubuntu now... I suppose there's some way to figure out who's pinging 113 on ssh connect since it's not xinetd and how to disable.