Unable to ssh into remote server, despite port open and server reachable

Question

As the title says, I am trying to create a home Linux server, and everything was going fine, including when I ssh into the server over LAN from both my laptop and my desktop.

I am a total beginner when it comes to networking and server administration, but have dabbled in Linux for a while now.

The server is an old MacBook from like 2014 with Linux server 22.04 installed. I connected it to my ISP router via a usb-to-ethernet adapter, since there is no Ethernet port on the MacBook.

Things grinded to a halt when I tried to ssh over WAN using my laptop connected to my phone's internet. I systematically get a timeout, no matter what I try.

I have been on this issue for a few days now, and I'm starting to lose it a little. Any suggestions as to what could cause the timeout? Is it a Firewall issue? Could it come from the USB to ethernet adapter? I have set up a different port for ssh over LAN, is that the problem? The only thing I haven't tried from the online recommandations is to create a VPN. Is that going to solve my problem?

What I've tried so far:

I ssh to a previously forwarded port

$ ssh -vvv -p FORWARDEDPORTNUMBER [email protected]

Output:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname public.ip.of.my.router is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/someguy/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/someguy/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to public.ip.of.my.router [public.ip.of.my.router] port FORWARDEDPORTNUMBER .
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address public.ip.of.my.router port FORWARDEDPORTNUMBER : Connection timed out
ssh: connect to host public.ip.of.my.router port FORWARDEDPORTNUMBER : Connection timed out

Is it portforwarding done wrong?

Nope, using tools such as https://www.yougetsignal.com/tools/open-ports/, The port is indeed open

~$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:6968            0.0.0.0:*               LISTEN      961/sshd: /usr/sbin
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      817/systemd-resolve
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      1002/smbd
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      1002/smbd
tcp        0      0 0.0.0.0:1445            0.0.0.0:*               LISTEN      961/sshd: /usr/sbin
tcp6       0      0 :::FORWARDEDPORTNUMBER  :::*                    LISTEN      961/sshd: /usr/sbin
tcp6       0      0 :::139                  :::*                    LISTEN      1002/smbd
tcp6       0      0 :::80                   :::*                    LISTEN      878/apache2
tcp6       0      0 :::445                  :::*                    LISTEN      1002/smbd
tcp6       0      0 :::1445                 :::*                    LISTEN      961/sshd: /usr/sbin

Does it come from the firewall?

I have tried disabling the server's firewall completely (I use ufw, so sudo ufw disable does the trick), as well as going on the lowest possible security on my ISP router

Pinging the server works just fine

Telnet also works

I think? I'm not 100% sure of what is going on here, I was just following recommendations online and find this

$curl -v telnet://public.ip.of.my.router:FORWARDEDPORTNUMBER 
*   Trying public.ip.of.my.router:FORWARDEDPORTNUMBER...
* Connected to public.ip.of.my.router (public.ip.of.my.router) port FORWARDEDPORTNUMBER (#0)
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4
some user input
Invalid SSH identification string.
some more user input
* Closing connection 0

nmapping shows the port is filtered

nmap public.ip.of.my.router -pFORWARDEDPORTNUMBER 
Starting Nmap 7.94 ( https://nmap.org ) at 2023-10-05 19:50 CEST
Nmap scan report for public.ip.of.my.router
Host is up (0.19s latency).

PORT                    STATE   SERVICE
FORWARDEDPORTNUMBER/tcp filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.09 seconds

ssh configuration

(/etc/ssh/sshd_config) is listening to the correct port ssh key authentification

I use ssh key authentification that works just fine over LAN on the same laptop

Answer 1

Do you access to your ISP's router interface? My best guess is that you will not only need forwarding rules for the firewall, but also a matching NAT rule. When I add a firewall rule to pfSense, it adds a corresponding NAT rule.

Answer 2

Found the culprit, it was my ISP router. Turns out there is another setting which is specific to this router, and I couldn't find anything about it online.