How to find out what service is listening on a specific port of a Ubuntu server without proccess id?

Question

I have decided to post this question even though there are a lot of similar questions, but none of them answered mine.

1. I periodically check the ports my server listens to.
2. My ubuntu OS by the output of lsb_relase -a:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:    20.04
Codename:   focal

3. I run the command netstat -tulpn4 and the output shows an unknown port:

tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      580/systemd-resolve 
tcp        0      0 0.0.0.0:62176           0.0.0.0:*               LISTEN      900/sshd: /usr/sbin 
tcp        0      0 0.0.0.0:1122            0.0.0.0:*               LISTEN      1158/sshd: username    
tcp        0      0 0.0.0.0:3333            0.0.0.0:*               LISTEN      1158/sshd: username    
udp        0      0 0.0.0.0:51820           0.0.0.0:*                           -                   
udp        0      0 127.0.0.53:53           0.0.0.0:*                           580/systemd-resolve 
udp        0      0 X.X.X.X:1194           0.0.0.0:*                           870/openvpn         

1. I have used many tools and guides to find out why my server is listening on port 51820 without finding an answer.
2. I have tried:
   • sudo lsof -i :51820 - and got an empty output.
   • sudo netstat -ltnp | grep -w ':51820' - and got an empty output.
   • sudo fuser 51820/udp - and got an empty output.
   • sudo netstat -peanut | grep ":51820" - the output is:

udp        0      0 0.0.0.0:51820           0.0.0.0:*                           0          26262      -                   
udp6       0      0 :::51820                :::*                                0          26263      - 

• sudo ss -nlp | grep 51820 the output is:

udp     UNCONN   0        0                                             0.0.0.0:51820                                             0.0.0.0:*                                                                                                     
udp     UNCONN   0        0                                                [::]:51820                                                [::]:*  

3. I have looked at the following answers and did not find a solution:
   • How to know what program is listening on a given port?
   • 3 Ways to Find Out Which Process Listening on a Particular Port
   • How to know what program is listening on a given port?
   • How can we find which process is using a particular port? [duplicate]

After searching the web about port 51820 I found out it is the "wireguard" I have installed that`s listening on that port, but the question remains the same as in any command I have tried I could not find that out.

Answer 1

You're running operating system that implements Wireguard as a kernel module. Your kernel is listening on this port, and since there's no process owning that socket no PID is reported by netstat.

Answer 2

Genreal:

After a two days search I did around the issue, I found like Michael Hampton and Peter Zhabin that there is no existing solution which shows a kernel process id via a listening port.

In addition during those two days i searched also for some commands combinations that can bring the wanted answer, and found none simple or convenient way to do this.

The solution I created is preliminary and I am sure it can be improved by the community members.

Discounts For Locating The Process

• The process is running on the kernel level or any other level which avoiding from the process to have an id (PID).
• The process id or the process program was not found by the output of the given commands: lsof, netstat, ss, fuser.
• We did found a listening port from the output of the above commands - But the we cannot configure the program or the pid that causing the listening.

About grep:

We will use grep to find more information about the open port.

grep - print lines that match patterns.

From grep man page via man grep command.

DESCRIPTION
grep  searches  for  PATTERNS  in  each  FILE.  PATTERNS is one or more
       patterns separated by newline characters, and  grep  prints  each  line
       that  matches a pattern.  Typically PATTERNS should be quoted when grep
       is used in a shell command.

• Here is a great topic of how use the command correctly and effectively.

How To Locate The PID Or The Program That Using The Given Port Via grep:

In my case executing sudo grep --exclude-dir={sys,proc} -rnw / -e 51820 | grep -i port solved the issue and showed allot information about the program that using the port.

The given output:

iptables.service:6:ExecStart=/usr/sbin/iptables -I INPUT -p udp --dport 51820 -j ACCEPT
/home/username/wireguard-install.sh:238:    read -p "Port [51820]: " port
/home/username/wireguard-install.sh:241:        read -p "Port [51820]: " port
/home/username/wireguard-install.sh:243:    [[ -z "$port" ]] && port="51820"
/usr/share/doc/netplan/examples/wireguard.yaml:9:      port: 51820
/etc/wireguard/wg0.conf:8:ListenPort = 51820
/etc/systemd/system/wg-iptables.service:6:ExecStart=/usr/sbin/iptables -I INPUT -p udp --dport 51820 -j ACCEPT
/etc/systemd/system/wg-iptables.service:10:ExecStop=/usr/sbin/iptables -D INPUT -p udp --dport 51820 -j ACCEPT

The reason for the many flags is that other combinations i have tried had a large amount of unwanted output.

What The Arguments Of grep Command Stands For:

• --exclude-dir - Skip any command-line directory with a name suffix that matches the pattern.

And specific ignoring sys and proc directories in our specific case is to avoid unwanted output.

Example: grep --exclude-dir={dir1,dir2} will avoid dir1 and dir2 during the search.

• -r or -R is recursive.
• -n is line number.
• -w stands for match the whole word.
• / stands for the "highest" directory to start the search from top to bottom.
• -e - is the pattern used during the search.
• 51820 in our specific case is the port number that was found by one of the network monitoring command above.
• | - is the pipe to redirect the output of the first command part to the second one.

In our case: redirecting sudo grep --exclude-dir={sys,proc} -rnw / -e 51820 output to the next command grep -i port

• -i - Ignore case distinctions in patterns and input data, so that characters that differ only in case match each other.
• port - Found in order to narrow the results to the purpose for which we performed the search, finding more information about the specific port that was defined after the -e flag.

Tips:

• Make the scan largest as possible in the first steps by starting from the / directory, and using minimum flags to filter the output, to ensure you won`t miss any detail which we could achieve.
• After finding the wanted output or having problems with finding the wanted output caused by allot of unwanted output, start adding flags one by one.
• Specify the port number as the pattern, after all this is our starting point and our ending goal.
• Use double grep commands redirecting the first scan into port pattern filter, it can pinpoint us and speed up the solution, after all we are looking for a number as a pattern and this can lead to many unwanted results.
• If you cannot get into conclusions with the given output, make a search over the web with a chosen key-words that you have found.