0

Here is some background information first, although it may not be relevant to the problem.

I am learning Kubernetes and I set up a cluster where pi-hole runs as a service. I can access the admin console from a browser running on another computer on my LAN.

Here are the services:

$ sudo kubectl get services --all-namespaces -o wide
NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)                      AGE   SELECTOR
default       kubernetes       ClusterIP      10.43.0.1       <none>                                      443/TCP                      70m   <none>
kube-system   kube-dns         ClusterIP      10.43.0.10      <none>                                      53/UDP,53/TCP,9153/TCP       70m   k8s-app=kube-dns
kube-system   metrics-server   ClusterIP      10.43.78.148    <none>                                      443/TCP                      69m   k8s-app=metrics-server
pihole        pihole-dhcp      NodePort       10.43.41.144    <none>                                      67:31097/UDP                 69m   app=pihole,release=pihole
pihole        pihole-dns-tcp   LoadBalancer   10.43.36.226    192.168.1.129,192.168.1.146,192.168.1.148   53:31213/TCP                 69m   app=pihole,release=pihole
pihole        pihole-dns-udp   LoadBalancer   10.43.191.104   192.168.1.129,192.168.1.146,192.168.1.148   53:31078/UDP                 69m   app=pihole,release=pihole
pihole        pihole-web       LoadBalancer   10.43.63.236    192.168.1.129,192.168.1.146,192.168.1.148   80:30081/TCP,443:31214/TCP   69m   app=pihole,release=pihole
kube-system   traefik          LoadBalancer   10.43.122.130   <pending>                                   80:30146/TCP,443:32021/TCP   68m   app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

Here is the mystery - who is listening on port 80? As I mentioned earlier, I can establish the connection to pi-hole from the browser or curl:

$ curl http://192.168.1.129
    <!doctype html>
    <html lang='en'>

HTML lines omitted...

    </html>

Yet, the following commands are executed from 192.168.1.129:


$ sudo lsof -i :80
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:5f:01:b8:fc:6f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.129/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0
       valid_lft 43882sec preferred_lft 33082sec
    inet6 2603:8001:8e00:1ca9:37e7:e1f8:8ea9:571b/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 464540sec preferred_lft 464540sec
    inet6 fe80::aba5:f484:5575:9864/64 scope link
       valid_lft forever preferred_lft forever

Many other Kubernetes interface omitted...

netstat and ss do not show anybody listening on the port either. But I can see the traffic on the port with tcpdump if I run it while connecting to the server from the browser.

What am I missing?

Improve this question
3
  • 2
    Check the firewall's NAT table.
    – Daniel B
    Commented May 29, 2022 at 7:39
  • Thank you, I see a bunch of forwarding rules apparently inserted by Kubernetes installation, including one that is relevant to me. I never thought of looking there.
    – Marko
    Commented May 29, 2022 at 19:57
  • If you found the information you sought, consider adding an answer, maybe with some example rules. You can then accept your own answer after 48 hours.
    – Daniel B
    Commented May 29, 2022 at 20:29

1 Answer 1

Reset to default
0

After Daniel B. had pointed me to the right direction all I had to do was to follow the firewall rules. I found this detailed explanation https://www.stackrox.io/blog/kubernetes-networking-demystified/ on how do do that. I just had to replace iptables command with equivalent nft command because the former did not work on my system.

I started with the server 192.168.1.129 and port 80 that I hit with the browser:

$ sudo nft -n list chain nat KUBE-SERVICES|grep "192.168.1.129.*[^0-9]80[^0-9]"
                meta l4proto 6 ip daddr 192.168.1.129  tcp dport 80 counter packets 3 bytes 156 jump KUBE-FW-ZVTMEM247U444IPT

Then I simply followed the chain rules:

$ sudo nft -n list chain nat KUBE-FW-ZVTMEM247U444IPT
table ip nat {
        chain KUBE-FW-ZVTMEM247U444IPT {
                 counter packets 3 bytes 156 jump KUBE-XLB-ZVTMEM247U444IPT
                 counter packets 0 bytes 0 jump KUBE-MARK-DROP
                 counter packets 0 bytes 0 jump KUBE-XLB-ZVTMEM247U444IPT
                 counter packets 0 bytes 0 jump KUBE-MARK-DROP
                 counter packets 0 bytes 0 jump KUBE-XLB-ZVTMEM247U444IPT
                 counter packets 0 bytes 0 jump KUBE-MARK-DROP
        }
}
$ sudo nft -n list chain nat KUBE-XLB-ZVTMEM247U444IPT
table ip nat {
        chain KUBE-XLB-ZVTMEM247U444IPT {
                ip saddr 10.42.0.0/16  counter packets 0 bytes 0 jump KUBE-SVC-ZVTMEM247U444IPT
                 fib saddr type local counter packets 0 bytes 0 jump KUBE-MARK-MASQ
                 fib saddr type local counter packets 0 bytes 0 jump KUBE-SVC-ZVTMEM247U444IPT
                 counter packets 3 bytes 156 jump KUBE-SEP-PXNUQBE4P5SOTECD
        }
}
$ sudo nft -n list chain nat KUBE-SVC-ZVTMEM247U444IPT
table ip nat {
        chain KUBE-SVC-ZVTMEM247U444IPT {
                meta l4proto 6 ip saddr != 10.42.0.0/16 ip daddr 10.43.29.63  tcp dport 80 counter packets 0 bytes 0 jump KUBE-MARK-MASQ
                 counter packets 0 bytes 0 jump KUBE-SEP-PXNUQBE4P5SOTECD
        }
}

The last command output a familiar IP address - this is where my service runs on:

$ sudo kubectl get services --all-namespaces
NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP                                 PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1       <none>                                      443/TCP                      104m
kube-system   kube-dns         ClusterIP      10.43.0.10      <none>                                      53/UDP,53/TCP,9153/TCP       104m
kube-system   metrics-server   ClusterIP      10.43.8.179     <none>                                      443/TCP                      104m
pihole        pihole-dhcp      NodePort       10.43.156.246   <none>                                      67:31575/UDP                 84m
pihole        pihole-dns-udp   LoadBalancer   10.43.167.163   192.168.1.129,192.168.1.146,192.168.1.148   53:31671/UDP                 84m
pihole        pihole-dns-tcp   LoadBalancer   10.43.219.243   192.168.1.129,192.168.1.146,192.168.1.148   53:30437/TCP                 84m
pihole        pihole-web       LoadBalancer   10.43.29.63     192.168.1.129,192.168.1.146,192.168.1.148   80:32526/TCP,443:30932/TCP   84m
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .