Router with nftables doesn't work well

Question

I have Debian 4.19.194-1 as a router server with LAN, WAN, PPPOE (as gateway) and COMPUTER1 in LAN network which should have access to internet through Debian router.

As firewall I use nftables with rules:

#!/usr/sbin/nft -f
flush ruleset

define EXTIF = "ppp0"
define LANIF = "enp1s0"
define WANIF = "enp4s0"
define LOCALIF = "lo"

table firewall {
    chain input {
        type filter hook input priority 0

        ct state {established, related} counter accept
        ct state invalid counter drop

        ip protocol icmp counter accept
        ip protocol igmp counter accept comment "Accept IGMP"
        ip protocol gre counter accept comment "Accept GRE"

        iifname { $LOCALIF, $LANIF } counter accept
        tcp dport 44122 counter accept
        udp dport 11897 counter accept
        udp dport 1194 counter accept
        udp dport {67,68} counter accept comment "DHCP"
        counter reject
    }

    chain forwarding {
        type filter hook forward priority 0

        # teleguide.info for ntf monitor
        ip daddr 46.29.166.30 meta nftrace set 1 counter accept
        ip saddr 46.29.166.30 meta nftrace set 1 counter accept

        udp dport 1194 counter accept
        tcp dport 5938 counter accept
        udp dport 5938 counter accept

        ip daddr 10.10.0.0/24 counter accept
        ip saddr 10.10.0.0/24 counter accept
        ip protocol gre counter accept comment "Accept GRE Forward"

        counter drop comment "all non described Forward drop"
    }

    chain outgoing {
        type filter hook output priority 0
        oifname $LOCALIF counter accept
    }
}

table nat {
    chain prerouting {
        type nat hook prerouting priority 0
        iifname $EXTIF udp dport 1194 counter dnat to 10.10.0.4
    }

    chain postrouting {
        type nat hook postrouting priority 0
        ip saddr 10.10.0.0/24 oifname $EXTIF counter masquerade
    }
}

lsmod:

tun                    53248  2
pppoe                  20480  2
pppox                  16384  1 pppoe
ppp_generic            45056  6 pppox,pppoe
slhc                   20480  1 ppp_generic
binfmt_misc            20480  1
i915                 1736704  0
ppdev                  20480  0
evdev                  28672  2
video                  49152  1 i915
drm_kms_helper        208896  1 i915
iTCO_wdt               16384  0
iTCO_vendor_support    16384  1 iTCO_wdt
parport_pc             32768  0
coretemp               16384  0
sg                     36864  0
serio_raw              16384  0
pcspkr                 16384  0
drm                   495616  3 drm_kms_helper,i915
parport                57344  2 parport_pc,ppdev
i2c_algo_bit           16384  1 i915
rng_core               16384  0
button                 20480  0
nft_masq_ipv4          16384  3
nft_masq               16384  1 nft_masq_ipv4
nft_reject_ipv4        16384  1
nf_reject_ipv4         16384  1 nft_reject_ipv4
nft_reject             16384  1 nft_reject_ipv4
nft_counter            16384  25
nft_ct                 20480  2
nft_connlimit          16384  0
nf_conncount           20480  1 nft_connlimit
nf_tables_set          32768  3
nft_tunnel             16384  0
nft_chain_nat_ipv4     16384  2
nf_nat_ipv4            16384  2 nft_chain_nat_ipv4,nft_masq_ipv4
nft_nat                16384  1
nf_tables             143360  112 nft_reject_ipv4,nft_ct,nft_nat,nft_chain_nat_ipv4,nft_tunnel,nft_counter,nft_masq,nft_connlimit,nft_masq_ipv4,nf_tables_set,nft_reject
nf_nat                 36864  2 nft_nat,nf_nat_ipv4
nfnetlink              16384  1 nf_tables
nf_conntrack          172032  8 nf_nat,nft_ct,nft_nat,nf_nat_ipv4,nft_masq,nf_conncount,nft_connlimit,nft_masq_ipv4
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
ip_tables              28672  0
x_tables               45056  1 ip_tables
autofs4                49152  2
ext4                  745472  2
crc16                  16384  1 ext4
mbcache                16384  1 ext4
jbd2                  122880  1 ext4
fscrypto               32768  1 ext4
ecb                    16384  0
crypto_simd            16384  0
cryptd                 28672  1 crypto_simd
glue_helper            16384  0
aes_x86_64             20480  1
raid10                 57344  0
raid456               172032  0
async_raid6_recov      20480  1 raid456
async_memcpy           16384  2 raid456,async_raid6_recov
async_pq               16384  2 raid456,async_raid6_recov
async_xor              16384  3 async_pq,raid456,async_raid6_recov
async_tx               16384  5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov
xor                    24576  1 async_xor
raid6_pq              122880  3 async_pq,raid456,async_raid6_recov
libcrc32c              16384  3 nf_conntrack,nf_nat,raid456
crc32c_generic         16384  5
raid0                  20480  0
multipath              16384  0
linear                 16384  0
raid1                  45056  2
md_mod                167936  8 raid1,raid10,raid0,linear,raid456,multipath
sd_mod                 61440  6
ata_generic            16384  0
ata_piix               36864  4
libata                270336  2 ata_piix,ata_generic
psmouse               172032  0
scsi_mod              249856  3 sd_mod,libata,sg
ehci_pci               16384  0
i2c_i801               28672  0
uhci_hcd               49152  0
lpc_ich                28672  0
ehci_hcd               94208  1 ehci_pci
mfd_core               16384  1 lpc_ich
usbcore               299008  3 ehci_pci,ehci_hcd,uhci_hcd
r8169                  90112  0
realtek                20480  2
libphy                 77824  2 r8169,realtek
usb_common             16384  1 usbcore

ntf monitor trace(verdict accept everywhere):

trace id 2c2a8923 ip firewall forwarding packet: iif "enp1s0" oif "ppp0" ether saddr xxx ether daddr xxx ip saddr 10.10.0.96 ip daddr 46.29.166.30 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 32611 ip length 52 tcp sport 62489 tcp dport https tcp flags == syn tcp window 8192 
trace id 2c2a8923 ip firewall forwarding rule ip daddr 46.29.166.30 nftrace set 1 counter packets 0 bytes 0 accept (verdict accept)
trace id 2c2a8923 ip nat postrouting packet: oif "ppp0" @ll,xxx ip saddr 10.10.0.96 ip daddr 46.29.166.30 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 32611 ip length 52 tcp sport 62489 tcp dport https tcp flags == syn tcp window 8192 
trace id 2c2a8923 ip nat postrouting rule ip saddr 10.10.0.0/24 oifname "ppp0" counter packets 0 bytes 0 masquerade (verdict accept)
trace id 73f8f405 ip firewall forwarding packet: iif "ppp0" oif "enp1s0" ip saddr 46.29.166.30 ip daddr 10.10.0.96 ip dscp af32 ip ecn not-ect ip ttl 58 ip id 0 ip length 52 tcp sport https tcp dport 62489 tcp flags == 0x12 tcp window 29200 
trace id 73f8f405 ip firewall forwarding rule ip saddr 46.29.166.30 nftrace set 1 counter packets 0 bytes 0 accept (verdict accept)
trace id ca8ec4f5 ip firewall forwarding packet: iif "enp1s0" oif "ppp0" ether saddr xxx ether daddr xxx ip saddr 10.10.0.96 ip daddr 46.29.166.30 ip dscp cs0 ip ecn not-ect ip ttl 127 ip id 32612 ip length 40 tcp sport 62489 tcp dport https tcp flags == ack tcp window 256 

And I don't know why, but some sites work fine from COMPUTER1, but some not with such rules.

For example: https://google.com works well from server and from computer1, but https://teleguide.info works well from server(wget), but not works from computer1.

Any idea whats wrong?

Answer 1

The firewall rules did not cause the problem. Instead, it's due to the MTU difference in "plain" Ethernet and PPPoE. Since PPP header takes up (at least) 8 bytes, and the usual MTU of Ethernet itself is 1500 bytes, the MTU of PPPoE in that case will be at most 1492 bytes.

I don't know MTU stuff well enough to tell the details, but as far as I know, if the TCP SYN packet advertise MSS to be larger than what can fit into the MTU of the interface that the replies will come in through, the replying traffic could end up having trouble from actually getting in.

AFAIK, the reason it works fine with the router/server itself is that, the MSS is derived from the MTU of its outbound interface (ppp0), while on the other hand, COMPUTER1's outbound interface is plain Ethernet.

For TCP traffics, one can workaround the problem by having a rule in a hook forwarding chain:

tcp flags syn tcp option maxseg size set 1452

1452 comes from 1500 - 8 - 40, where the 40 is the size of a IPv4 header. For IPv6 you may need 1500 - 8 - 60 = 1432.

You might need to have the rule ordered before any accept rules. (It could depend on the whole structure of the ruleset though, I think.)

P.S. Not sure if you need any measure for UDP traffics.

---

Alternatively, you can probably just set the MTU of the Ethernet interfaces of all the LAN "clients" of this "router" (and that of its LANIF) to 1492. It's probably a less "workaround" approach, but could be quite a hassle.