1

Using MTR (sending TCP with or without SYN), I'm consistently seeing packet loss on one intermediate router in the path. UDP does the same. I'm use TCP to avoid the ICMP packet flood limits set.

I highly suspect the packet loss shown by MTR is not real, as many other tools, ie, TCPing.exe, hping3, PRTG QOS, do not show TCP loss (only ICMP loss, which is normal).

What can possibly account for MTR showing consistently different results than other tools?

Hop 2 is the Cisco router with ICMP flood limiting: and look at the ms times in MTR:

mtr -P 445 -T -rn 172.31.xx.5
Start: 2020-09-02T11:54:26+0800
HOST: xxxx                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.20.x.254               0.0%    10    0.6   0.7   0.5   1.9   0.4
  2.|-- 172.18.x.239              10.0%    10  7014. 3339.   0.2 7018. 3506.6
  3.|-- 172.31.x.32                0.0%    10    4.3   2.4   2.0   4.3   0.7
  4.|-- 172.18.x.211               0.0%    10    2.4   2.5   2.4   2.6   0.0
  5.|-- 172.31.x.5                 0.0%    10   81.4  81.7  81.4  82.5   0.4

hping:

sudo hping3 -q --fast -n -c 100 172.31.x.5 -p 445 -T
HPING 172.31.x.5 (ens160 172.31.x.5): NO FLAGS are set, 40 headers + 0 data bytes
hop=1 TTL 0 during transit from ip=172.20.x.254
hop=1 hoprtt=0.9 ms
hop=2 TTL 0 during transit from ip=172.18.x.239
hop=2 hoprtt=0.8 ms
hop=3 TTL 0 during transit from ip=172.31.x.32
hop=3 hoprtt=2.8 ms
hop=4 TTL 0 during transit from ip=172.18.x.211
hop=4 hoprtt=2.8 ms

--- 172.31.x.5 hping statistic ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 0.8/78.7/86.4 ms

TCPing.exe:

./tcping -i .1 -p 445 -n 50 172.31.x.5
Probing 172.31.x.5:445/tcp - Port is open - time=81.423ms
Probing 172.31.x.5:445/tcp - Port is open - time=81.375ms
Probing 172.31.x.5:445/tcp - Port is open - time=81.246ms

Ping statistics for 172.31.x.5:445
     50 probes sent.
     50 successful, 0 failed.  (0.00% fail)
Approximate trip times in milli-seconds:
     Minimum = 81.246ms, Maximum = 85.690ms, Average = 81.628ms

UPDATED with TCP DUMP:


[root@xxxxx ~]#  mtr -P 445 -T --show-ips --first-ttl 2 172.31.x.5
Start: 2020-09-07T09:38:14+0800
HOST: xxxxx.xxxxx                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  2.|-- 172.18.x.239              10.0%    10    0.2 556.8   0.2 3007. 1016.0
  3.|-- 172.31.x.32                0.0%    10   25.9   4.6   2.0  25.9   7.5
  4.|-- 172.18.x.211               0.0%    10    2.6   4.4   2.4  14.2   4.2
  5.|-- 172.31.x.5                 0.0%    10   92.8  91.4  81.5 106.6   8.7

and the dump:

[root@xxxx ~]# tcpdump -v -n -i any host 172.31.x.5
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
09:38:14.690912 IP (tos 0x0, ttl 2, id 49126, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33000 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xe0db), seq 804539254, win 29200, options [mss 1460,sackOK,TS val 2679764112 ecr 0,nop,wscale 7], length 0
09:38:14.791006 IP (tos 0x0, ttl 3, id 13695, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33001 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x9760), seq 49660298, win 29200, options [mss 1460,sackOK,TS val 2679764213 ecr 0,nop,wscale 7], length 0
09:38:14.891129 IP (tos 0x0, ttl 4, id 29895, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33002 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x529f), seq 2268955038, win 29200, options [mss 1460,sackOK,TS val 2679764313 ecr 0,nop,wscale 7], length 0
09:38:14.991229 IP (tos 0x0, ttl 5, id 29325, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33003 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x914f), seq 1117161264, win 29200, options [mss 1460,sackOK,TS val 2679764413 ecr 0,nop,wscale 7], length 0
09:38:15.083219 IP (tos 0x0, ttl 124, id 2115, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33003: Flags [S.], cksum 0x7916 (correct), seq 3845735035, ack 1117161265, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299321 ecr 2679764413], length 0
09:38:15.083257 IP (tos 0x0, ttl 5, id 29326, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33003 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0xc64d), ack 1, win 229, options [nop,nop,TS val 2679764505 ecr 354299321], length 0
09:38:15.083295 IP (tos 0x0, ttl 5, id 29327, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33003 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0xc64c), seq 1, ack 1, win 229, options [nop,nop,TS val 2679764505 ecr 354299321], length 0
09:38:15.083432 IP (tos 0x0, ttl 6, id 25182, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33004 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xf06b), seq 3590689863, win 29200, options [mss 1460,sackOK,TS val 2679764505 ecr 0,nop,wscale 7], length 0
09:38:15.182702 IP (tos 0x0, ttl 124, id 2116, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33003: Flags [.], cksum 0xc623 (correct), ack 2, win 260, options [nop,nop,TS val 354299331 ecr
2679764505], length 0
09:38:15.182724 IP (tos 0x0, ttl 124, id 2117, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33003: Flags [R.], cksum 0xd8ef (correct), seq 1, ack 2, win 0, length 0
09:38:15.183520 IP (tos 0x0, ttl 124, id 2118, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33004: Flags [S.], cksum 0x5d28 (correct), seq 3845438848, ack 3590689864, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299331 ecr 2679764505], length 0
09:38:15.183551 IP (tos 0x0, ttl 6, id 25183, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33004 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0xaa57), ack 1, win 229, options [nop,nop,TS val 2679764605 ecr 354299331], length 0
09:38:15.183573 IP (tos 0x0, ttl 6, id 25184, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33004 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0xaa56), seq 1, ack 1, win 229, options [nop,nop,TS val 2679764605 ecr 354299331], length 0
09:38:15.250672 IP (tos 0x0, ttl 2, id 17819, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33005 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xa4cf), seq 31896411, win 29200, options [mss 1460,sackOK,TS val 2679764672 ecr 0,nop,wscale 7], length 0
09:38:15.288713 IP (tos 0x0, ttl 124, id 2119, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33004: Flags [.], cksum 0xaa2d (correct), ack 2, win 260, options [nop,nop,TS val 354299341 ecr
2679764605], length 0
09:38:15.288730 IP (tos 0x0, ttl 124, id 2120, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33004: Flags [R.], cksum 0xbd67 (correct), seq 1, ack 2, win 0, length 0
09:38:15.417055 IP (tos 0x0, ttl 3, id 53057, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33006 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x83c4), seq 2642294822, win 29200, options [mss 1460,sackOK,TS val 2679764839 ecr 0,nop,wscale 7], length 0
09:38:15.583905 IP (tos 0x0, ttl 4, id 14265, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33007 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xf2c4), seq 2521354164, win 29200, options [mss 1460,sackOK,TS val 2679765005 ecr 0,nop,wscale 7], length 0
09:38:15.693956 IP (tos 0x0, ttl 2, id 49127, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33000 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xdcef), seq 804539254, win 29200, options [mss 1460,sackOK,TS val 2679765116 ecr 0,nop,wscale 7], length 0
09:38:15.750771 IP (tos 0x0, ttl 5, id 11215, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33008 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xb79e), seq 3439317883, win 29200, options [mss 1460,sackOK,TS val 2679765172 ecr 0,nop,wscale 7], length 0
09:38:15.835965 IP (tos 0x0, ttl 124, id 2121, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33008: Flags [S.], cksum 0x1083 (correct), seq 1022589272, ack 3439317884, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299397 ecr 2679765172], length 0
09:38:15.836007 IP (tos 0x0, ttl 5, id 11216, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33008 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0x5dc0), ack 1, win 229, options [nop,nop,TS val 2679765258 ecr 354299397], length 0
09:38:15.836042 IP (tos 0x0, ttl 5, id 11217, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33008 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0x5dbf), seq 1, ack 1, win 229, options [nop,nop,TS val 2679765258 ecr 354299397], length 0
09:38:15.920353 IP (tos 0x0, ttl 124, id 2122, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33008: Flags [.], cksum 0x5d98 (correct), ack 2, win 260, options [nop,nop,TS val 354299405 ecr
2679765258], length 0
09:38:15.920374 IP (tos 0x0, ttl 124, id 2123, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33008: Flags [R.], cksum 0x739f (correct), seq 1, ack 2, win 0, length 0
09:38:15.950953 IP (tos 0x0, ttl 2, id 36817, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33009 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x13e4), seq 3435099820, win 29200, options [mss 1460,sackOK,TS val 2679765373 ecr 0,nop,wscale 7], length 0
09:38:16.151206 IP (tos 0x0, ttl 3, id 49264, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33010 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x21f6), seq 4200282677, win 29200, options [mss 1460,sackOK,TS val 2679765573 ecr 0,nop,wscale 7], length 0
09:38:16.351417 IP (tos 0x0, ttl 4, id 50733, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33011 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x7b07), seq 2643868960, win 29200, options [mss 1460,sackOK,TS val 2679765773 ecr 0,nop,wscale 7], length 0
09:38:16.551648 IP (tos 0x0, ttl 5, id 39489, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33012 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x499f), seq 2848022932, win 29200, options [mss 1460,sackOK,TS val 2679765973 ecr 0,nop,wscale 7], length 0
09:38:16.633062 IP (tos 0x0, ttl 124, id 2124, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33012: Flags [S.], cksum 0x988a (correct), seq 278638425, ack 2848022933, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299477 ecr 2679765973], length 0
09:38:16.633099 IP (tos 0x0, ttl 5, id 39490, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33012 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0xe5cb), ack 1, win 229, options [nop,nop,TS val 2679766055 ecr 354299477], length 0
09:38:16.633138 IP (tos 0x0, ttl 5, id 39491, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33012 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0xe5ca), seq 1, ack 1, win 229, options [nop,nop,TS val 2679766055 ecr 354299477], length 0
09:38:16.715415 IP (tos 0x0, ttl 124, id 2125, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33012: Flags [.], cksum 0xe5a3 (correct), ack 2, win 260, options [nop,nop,TS val 354299485 ecr
2679766055], length 0
09:38:16.715445 IP (tos 0x0, ttl 124, id 2126, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33012: Flags [R.], cksum 0xff17 (correct), seq 1, ack 2, win 0, length 0
09:38:16.751857 IP (tos 0x0, ttl 2, id 41596, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33013 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x8036), seq 1964269281, win 29200, options [mss 1460,sackOK,TS val 2679766173 ecr 0,nop,wscale 7], length 0
09:38:16.952151 IP (tos 0x0, ttl 3, id 48974, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33014 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xc311), seq 1311261224, win 29200, options [mss 1460,sackOK,TS val 2679766374 ecr 0,nop,wscale 7], length 0
09:38:16.953945 IP (tos 0x0, ttl 2, id 36818, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33009 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x0ff9), seq 3435099820, win 29200, options [mss 1460,sackOK,TS val 2679766376 ecr 0,nop,wscale 7], length 0
09:38:17.152324 IP (tos 0x0, ttl 4, id 38011, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33015 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x6ad1), seq 2104650325, win 29200, options [mss 1460,sackOK,TS val 2679766574 ecr 0,nop,wscale 7], length 0
09:38:17.352609 IP (tos 0x0, ttl 5, id 53734, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33016 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x2f16), seq 4087951120, win 29200, options [mss 1460,sackOK,TS val 2679766774 ecr 0,nop,wscale 7], length 0
09:38:17.434182 IP (tos 0x0, ttl 124, id 2127, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33016: Flags [S.], cksum 0x41e4 (correct), seq 2618580909, ack 4087951121, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299557 ecr 2679766774], length 0
09:38:17.434218 IP (tos 0x0, ttl 5, id 53735, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33016 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0x8f25), ack 1, win 229, options [nop,nop,TS val 2679766856 ecr 354299557], length 0
09:38:17.434261 IP (tos 0x0, ttl 5, id 53736, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33016 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0x8f24), seq 1, ack 1, win 229, options [nop,nop,TS val 2679766856 ecr 354299557], length 0
09:38:17.517657 IP (tos 0x0, ttl 124, id 2128, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33016: Flags [.], cksum 0x8efd (correct), ack 2, win 260, options [nop,nop,TS val 354299565 ecr
2679766856], length 0
09:38:17.517686 IP (tos 0x0, ttl 124, id 2129, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33016: Flags [R.], cksum 0xabe2 (correct), seq 1, ack 2, win 0, length 0
09:38:17.552773 IP (tos 0x0, ttl 2, id 4552, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33017 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x8700), seq 3683642998, win 29200, options [mss 1460,sackOK,TS val 2679766974 ecr 0,nop,wscale 7], length 0
09:38:17.697977 IP (tos 0x0, ttl 2, id 49128, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33000 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xd51b), seq 804539254, win 29200, options [mss 1460,sackOK,TS val 2679767120 ecr 0,nop,wscale 7], length 0
09:38:17.753038 IP (tos 0x0, ttl 3, id 1519, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33018 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xe0be), seq 1779828072, win 29200, options [mss 1460,sackOK,TS val 2679767175 ecr 0,nop,wscale 7], length 0
09:38:17.953256 IP (tos 0x0, ttl 4, id 28439, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33019 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xc02d), seq 552496728, win 29200, options [mss 1460,sackOK,TS val 2679767375 ecr 0,nop,wscale 7], length 0
09:38:18.153445 IP (tos 0x0, ttl 5, id 12495, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33020 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xad9d), seq 1848980184, win 29200, options [mss 1460,sackOK,TS val 2679767575 ecr 0,nop,wscale 7], length 0
09:38:18.243652 IP (tos 0x0, ttl 124, id 2130, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33020: Flags [S.], cksum 0xf9ea (correct), seq 2220244380, ack 1848980185, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299637 ecr 2679767575], length 0
09:38:18.243691 IP (tos 0x0, ttl 5, id 12496, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33020 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0x4724), ack 1, win 229, options [nop,nop,TS val 2679767665 ecr 354299637], length 0
09:38:18.243737 IP (tos 0x0, ttl 5, id 12497, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33020 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0x4723), seq 1, ack 1, win 229, options [nop,nop,TS val 2679767665 ecr 354299637], length 0
09:38:18.325455 IP (tos 0x0, ttl 124, id 2131, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33020: Flags [.], cksum 0x46fb (correct), ack 2, win 260, options [nop,nop,TS val 354299646 ecr
2679767665], length 0
09:38:18.325478 IP (tos 0x0, ttl 124, id 2132, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33020: Flags [R.], cksum 0x675a (correct), seq 1, ack 2, win 0, length 0
09:38:18.353631 IP (tos 0x0, ttl 2, id 64122, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33021 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x8c62), seq 3369728165, win 29200, options [mss 1460,sackOK,TS val 2679767775 ecr 0,nop,wscale 7], length 0
09:38:18.553807 IP (tos 0x0, ttl 3, id 54456, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33022 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xb96f), seq 3163150111, win 29200, options [mss 1460,sackOK,TS val 2679767975 ecr 0,nop,wscale 7], length 0
09:38:18.553951 IP (tos 0x0, ttl 2, id 4553, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33017 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x8316), seq 3683642998, win 29200, options [mss 1460,sackOK,TS val 2679767976 ecr 0,nop,wscale 7], length 0
09:38:18.754092 IP (tos 0x0, ttl 4, id 3074, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33023 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xb053), seq 2475231346, win 29200, options [mss 1460,sackOK,TS val 2679768176 ecr 0,nop,wscale 7], length 0
09:38:18.954312 IP (tos 0x0, ttl 5, id 46512, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33024 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xc732), seq 3553276040, win 29200, options [mss 1460,sackOK,TS val 2679768376 ecr 0,nop,wscale 7], length 0
09:38:18.957938 IP (tos 0x0, ttl 2, id 36819, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33009 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x0825), seq 3435099820, win 29200, options [mss 1460,sackOK,TS val 2679768380 ecr 0,nop,wscale 7], length 0
09:38:19.060779 IP (tos 0x0, ttl 124, id 2133, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33024: Flags [S.], cksum 0x1659 (correct), seq 2668896181, ack 3553276041, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299717 ecr 2679768376], length 0
09:38:19.060825 IP (tos 0x0, ttl 5, id 46513, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33024 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0x6382), ack 1, win 229, options [nop,nop,TS val 2679768482 ecr 354299717], length 0
09:38:19.060859 IP (tos 0x0, ttl 5, id 46514, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33024 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0x6381), seq 1, ack 1, win 229, options [nop,nop,TS val 2679768482 ecr 354299717], length 0
09:38:19.061069 IP (tos 0x0, ttl 2, id 4374, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33025 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xbe5d), seq 2013729973, win 29200, options [mss 1460,sackOK,TS val 2679768483 ecr 0,nop,wscale 7], length 0
09:38:19.164968 IP (tos 0x0, ttl 124, id 2134, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33024: Flags [.], cksum 0x6357 (correct), ack 2, win 260, options [nop,nop,TS val 354299728 ecr
2679768482], length 0
09:38:19.164986 IP (tos 0x0, ttl 124, id 2135, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33024: Flags [R.], cksum 0x8739 (correct), seq 1, ack 2, win 0, length 0
09:38:19.261329 IP (tos 0x0, ttl 3, id 64924, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33026 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xdbd5), seq 1921252343, win 29200, options [mss 1460,sackOK,TS val 2679768683 ecr 0,nop,wscale 7], length 0
09:38:19.461561 IP (tos 0x0, ttl 4, id 19188, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33027 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x08d0), seq 4075834823, win 29200, options [mss 1460,sackOK,TS val 2679768883 ecr 0,nop,wscale 7], length 0
09:38:19.661831 IP (tos 0x0, ttl 5, id 64151, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33028 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x698d), seq 3787783532, win 29200, options [mss 1460,sackOK,TS val 2679769083 ecr 0,nop,wscale 7], length 0
09:38:19.759754 IP (tos 0x0, ttl 124, id 2136, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33028: Flags [S.], cksum 0x71a1 (correct), seq 398781904, ack 3787783533, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299788 ecr 2679769083], length 0
09:38:19.759790 IP (tos 0x0, ttl 5, id 64152, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33028 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0xbed2), ack 1, win 229, options [nop,nop,TS val 2679769181 ecr 354299788], length 0
09:38:19.759817 IP (tos 0x0, ttl 5, id 64153, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33028 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0xbed1), seq 1, ack 1, win 229, options [nop,nop,TS val 2679769181 ecr 354299788], length 0
09:38:19.851078 IP (tos 0x0, ttl 124, id 2137, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33028: Flags [.], cksum 0xbea8 (correct), ack 2, win 260, options [nop,nop,TS val 354299798 ecr
2679769181], length 0
09:38:19.851100 IP (tos 0x0, ttl 124, id 2138, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33028: Flags [R.], cksum 0xe58b (correct), seq 1, ack 2, win 0, length 0
09:38:19.861971 IP (tos 0x0, ttl 2, id 61692, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33029 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xf793), seq 1479079992, win 29200, options [mss 1460,sackOK,TS val 2679769284 ecr 0,nop,wscale 7], length 0
09:38:20.061962 IP (tos 0x0, ttl 2, id 4375, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33025 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xba74), seq 2013729973, win 29200, options [mss 1460,sackOK,TS val 2679769484 ecr 0,nop,wscale 7], length 0
09:38:20.062147 IP (tos 0x0, ttl 3, id 40320, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33030 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x484c), seq 1792250892, win 29200, options [mss 1460,sackOK,TS val 2679769484 ecr 0,nop,wscale 7], length 0
09:38:20.262363 IP (tos 0x0, ttl 4, id 35013, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33031 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xb872), seq 190808721, win 29200, options [mss 1460,sackOK,TS val 2679769684 ecr 0,nop,wscale 7], length 0
09:38:20.462652 IP (tos 0x0, ttl 5, id 20739, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33032 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x8028), seq 1800559135, win 29200, options [mss 1460,sackOK,TS val 2679769884 ecr 0,nop,wscale 7], length 0
09:38:20.545763 IP (tos 0x0, ttl 124, id 2139, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33032: Flags [S.], cksum 0x4e66 (correct), seq 3540282390, ack 1800559136, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299868 ecr 2679769884], length 0
09:38:20.545800 IP (tos 0x0, ttl 5, id 20740, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33032 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0x9ba6), ack 1, win 229, options [nop,nop,TS val 2679769967 ecr 354299868], length 0
09:38:20.545847 IP (tos 0x0, ttl 5, id 20741, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33032 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0x9ba5), seq 1, ack 1, win 229, options [nop,nop,TS val 2679769967 ecr 354299868], length 0
09:38:20.627359 IP (tos 0x0, ttl 124, id 2140, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33032: Flags [.], cksum 0x9b7e (correct), ack 2, win 260, options [nop,nop,TS val 354299876 ecr
2679769967], length 0
09:38:20.627395 IP (tos 0x0, ttl 124, id 2141, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33032: Flags [R.], cksum 0xc5c1 (correct), seq 1, ack 2, win 0, length 0
09:38:20.662792 IP (tos 0x0, ttl 2, id 64580, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33033 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x7822), seq 3574462368, win 29200, options [mss 1460,sackOK,TS val 2679770084 ecr 0,nop,wscale 7], length 0
09:38:20.862968 IP (tos 0x0, ttl 3, id 34171, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33034 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xe06a), seq 3653601750, win 29200, options [mss 1460,sackOK,TS val 2679770285 ecr 0,nop,wscale 7], length 0
09:38:21.063216 IP (tos 0x0, ttl 4, id 28386, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33035 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x6f56), seq 1884316567, win 29200, options [mss 1460,sackOK,TS val 2679770485 ecr 0,nop,wscale 7], length 0
09:38:21.263399 IP (tos 0x0, ttl 5, id 5287, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33036 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0x87e3), seq 2683771546, win 29200, options [mss 1460,sackOK,TS val 2679770685 ecr 0,nop,wscale 7], length 0
09:38:21.365071 IP (tos 0x0, ttl 124, id 2142, offset 0, flags [DF], proto TCP (6), length 60)
    172.31.x.5.microsoft-ds > 172.20.x.129.33036: Flags [S.], cksum 0x6362 (correct), seq 2176233427, ack 2683771547, win 8192, options [mss 1375,nop,wscale 8,sackOK,TS val 354299948 ecr 2679770685], length 0
09:38:21.365106 IP (tos 0x0, ttl 5, id 5288, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33036 > 172.31.x.5.microsoft-ds: Flags [.], cksum 0x62e0 (incorrect -> 0xb08f), ack 1, win 229, options [nop,nop,TS val 2679770787 ecr 354299948], length 0
09:38:21.365140 IP (tos 0x0, ttl 5, id 5289, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.x.129.33036 > 172.31.x.5.microsoft-ds: Flags [F.], cksum 0x62e0 (incorrect -> 0xb08e), seq 1, ack 1, win 229, options [nop,nop,TS val 2679770787 ecr 354299948], length 0
09:38:21.462413 IP (tos 0x0, ttl 124, id 2143, offset 0, flags [DF], proto TCP (6), length 52)
    172.31.x.5.microsoft-ds > 172.20.x.129.33036: Flags [.], cksum 0xb065 (correct), ack 2, win 260, options [nop,nop,TS val 354299958 ecr
2679770787], length 0
09:38:21.462434 IP (tos 0x0, ttl 124, id 2144, offset 0, flags [DF], proto TCP (6), length 40)
    172.31.x.5.microsoft-ds > 172.20.x.129.33036: Flags [R.], cksum 0xde2e (correct), seq 1, ack 2, win 0, length 0
09:38:21.463542 IP (tos 0x0, ttl 2, id 59880, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.x.129.33037 > 172.31.x.5.microsoft-ds: Flags [S], cksum 0x62e8 (incorrect -> 0xdc5d), seq 2120279789, win 29200, options [mss 1460,sackOK,TS val 2679770885 ecr 0,nop,wscale 7], length 0
09:38:21.663738 IP (tos 0x0, ttl 3, id 31404, offset 0, flags [DF], proto TCP (6), length 60)
<truncated>
^C
103 packets captured
103 packets received by filter
0 packets dropped by kernel
Improve this question

1 Answer 1

Reset to default
0

MTR, and all traceroute-like tools, rely on the intermediate routers receiving a packet with TTL = 0, being unable to further-decrement the TTL of that packet as required by the forwarding process, and instead, generating an ICMP TTL Exceeded error message which is sent back to the origin host.

There is a rate-limit (or policer) on this type of forwarding exception. When many packets with TTL=0 are being processed by a particular router (or interface, linecard; the specifics are device-dependent) not all of them will result in error messages being returned. This is especially true if many customers are running tools like MTR or PingPlotter simultaneously through the same hops.

The rate-limit for this kind of action is often distinct from different rate-limits for responding to ping requests.

Under the Hood

mtr and other traceroute-like tools are sending packets with the destination IP you specify on the command line, and receiving replies from intermediate hops as a side-effect of the TTL of probe packets expiring during transit. Here is what a series of mtr probes look like when viewed using tcpdump on my Macbook Pro. Note all the packets are destined to 8.8.8.8 but the ttl varies; that's how it discovers intermediate hops.

mtr command

jsw@athena:~$ sudo mtr --show-ips --first-ttl 2 8.8.8.8

                                            My traceroute  [v0.93]
athena (172.19.87.106)                                                    2020-09-04T08:17:51-0400
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                     Packets               Pings
 Host                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 2. 142.254.146.57 (142.254.146.57)                                 0.0%   423    8.5  10.2   7.5 163.7  11.8
 3. ae61.jfvlinbj02h.midwest.rr.com (74.128.7.229)                  0.0%   423   54.7 135.7  12.4 960.2 224.9
 4. be34.lsvmkyzo01r.midwest.rr.com (65.29.27.176)                  0.0%   423   12.2  15.2   9.0 172.1  12.0
 5. be24.clmkohpe01r.midwest.rr.com (65.189.140.162)                0.0%   423   22.5  22.9  16.9 173.8  11.4
 6. 107.14.17.252 (107.14.17.252)                                   0.0%   423   33.1  32.7  26.5 213.2  13.6
 7. bu-ether11.chcgildt87w-bcr00.tbone.rr.com (66.109.6.20)         0.0%   422   27.5  30.4  26.7 177.2  12.3
 8. 72.14.222.248 (72.14.222.248)                                   0.0%   422   26.8  29.1  22.3 176.8  13.1
 9. 108.170.243.225 (108.170.243.225)                               0.0%   422   28.5  30.0  27.4 221.8  13.6
10. 216.239.42.153 (216.239.42.153)                                 0.0%   422   26.9  29.1  26.3 229.9  15.1
11. dns.google (8.8.8.8)                                            0.0%   422   27.0  29.1  26.4 248.4  14.5

tcpdump output capturing mtr's probe packets

athena:~ root# tcpdump -v -n -i any host 8.8.8.8
tcpdump: data link type PKTAP
tcpdump: listening on any, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
08:14:05.304930 IP (tos 0x0, ttl 6, id 46614, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34755, length 44
08:14:05.400777 IP (tos 0x0, ttl 7, id 62610, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34756, length 44
08:14:05.492567 IP (tos 0x0, ttl 8, id 60212, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34757, length 44
08:14:05.588759 IP (tos 0x0, ttl 9, id 7176, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34758, length 44
08:14:05.680667 IP (tos 0x0, ttl 10, id 1559, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34759, length 44
08:14:05.776757 IP (tos 0x0, ttl 11, id 51614, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34760, length 44
08:14:05.803459 IP (tos 0x80, ttl 115, id 0, offset 0, flags [none], proto ICMP (1), length 64)
    8.8.8.8 > 172.19.87.106: ICMP echo reply, id 39132, seq 34760, length 44
08:14:05.872733 IP (tos 0x0, ttl 2, id 19563, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34761, length 44
08:14:05.968560 IP (tos 0x0, ttl 3, id 10474, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34762, length 44
08:14:06.064073 IP (tos 0x0, ttl 4, id 39508, offset 0, flags [none], proto ICMP (1), length 64)
    172.19.87.106 > 8.8.8.8: ICMP echo request, id 39132, seq 34763, length 44
Improve this answer
5
  • Hi @jeff-wheeler - thanks. But does the TTL apply to TCP packets? I'm using TCP in all my examples, and only MTR is showing the loss - not the other tools.
    – Jason
    Commented Sep 4, 2020 at 4:45
  • All IP packets including TCP packets have a TTL field. In your TCPing.exe and hping3 -T invocations, the packet loss results you're viewing are not from traceroute-like tests; they're from ping-like tests. The ping-like tests send packets with a destination IP of that host, and these are processed by a different rate-limiting mechanism than the traceroute-like packets. The traceroute-like packets come to the attention of a given hop because the packet's TTL is zero and therefore, the packet cannot be forwarded any further.
    – Jeff Wheeler
    Commented Sep 4, 2020 at 12:01
  • You might want to run tcpdump or wireshark while performing each test. Looking at the packet capture might help you better understand what's happening under the hood. I'll add some output to my answer, above, which might make what's happening more clear.
    – Jeff Wheeler
    Commented Sep 4, 2020 at 12:14
  • Hi Jeff - thanks for the details, however, I'm using TCP in all my tests, not ICMP. So I'm confused why TCP is still dropping (or showing as dropping). Plus you mentioned the rate limiting on "forwarding exceptions" is different than ICMP rate limiting - is that applying here based on my trace? I did the TCP dump - I'll update my question with that. I also notice a high number of "incorrect" ack checksums. Is that related?
    – Jason
    Commented Sep 7, 2020 at 1:56
  • All your probe packets are destined to 172.31.x.5. They're not "ping-like" packets, they are "traceroute-like" packets. This has nothing to do with whether they're TCP or ICMP. The distinction is that the hop showing loss, 172.18.x.239, isn't the destination -- probe packets are processed by that host because their TTL has reached zero at that point, and they can't be forwarded further.
    – Jeff Wheeler
    Commented Sep 7, 2020 at 14:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.