How to understand the output of traceroute?

Question

I see some entries have multiple rows. Does anybody know why? Thanks.

$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
...
 8  108.170.252.129 (108.170.252.129)  36.476 ms
    108.170.240.129 (108.170.240.129)  33.678 ms
    108.170.240.193 (108.170.240.193)  33.947 ms
 9  108.170.226.183 (108.170.226.183)  38.487 ms
    72.14.232.167 (72.14.232.167)  129.904 ms
    108.170.231.71 (108.170.231.71)  140.930 ms
10  dns.google (8.8.8.8)  30.013 ms  31.672 ms  29.138 ms

Answer 1

Each numbered line represents one router “hop”, and shows the path that the packet takes from the source to the specified destination.

By default, most classic Traceroute applications will send three probes per router hop, resulting in three latency measurements to each hop. These measurements are reported on the right hand side, and are generally given in milliseconds (ms).

The difference between the three hops can be explained by each packet taking a different route to the target, or by momentary network congestion.

I would guess that the further that the target is from your computer, so the number of possible intermediate routes will increase, and with it will also increase the likelihood for larger differences in the timings.

Answer 2

Each line represents a layer 3 device. First device will be a default gateway with its ip address. If you have a total of 7 lines, 7 later 3 devices where used to reach your final destination. The ms column is timings. How long it takes for that particular leg.

Answer 3

Many network operators employ ECMP (equal-cost multipath) to combine multiple paths to the same destination. For example, the traffic could be just spread across two or four routers (each handling 1/2nd or 1/4th of all packets), or it could be spread across two entirely different ISPs.

Usually paths in ECMP are chosen by the packet's IP and TCP/UDP layer headers (so that all packets of the same connection will take the same path). If your traceroute tool was using ICMP probes, then all probes would look identical enough to always select the same path – but because Linux traceroute uses UDP with random port numbers by default, each probe can end up taking a different path. Out of three probes, you ended up getting three paths.

Important to note that not all paths are of identical length (e.g. when they go through different upstream ISPs). If traceroute probes with the same TTL end up choosing different-length paths, the trace output will be practically impossible to decipher beyond that point.

Answer 4

Quick info:

There is an alternative to traceroute, with a bit more comprehensive output, that might help in further analyzing any trouble:

$ sudo apt-get install mtr    # note also; mtr-tiny  (no "X")

As can be seen below there are a good load of options, to create reports of different kinds and output format on top of pure network settings.

Running it can be as simple as this though:

$ mtr -i 15  8.8.8.8

... which will create a periodically (15 second delay) updated table, and show that until you tell it to stop. Hit d and/or y to change the display format (repeat to loop through options). p/space to pause/unpause (e.g. to allow copy & paste) h for terse help, q to quit.

HTH

---

$ mtr --version
mtr 0.93

$ mtr --help

Usage:
 mtr [options] hostname

 -F, --filename FILE        read hostname(s) from a file
 -4                         use IPv4 only
 -6                         use IPv6 only
 -u, --udp                  use UDP instead of ICMP echo
 -T, --tcp                  use TCP instead of ICMP echo
 -I, --interface NAME       use named network interface
 -a, --address ADDRESS      bind the outgoing socket to ADDRESS
 -f, --first-ttl NUMBER     set what TTL to start
 -m, --max-ttl NUMBER       maximum number of hops
 -U, --max-unknown NUMBER   maximum unknown host
 -P, --port PORT            target port number for TCP, SCTP, or UDP
 -L, --localport LOCALPORT  source port number for UDP
 -s, --psize PACKETSIZE     set the packet size used for probing
 -B, --bitpattern NUMBER    set bit pattern to use in payload
 -i, --interval SECONDS     ICMP echo request interval
 -G, --gracetime SECONDS    number of seconds to wait for responses
 -Q, --tos NUMBER           type of service field in IP header
 -e, --mpls                 display information from ICMP extensions
 -Z, --timeout SECONDS      seconds to keep probe sockets open
 -M, --mark MARK            mark each sent packet
 -r, --report               output using report mode
 -w, --report-wide          output wide report
 -c, --report-cycles COUNT  set the number of pings sent
 -j, --json                 output json
 -x, --xml                  output xml
 -C, --csv                  output comma separated values
 -l, --raw                  output raw format
 -p, --split                split output
 -t, --curses               use curses terminal interface
     --displaymode MODE     select initial display mode
 -n, --no-dns               do not resolve host names
 -b, --show-ips             show IP numbers and host names
 -o, --order FIELDS         select output fields
 -y, --ipinfo NUMBER        select IP information in output
 -z, --aslookup             display AS number
 -h, --help                 display this help and exit
 -v, --version              output version information and exit

See the 'man 8 mtr' for details.