How to process/pipe TCPDUMPs output in realtime

Question

If I want to tcpdump DNS requests by clients (on an OpenWrt 10.04 router), then I

root@ROUTER:/etc# tcpdump -n -i br-lan dst port 53 2>&1       
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 96 bytes
22:29:38.989412 IP 192.168.1.200.55919 > 192.168.1.1.53: 5697+ A? foo.org. (25)
22:29:39.538981 IP 192.168.1.200.60071 > 192.168.1.1.53: 17481+ PTR? 150.33.87.208.in-addr.arpa. (44)
^C
2 packets captured
3 packets received by filter
0 packets dropped by kernel

That's fully ok. But. Why can't I pipe the tcpdumps output in realtime?

root@ROUTER:/etc# tcpdump -n -i br-lan dst port 53 2>&1 | awk '/\?/ {print $3}'
^C
root@ROUTER:/etc# 

If I awk, etc. anything after tcpdump, I don't get ANY output. Why is that? Why can't I process the output of tcpdump with pipelining in realtime? (so that e.g.: in the example in only outputs the 3rd column)

Are there any solutions for this?

Answer 1

Straight out of man tcpdump

-l     Make stdout line buffered.  Useful if you want to see the data while 
       capturing it.  E.g.,

              tcpdump -l | tee dat

       or

              tcpdump -l > dat & tail -f dat

       Note that on Windows,``line buffered'' means ``unbuffered'', so that 
       WinDump will write each character individually if -l is specified.

       -U is similar to -l in its behavior, but it will cause output to be 
       ``packet-buffered'', so that the output is written to stdout at the 
       end of each packet rather than at the end of each line; this is 
       buffered on all platforms, including Windows.

Answer 2

Use the option -U in combination with -w so that tcpdump writes packets immediately.

Answer 3

The tcpdump is apparently buffering output when it writes to a pipe. It's not flushing output for each write, so the system will write the output in about 4k byte chunks. Your filter is limiting out put so you won't see anything until that filter has written enough output. Once it collects enough it will be written out in a chunk and you should see several lines emitted then.

Try to trigger DNS lookups many times and see what happens then.

Answer 4

I'm building a real-time monitoring wrapper around tcpdump that needs to see packets as soon as they are available. Even with -l there is some delay.

tcpdump now has --immediate-mode, which solved this problem for me. In order to get it to work I used it in conjunction with -l.

See this answer.

Answer 5

expect has an unbuffer command to fool commands into assuming they are writing to a tty, so they don't buffer.