How much memory is consumed by the Linux kernel per TCP/IP network connection?

Question

How much memory on average is consumed by the Linux kernel (in kernel address space) per TCP/IP network connection?

Answer 1

For a TCP connection memory consumed depends on

1. size of sk_buff (internal networking structure used by linux kernel)

2. the read and write buffer for a connection

the size of buffers can be tweaked as required

root@x:~# sysctl -A | grep net | grep mem

check for these variables

these specify the maximum default memory buffer usage for all network connections in kernel

net.core.wmem_max = 131071

net.core.rmem_max = 131071

net.core.wmem_default = 126976

net.core.rmem_default = 126976

these specify buffer memory usage specific to tcp connections

net.ipv4.tcp_mem = 378528   504704  757056

net.ipv4.tcp_wmem = 4096    16384   4194304

net.ipv4.tcp_rmem = 4096    87380   4194304

the three values specified are " min default max" buffer sizes. So to start with linux will use the default values of read and write buffer for each connection. As the number of connection increases , these buffers will be reduced [at most till the specified min value] Same is the case for max buffer value.

These values can be set using this sysctl -w KEY=KEY VALUE

eg. The below command ensures the read and write buffers for each connection are 4096 each.

sysctl -w net.ipv4.tcp_rmem='4096 4096 4096'

sysctl -w net.ipv4.tcp_wmem='4096 4096 4096'

Answer 2

Also depends on which layer. In case of a pure bridging scenario, there's just the bridge-level FDB. When routing comes into play, there's the routing table and the IP-level FDB/neighbor db. And finally, once a local socket is in the play, you have of course window size, socket buffers (both send and receive, and they default to 128k last time I checked), fragment lists (if used), so that is where your memory goes, but a clear-cut answer is hard to make with all the parts in use. You can use ss -m to obtain a few memory statistics of local stream sockets.

Answer 3

It depends. On many many things.
I think an idle connection will take a few hundreds of bytes.
But if there's data in the transmit and/or receive data, then the consumption increases. The window size can roughly limit this consumption.
The extra consumption for data isn't just the bytes in the receive/transmit queue. There are overheads, so a segment with one byte might take something like 2K. TCP tries to reduce this, for example by merging segments into a single sk_buff, but it doesn't always succeed.

Answer 4

If you mean the active size of the buffers instead of max buffer limits that define the potentially used memory, you can try to query current usage like follows:

while true; do sleep 1; ss -at | awk '$2 > 0 || $3 > 0 { totalreceive += $2; totalsend += $3; } END { print "Totals: Recv-Q", totalreceive, " Send-Q:", totalsend}'; done

That one-liner will query current status of the system every second (logically take snapshot of the system status at that moment) and compute total bytes used for receive and send buffers. The total amount of bytes needed is typically surprisingly low for the kernel because the data is typically transferred to user process pretty fast and user mode process typically has its own internal buffers maintained by the programmer.

I don't know if there's easy way to know how much RAM is actually mapped for these buffers because the kernel probably avoids acquiring and releasing the memory all the time. The above one-liner will output the logical amount of bytes used for the buffering including the buffers waiting for new connection to listened ports.

The output might look something like this:

Totals: Recv-Q 0  Send-Q: 10497
Totals: Recv-Q 11584  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 10513
Totals: Recv-Q 2092360  Send-Q: 10175
Totals: Recv-Q 0  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 1446152
Totals: Recv-Q 0  Send-Q: 1758452
Totals: Recv-Q 1973624  Send-Q: 10513
Totals: Recv-Q 0  Send-Q: 9572
Totals: Recv-Q 11584  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 614740
Totals: Recv-Q 0  Send-Q: 1446152
Totals: Recv-Q 0  Send-Q: 1600220
Totals: Recv-Q 0  Send-Q: 1586340
Totals: Recv-Q 0  Send-Q: 775602
Totals: Recv-Q 1448  Send-Q: 9572
Totals: Recv-Q 57920  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 9572
Totals: Recv-Q 23168  Send-Q: 9572
Totals: Recv-Q 0  Send-Q: 9572

even if you run the whole test at speed.cloudflare.com with results in hundreds of megabits. That means that the total amount of TCP/IP traffic bytes in kernel buffers was less than 2 MB at all times even though I trasmitted and received 100+ MB of data as fast as possible.