If sort created temporary files, does it mean it's out of RAM memory?

Question

I was sorting a large file (101MB - about 700MB after unzipping) using sort command on server that has 48GB of memory. It was the only heavy job it was doing at that time. However I noticed that sort created lots of temporary files. Does it mean it was lacking RAM memory?

Or is it so that sort always creates files? Can I speed up sorting process by passing a folder filesystem mounted in RAM with -T command? I tried it, but I haven't noticed significant speed up and I'm wondering whether I constructed the test wrong or I'm just not understanding what's going on correctly.

This is a command I issued:

zcat file0.nq.gz | sort

In about 20 seconds I have the following files in /tmp

nuoritoveri@nubis:/tmp[127]$ ls
sortecuGwN  sorteKeowj  sortGn7dCr  sortkdk5Ws  sortNb9Khh  sortPGTQ6b  sortQearCg  sortvBB5eS  sortZW2mWj
sort1UsQla  sortEGauDb  sortFMn7bW  sortiUDJYd  sortlaGUgo  sortpEmGb5  sortPQUNQx  sortqlb7jh  sortxcjjuM
sortaVKeEN  sortejgptJ  sortgAJJ9l  sortJRq2GB  sortmQf888  sortpFfWdy  sortpv9kO8  sortT52TVQ  sortxq8r80

The files disappear when commands finishes. I also checked what happens when I don't pipe, but just sort unzipped file:

sort file0.nq

The files in /tmp appear also, but not such a fast rate (maybe because it has to read the file by itself).

Answer 1

In general "keep using memory until you run out" is a poor strategy, you may cause problems for other users, you may end up using swap which looks like memory but has far worse performance characteristics or (because Linux overcommits memory by default) you may end up just getting killed by the OOM killer.

When sorting large volumes of data a common strategy is a "batch merge", you split the data into batches which are sorted in-memory and written to temporary files. Then you have a merge process that reads the batches and merges them together. If the data set is very large there may be multiple layers of merging.

I had a quick dig through the code for sort at https://sources.debian.org/src/coreutils/8.30-3/src/sort.c/

It looks like the buffer size sort decides to use depends on a number of factors, including ulimit values, free memory, the -S parameter if specified and the size of the input files.

It seems that in the case of unknown input size (e.g. input from a pipe), no particular memory pressure, and no particular sort size specified that sort uses a buffer size determined from "INPUT_FILE_SIZE_GUESS", which according to the comments works out to about a 17 megabyte buffer (note that said buffer doesn't just store the raw line text, so it may not fit 17 megabytes of input).