is there any consequence if I create a lot of pipes in the same program

Question

I need to create a lot of pipes, I want to use them as fifo queues between threads in the same process, and then do select/poll on them.

my question is, if I create a lot of queues, wiil this have big impact on the performance of the program? like pipe number limitations, resource-consuming, etc

thanks!

Answer 1

There is a per-process limit on the number of files you can keep open, and pipes are files, so you consume that limit.

Answer 2

select can only handle file descriptors with values up to FD_SETSIZE (1024 is the typical value), so you should definitely use poll and forget about select if you'll have lots. Note however that the default ulimit for open files in a process is typically the same as FD_SETSIZE, so you'll also want to increase that if you need more. In addition, pipes are mildly expensive in terms of kernelspace memory usage (at least a page or two for each open pipe, probably more) and performance (sending/receiving data requires multiple copies and transitions between user and kernel space).

As long as you only have one pipe per thread, you'll probably be spending a lot more resources on having the thread around than the pipe, so your design doesn't look so bad. However, if you're using threads, it could be a lot more efficient to use your own userspace-only queues based on condition variables. This is much more idiomatic in the POSIX threads scheme of things, but less idiomatic in the way Linux-trained application developers tend to work. It would also make it so there's essentially no limit on the number of queues you can have (except available memory).

If possible, rather than focusing on the cost and limits, I would focus on the interface requirements. Is there a strong reason you want to be using poll or select, e.g. having your threads wait simultaneously on your queues and other file-descriptor-based events? Or are you just leaning towards pipes because it's what you know? In the former case, I think pipes probably make sense, but in the latter case I would spend a little time researching the other options and see which really makes more sense to your problem.