Skip to main content

Return to Question

Add a link to CentOS wiki page
Source Link
edit approved Jun 6, 2014 at 12:31
David Guyon
edit approved Jun 6, 2014 at 12:31
David Guyon
  • 2.8k
  • 1
  • 29
  • 40

I'm trying to pipe extremely high speed data from one application to another using 64-bit Centos6CentOS6. II have done the following benchmarks using dddd to discover that the pipes are holding me back and not the algorithm in my program. MyMy goal is to achieve somewhere around 1.5 GB/s.

First, without pipes:

dd if=/dev/zero of=/dev/null bs=8M count=1000
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 0.41925 s, 20.0 GB/s

Next, a pipe between two dd processes:

dd if=/dev/zero bs=8M count=1000 | dd of=/dev/null bs=8M
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 9.39205 s, 893 MB/s

Are there any tweaks I can make to the kernel or anything else that will improve performance of running data through a pipe? II have tried named pipes as well, and gotten similar results.

I'm trying to pipe extremely high speed data from one application to another using 64-bit Centos6. I have done the following benchmarks using dd to discover that the pipes are holding me back and not the algorithm in my program. My goal is to achieve somewhere around 1.5 GB/s.

First, without pipes:

dd if=/dev/zero of=/dev/null bs=8M count=1000
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 0.41925 s, 20.0 GB/s

Next, a pipe between two dd processes:

dd if=/dev/zero bs=8M count=1000 | dd of=/dev/null bs=8M
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 9.39205 s, 893 MB/s

Are there any tweaks I can make to the kernel or anything else that will improve performance of running data through a pipe? I have tried named pipes as well, and gotten similar results.

I'm trying to pipe extremely high speed data from one application to another using 64-bit CentOS6. I have done the following benchmarks using dd to discover that the pipes are holding me back and not the algorithm in my program. My goal is to achieve somewhere around 1.5 GB/s.

First, without pipes:

dd if=/dev/zero of=/dev/null bs=8M count=1000
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 0.41925 s, 20.0 GB/s

Next, a pipe between two dd processes:

dd if=/dev/zero bs=8M count=1000 | dd of=/dev/null bs=8M
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 9.39205 s, 893 MB/s

Are there any tweaks I can make to the kernel or anything else that will improve performance of running data through a pipe? I have tried named pipes as well, and gotten similar results.

Source Link
KyleL
KyleL
  • 1.4k
  • 2
  • 16
  • 37

Is there a way to improve performance of linux pipes?

I'm trying to pipe extremely high speed data from one application to another using 64-bit Centos6. I have done the following benchmarks using dd to discover that the pipes are holding me back and not the algorithm in my program. My goal is to achieve somewhere around 1.5 GB/s.

First, without pipes:

dd if=/dev/zero of=/dev/null bs=8M count=1000
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 0.41925 s, 20.0 GB/s

Next, a pipe between two dd processes:

dd if=/dev/zero bs=8M count=1000 | dd of=/dev/null bs=8M
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 9.39205 s, 893 MB/s

Are there any tweaks I can make to the kernel or anything else that will improve performance of running data through a pipe? I have tried named pipes as well, and gotten similar results.