Revisions to How to output as fast as possible a fixed buffer?

Post Reopened by mao, Tanktalus, slfan, Vignesh Pichamani, Zibri

occurred Mar 31, 2019 at 10:08

clarified in BOLD the reason of the question.

Source Link

edited Mar 29, 2019 at 9:27

9.6k
3
57
47

Sample code:

#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <pthread.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  pthread_t this_thread = pthread_self ();

  struct sched_param params;

  params.sched_priority = sched_get_priority_max (SCHED_RR);

  pthread_setschedparam (this_thread, SCHED_RR, &params);


  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

It looks like 4 people didn't understand this simple question. I even put in bold the reason of the question.

Sample code:

#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <pthread.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  pthread_t this_thread = pthread_self ();

  struct sched_param params;

  params.sched_priority = sched_get_priority_max (SCHED_RR);

  pthread_setschedparam (this_thread, SCHED_RR, &params);


  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Sample code:

#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <pthread.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  pthread_t this_thread = pthread_self ();

  struct sched_param params;

  params.sched_priority = sched_get_priority_max (SCHED_RR);

  pthread_setschedparam (this_thread, SCHED_RR, &params);


  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

It looks like 4 people didn't understand this simple question. I even put in bold the reason of the question.

added 271 characters in body

Source Link

edited Mar 28, 2019 at 17:41

Zibri

9.6k
3
57
47

Sample code:

#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <pthread.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  pthread_t this_thread = pthread_self ();

  struct sched_param params;

  params.sched_priority = sched_get_priority_max (SCHED_RR);

  pthread_setschedparam (this_thread, SCHED_RR, &params);


  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Sample code:

#include <stdio.h>
#include <unistd.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Sample code:

#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <pthread.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  pthread_t this_thread = pthread_self ();

  struct sched_param params;

  params.sched_priority = sched_get_priority_max (SCHED_RR);

  pthread_setschedparam (this_thread, SCHED_RR, &params);


  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Post Closed as "Needs details or clarity" by oguz ismail, Achal, Support Ukraine, Mathieu, double-beep

occurred Mar 28, 2019 at 17:28

added 103 characters in body

Source Link

edited Mar 28, 2019 at 16:20

Zibri

9.6k
3
57
47

Sample code:

#include <stdio.h>
#include <unistd.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Sample code:

#include <stdio.h>
#include <unistd.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

Sample code:

#include <stdio.h>
#include <unistd.h>

int
main (int argc, char **argv)
{

  unsigned char buffer[128];
  char buf[0x4000];
  setvbuf (stdout, buf, _IOFBF, 0x4000);
  fork ();
  fork ();

  while (1)
    {
      fwrite (&buffer, 128, 1, stdout);
    }
}

This program opens 4 threads and outputs on stdout the contents of "buffer" which is 128 bytes or 16 long ints on a 64 bit cpu.

If I then run:

./writetest | pv -ptebaSs 800G >/dev/null

I get a speed of about 7.5 GB/s.

Incidentally, that is the same speed I get if I do:

$ mkfifo out
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
$ dd if=/dev/zero bs=16384 >out &
pv <out -ptebaSs 800G >/dev/null

Is there any way to make this faster? Note. the buffer in the real program is not filled with zeroes.

my curiosity is to understand how much data can a single program (mutithreaaded or multiprocess) output

Source Link

asked Mar 28, 2019 at 10:22

Zibri

9.6k
3
57
47

Loading

Collectives™ on Stack Overflow

Return to Question