I have a list of about 100,000 small problems to be solved in my cluster of Xubuntu machines. Each takes a couple of minutes on a single thread. More problems -- somewhat smaller -- will be produced by some of these questions. The questions arise on my strongest machine -- dual Xeon CPUs with 32 hyperthreads total, but there are another 6 smaller machines adding up to around another 32 threads. Still, it's going to take hundreds of hours.
Each problem is stated in a short text file -- about 500-1500 bytes. Answers are similar, but at the high end of that size range. There's RAID of 4TB drives for storage.
I'm prepared to use 0MQ, postgreSQL, flat files, or whatever, but never having faced a workload like this, I don't know which to prefer. And setting up a solution is enough work I would prefer to not have to re-do it.
What's the best way to set this up? Note that the results will certainly be stored on PostgreSQL, so it will definitely be there. But all hosts have 0MQ installed too. I have never seriously used either one before.
