Linux: CPU load sharing on new server

Question

We have got a new Linux el6.x86_64 server and after moving all our jobs (primarily java/C++/python/shell) processes onto this new server, we have observed the processes running slow. E.g. A python script running on previous server would finish in 30 minutes, on new server it takes 4 hours to process.

Below is the CPU usage statistics (captured with top) from new server, which shows only one task active at a time. I need your help to understand following:

• If it has 32CPUs, shouldn't there be 32 tasks running in parallel?
• All these CPUs are online, verified from /sys/devices/system/cpu. Could there be any other system setting CPU availability very low?

 
top - 02:04:04 up 5 days,  6:56, 32 users,  load average: 3.63, 3.91, 3.47
Tasks: 1208 total,   1 running, 1207 sleeping,   0 stopped,   0 zombie
Cpu0  : 23.4%us,  3.0%sy,  0.0%ni, 73.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 15.4%us,  5.6%sy,  0.0%ni, 79.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 12.2%us,  1.0%sy,  0.0%ni, 86.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  3.9%us,  0.3%sy,  0.0%ni, 95.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 19.3%us,  2.0%sy,  0.0%ni, 78.4%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu5  :  6.9%us,  0.3%sy,  0.0%ni, 92.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  1.0%us,  0.3%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu16 : 42.2%us,  4.0%sy,  0.0%ni, 53.5%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu17 :  0.7%us,  0.7%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu18 : 26.0%us,  2.0%sy,  0.0%ni, 71.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu19 :  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu20 : 53.8%us,  4.0%sy,  0.0%ni, 41.2%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu21 : 29.3%us,  1.3%sy,  0.0%ni, 69.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu22 :  2.0%us,  0.7%sy,  0.0%ni, 97.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu23 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu24 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu25 :  0.7%us,  1.0%sy,  0.0%ni, 98.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu26 :  0.7%us,  1.3%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu27 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu28 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu29 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu30 :  2.6%us,  0.0%sy,  0.0%ni, 97.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu31 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  396867664k total, 391584344k used,  5283320k free,   414764k buffers

Answer 1

The processes will not utilize the extra cores to run faster unless they are designed for it by using threads or other mechanisms for parallelism.

Python in particular is typically bad at exploiting multiple cores due to the GIL (Global interpreter Lock). You could try to use the 'multiprocessing' Python module to achieve more preferment software.

There is not much you can do to increase performance of single-core, not I/O bound processes, except buying CPUs with higher clock speeds. But sometimes you can find memory access and memory caching bottlenecks. For example, if your machine contains multiple CPUs that are connected together over NUMA connections, and processes move frequently between those CPUs, the situation may arise where CPUs have to frequently access memory over the NUMA connections through other CPUs (Memory connected to another CPU can be significantly slower to access then the memory connected directly to 'this' CPU).

It might be worthwhile to try and pin processes to particular CPUs by using the cpuset(1) command. If this works, you should also look into the various Kenel NUMA behaviour tuning parameters.

Edit to add: Another area to look at is power management, perhaps the CPU frequencies are getting scaled down because of power saving processes. You can use the cpupower(1) command to check and set VPU power management settings.