Automatic launch configuration does not work as expected if Hyper Threading is disabled in the Kernel
When Hyper Threading (HT) is disabled in the Kernel (not BIOS) GROMACS detects that only the physical cores are online, but decides to launch as many threads as there a logical cores. This results in a over subscription of the available hardware and sub optimal performance:
64 CPUs configured, but only 32 of them are online. This can happen on embedded platforms (e.g. ARM) where the OS shuts some cores off to save power, and will turn them back on later when the load increases. However, this will likely mean GROMACS cannot pin threads to those cores. You will likely see much better performance by forcing all cores to be online, and making sure they run at their full clock frequency. Number of logical cores detected (64) does not match the number reported by OpenMP (32). Consider setting the launch configuration manually! Running on 1 node with total 32 cores, 64 logical cores, 2 compatible GPUs Hardware detected: CPU info: Vendor: GenuineIntel Brand: Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz SIMD instructions most likely to fit this hardware: AVX2_256 SIMD instructions selected at GROMACS compile time: AVX2_256 GPU info: Number of GPUs detected: 2 #0: NVIDIA Tesla K80, compute cap.: 3.7, ECC: no, stat: compatible #1: NVIDIA Tesla K80, compute cap.: 3.7, ECC: no, stat: compatible Reading file /home-2/award/GROMACS/water-cut1.0_GMX50_bare_5.1.2/0024/topol.tpr, VERSION 5.1.2 (single precision) Changing nstlist from 10 to 20, rlist from 1.067 to 1.172 Non-default thread affinity set, disabling internal thread affinity Overriding nsteps with value passed on the command line: 2000 steps, 4 ps Using 16 MPI threads Using 4 OpenMP threads per tMPI thread >
Furthermore manually configuring the launch does not get rid of the warning.
If we're not on ARM and sysconf(_SC_NPROCESSORS_ONLN) doesn't match
sysconf(_SC_NPROCESSORS_CONF), we should use the former, as that is
what the correct count on x86 with hyperthreading disabled in the kernel.
Added some comments on assumptions and future possible problems.
#4 Updated by Berk Hess over 3 years ago
- Status changed from New to Fix uploaded
- Target version set to 5.1.3
I uploaded a fix, but left most of the warning message in place. It is still a useful hint that you can get better performance on x86 with HT turned on. I can change or remove the warning message of x86, if we think that is useful.