mdrun -pinstride defaults are too confusing
In a recent thread on gmx-users (https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-August/108043.html),
Szilard observed that the default
mdrun -pinstride 0 changes behaviour with whether the value of
mdrun -pinoffset is not zero (ie default). This isn't documented in
mdrun -h, and has led to incorrect recommendations in
docs/userguide/mdrun-performance.rst and http://www.gromacs.org/Documentation/Acceleration_and_parallelization.
We should fix at least the documentation, and perhaps the implementation
#1 Updated by Mark Abraham about 4 years ago
gmx mdrun -ntmpi 1 -ntomp 6 -pinoffset 0 on a 6-core node with two hyperthreads per core places a thread on each core (equivalent to
-pinstride 2 I think), which is reasonable if we assume the intention is to run one job per node. But if the intent is to then run
gmx mdrun -ntmpi 1 -ntomp 6 -pinoffset 3 to place two mdrun jobs on the node, that assumption is wrong, ( now mdrun now guesses that behaviour like
-pinstride 1 is more appropriate).
Since reasonable behaviour for
mdrun -pinoffset -pinstride when only one of them has a non-default value depends on whether the user intends to run multiple jobs per node (whether mdrun or not), I think mdrun should refuse to run if the user sets only one of them, rather than make assumptions about their intentions.
#3 Updated by Szilárd Páll about 4 years ago
nit: in the second case one needs to use
-pinoffset 6 because the offset is applied to the logical indexing i.e. in the physical "CPU" list that looks like 0,6, 1,7, 2,8,...
That alone is sadly confusing.
In principle, the suggestion of requiring both offset and stride if either is passed seems good. However,
-pinoffset has no special default values that means "auto" right now (
-pinstride does), we won't be able to know whether it is set by the user. Additionally, if there is no HT, we may not want to bother the user with requiring stride (but this info should be available).
#4 Updated by Mark Abraham about 4 years ago
Yes, there are also other things that mean we should port the options support to mdrun, so that we will generally have the ability to know if there was a default chosen. But we could also hack
-pinoffset to default to -1 and now we probably have the ability to solve the immediate problem swiftly.
#5 Updated by Szilárd Páll almost 4 years ago
Mark Abraham wrote:
But we could also hack
-pinoffsetto default to -1 and now we probably have the ability to solve the immediate problem swiftly.
Sounds like an option if combined with a check that detects whether pinstride makes sense. Does it not seem like an overkill to force users to pass the stride when there is only one value (=1) that makes sense?
#6 Updated by Erik Lindahl almost 3 years ago
Not sure if anything happened here the last year, but even with a quick glance this looks like the usual problem where we try to be smart, but aren't smart enough to consider all cases :-)
I'll cast a strong vote for always requiring both pinstride and pinoffset, simply because we know that shouldn't come back to bite us. Almost every time we try those checks, it later turns out there something else we didn't think of.
#8 Updated by Mark Abraham almost 3 years ago
gmx mdrun -pinoffset n for all n > 0 disables pinning, saying
Using 4 MPI threads Applying core pinning offset 2 NOTE: Requested offset too large for available cores, thread pinning disabled.
which doesn't tell the user that the problem is that auto already filled all the hwthreads, so their choice of offset is inconsistent with their choice of thread number (ie auto. Nor is that aspect documented.
As with other such options (e.g.
-nt) things seem fine when they are implemented, but break under maintenance or extension when we permit auto behaviour to be partially specified.
There's not a clearly good behaviour for
-pin auto when other values are inconsistent, so I propose that we require the user to set
-pin on if they want to set either of
-pinstride (and also that either of those requires the other), and then we then do whatever they said. Also giving a warning in this particular case is reasonable. This will mean that the user is still getting bad performance from
gmx mdrun -pin on -pinoffset 2 -pinstride 2 (or whatever they are trying to do) but gives us an easy time recognizing that the right thing to warn about is that there's too many threads for the cores the user selected.