Project

General

Profile

Bug #2039

mdrun -pinstride defaults are too confusing

Added by Mark Abraham over 3 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
mdrun
Target version:
-
Affected version - extra info:
all since 4.5
Affected version:
Difficulty:
uncategorized
Close

Description

In a recent thread on gmx-users (https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2016-August/108043.html),
Szilard observed that the default mdrun -pinstride 0 changes behaviour with whether the value of mdrun -pinoffset is not zero (ie default). This isn't documented in mdrun -h, and has led to incorrect recommendations in docs/userguide/mdrun-performance.rst and http://www.gromacs.org/Documentation/Acceleration_and_parallelization.

We should fix at least the documentation, and perhaps the implementation

History

#1 Updated by Mark Abraham over 3 years ago

Currently gmx mdrun -ntmpi 1 -ntomp 6 -pinoffset 0 on a 6-core node with two hyperthreads per core places a thread on each core (equivalent to -pinstride 2 I think), which is reasonable if we assume the intention is to run one job per node. But if the intent is to then run gmx mdrun -ntmpi 1 -ntomp 6 -pinoffset 3 to place two mdrun jobs on the node, that assumption is wrong, ( now mdrun now guesses that behaviour like -pinstride 1 is more appropriate).

Since reasonable behaviour for mdrun -pinoffset -pinstride when only one of them has a non-default value depends on whether the user intends to run multiple jobs per node (whether mdrun or not), I think mdrun should refuse to run if the user sets only one of them, rather than make assumptions about their intentions.

#2 Updated by Berk Hess over 3 years ago

Yes, requiring both is a good idea.

#3 Updated by Szilárd Páll over 3 years ago

nit: in the second case one needs to use -pinoffset 6 because the offset is applied to the logical indexing i.e. in the physical "CPU" list that looks like 0,6, 1,7, 2,8,...

That alone is sadly confusing.

In principle, the suggestion of requiring both offset and stride if either is passed seems good. However, -pinoffset has no special default values that means "auto" right now (-pinstride does), we won't be able to know whether it is set by the user. Additionally, if there is no HT, we may not want to bother the user with requiring stride (but this info should be available).

#4 Updated by Mark Abraham over 3 years ago

Yes, there are also other things that mean we should port the options support to mdrun, so that we will generally have the ability to know if there was a default chosen. But we could also hack -pinoffset to default to -1 and now we probably have the ability to solve the immediate problem swiftly.

#5 Updated by Szilárd Páll about 3 years ago

Mark Abraham wrote:

But we could also hack -pinoffset to default to -1 and now we probably have the ability to solve the immediate problem swiftly.

Sounds like an option if combined with a check that detects whether pinstride makes sense. Does it not seem like an overkill to force users to pass the stride when there is only one value (=1) that makes sense?

#6 Updated by Erik Lindahl almost 2 years ago

Not sure if anything happened here the last year, but even with a quick glance this looks like the usual problem where we try to be smart, but aren't smart enough to consider all cases :-)

I'll cast a strong vote for always requiring both pinstride and pinoffset, simply because we know that shouldn't come back to bite us. Almost every time we try those checks, it later turns out there something else we didn't think of.

#7 Updated by Mark Abraham almost 2 years ago

I'll make a patch to require that the user set both or neither.

#8 Updated by Mark Abraham almost 2 years ago

Also gmx mdrun -pinoffset n for all n > 0 disables pinning, saying

Using 4 MPI threads
Applying core pinning offset 2

NOTE: Requested offset too large for available cores, thread pinning disabled.

which doesn't tell the user that the problem is that auto already filled all the hwthreads, so their choice of offset is inconsistent with their choice of thread number (ie auto. Nor is that aspect documented.

As with other such options (e.g. -gpu_id, or -nt) things seem fine when they are implemented, but break under maintenance or extension when we permit auto behaviour to be partially specified.

There's not a clearly good behaviour for -pin auto when other values are inconsistent, so I propose that we require the user to set -pin on if they want to set either of -pinoffset or -pinstride (and also that either of those requires the other), and then we then do whatever they said. Also giving a warning in this particular case is reasonable. This will mean that the user is still getting bad performance from gmx mdrun -pin on -pinoffset 2 -pinstride 2 (or whatever they are trying to do) but gives us an easy time recognizing that the right thing to warn about is that there's too many threads for the cores the user selected.

Also available in: Atom PDF