Project

General

Profile

Bug #2025

Possible deadlock with tmpi and pin=auto

Added by Roland Schulz over 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
High
Assignee:
-
Category:
mdrun
Target version:
Affected version - extra info:
Affected version:
Difficulty:
uncategorized
Close

Description

It is possible that bAllSet isn't the same for all tmpi ranks at threadaffinity.cpp:525. This in turn than causes a deadlock because setting the affinity requires global communication. This can't happen with lib-MPI because in that case MPI_Allreduce is used. I suspect that it isn't the same for all ranks because the affinity gets changed by some threads (not sure whether by OpenMP/MPI or by GROMACS) while others test.

Not sure what the best solution is because tmpi isn't initialized yet at that spot. Thus one cannot simply do a MPI_Barrier or MPI_Allreduce.

This is with ICC 17beta1 on KNL but should be possible to reproduce on other compiler/CPUs.

Associated revisions

Revision 82216120 (diff)
Added by Berk Hess over 3 years ago

Fix deadlock with thread-MPI

With thread-MPI mdrun could deadlock while pinning threads.

Fixes #2025.

Change-Id: Ib42e9625134531b1e2f910b11339aa0f78b80624

History

#1 Updated by Berk Hess over 3 years ago

How can bAllSet be different with tmpi? All tmpi threads will be running on the same hardware and I don't see any input that depends on the tmpi thread index.

#2 Updated by Teemu Murtola over 3 years ago

Different ranks call the function (and so also sched_getaffinity()) at different times, so if some library call has changed the affinity (most likely between the first rank and subsequent ranks), then this could possibly happen.

#3 Updated by Roland Schulz over 3 years ago

Exactly. If I print bAllSet I see that different ranks have different values. Because it depends on timing it doesn't always happen. But most of the time.

#4 Updated by Berk Hess over 3 years ago

Shouldn't this check be done only on a single thread with thread-MPI?
If so, we can simply move the return at 487 out of the !bAfterOpenmpInit conditional.

#5 Updated by Berk Hess over 3 years ago

I think I see what the issue is now. In the after-OpenMP-init call, one thread is probably changing the affinities while others are detecting. The solution is to simply do the reduction also with thread-MPI after openMP init. I will upload a fix.

#6 Updated by Gerrit Code Review Bot over 3 years ago

Gerrit received a related patchset '1' for Issue #2025.
Uploader: Berk Hess ()
Change-Id: Ib42e9625134531b1e2f910b11339aa0f78b80624
Gerrit URL: https://gerrit.gromacs.org/6107

#7 Updated by Berk Hess over 3 years ago

  • Category set to mdrun
  • Status changed from New to Fix uploaded
  • Priority changed from Normal to High
  • Target version set to 2016.1

#8 Updated by Berk Hess over 3 years ago

  • Status changed from Fix uploaded to Resolved

#9 Updated by Szilárd Páll about 3 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF