Project

General

Profile

Task #2084

MPI and stderr/log output

Added by Aleksei Iupinov about 3 years ago. Updated about 3 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Difficulty:
hard
Close

Description

During a trivial code moving the topic of the graceful error output with the MPI came up.

As requested on gerrit (https://gerrit.gromacs.org/#/c/6305/3), attached are the logs from the specific cases:
a user runs mdrun with thread-MPI or heterogenous MPI and requests non-existent GPUs.
The patchset in question mostly just moves code around, but one additional thing is removing the redundant GPU descriptions output in the thread-MPI case - tMPI already runs on one node, so all the GPUs are already described in the log/stderr.
The comparison is between stderr outputs before and after the patch.
The full log files are also included, but they obviously end abruptly, as the program just calls MPI_Abort.
I ran this across tcbs13 and tcbs14 in the MPI case, requesting "-gpu_id 00001111", where tcbs14 has 2 GPUs and tcbs13 has only one.

In the future it would be nice if the multi-node simulation could finalize the log gracefully as well.

gerrit-6305-ps3-logs.tar (100 KB) gerrit-6305-ps3-logs.tar Aleksei Iupinov, 11/28/2016 04:30 PM

Related issues

Related to GROMACS - Task #1505: improve handling of loggingNew

History

#1 Updated by Mark Abraham about 3 years ago

  • Related to Task #1505: improve handling of logging added

#2 Updated by Aleksei Iupinov about 3 years ago

Sorry, couldn't find the previous issue on the topic.

#3 Updated by Mark Abraham about 3 years ago

Thanks.

In the future it would be nice if the multi-node simulation could finalize the log gracefully as well.

In general, this is a difficult problem. One could implement such a fatal error via throwing a suitable exception, catching it higher up the call stack, converting it to an MPI message (one for the error flag and another for a potentially custom-length message string), and imposing some kind of synchronization requirement on ranks (probably doesn't have to be MPI_Alltoall, but that would be simplest), so that they can all exit gracefully, with each simulation writing appropriately to its log file. Something a bit crude could be acceptable for setup code, but we also have to consider how to implement fatal errors from e.g. "blowing up," so that an unrecoverable simulation problem on one rank of one simulation doesn't wreak havoc on another simulation, and doesn't require frequent synchronization.

Note also that errors generated by mis-use of MPI or hardware or middleware failure are basically ignored by MPI 3.0 standard (and implementations, see http://link.springer.com/book/10.1007%2F978-3-319-20943-2) and there's an active working group trying to find some consensus around which to move forward. If it existed, we might have a path forward, but it doesn't yet.

Also available in: Atom PDF