Project

General

Profile

Bug #3399

out of memory errors cause abort during GPU detection

Added by Szilárd Páll about 1 month ago. Updated 19 days ago.

Status:
Closed
Priority:
Normal
Category:
mdrun
Target version:
Affected version - extra info:
Affected version:
Difficulty:
uncategorized
Close

Description

Report from user of an issue where the GPU detection / sanity checks abort mdrun due to out-of-memory error, instead of just marking the device not available:
https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2020-February/128568.html

#report #request


Related issues

Related to GROMACS - Bug #3178: Fatal Error when launching mdrun on host with busy/unavailable GPU(s)Closed

Associated revisions

Revision 0a9b0ba7 (diff)
Added by Szilárd Páll about 1 month ago

Avoid mdrun terminate due to GPU sanity check errors

When a GPU is a exclusive or prohibited mode, early detection calls can
fail and as a result an mdrun run abort with an error, even if all GPU
offload is explicitly disabled by the user.
This change adds a status code to handle the case of devices being
unavailable.

Additionally, other errors may be encountered during the dummy kernel
sanity check (e.g. out of memory), but since the change that switches
to using launchGpuKernel() wrapper did not handle the exception in the
sanity checking, this can also abort a run even if the GPU in question
is not selected to be used.
This change adds code to catch the exception this and report the error
and avoid abort the run.

Fixes #3178 #3399

Change-Id: I0cdedbc02769084c172e4a42fe5c1af192007cec

History

#1 Updated by Szilárd Páll about 1 month ago

  • Related to Bug #3178: Fatal Error when launching mdrun on host with busy/unavailable GPU(s) added

#2 Updated by Szilárd Páll about 1 month ago

  • Status changed from New to Resolved

#3 Updated by Paul Bauer about 1 month ago

  • Status changed from Resolved to Closed

#4 Updated by Szilárd Páll 19 days ago

  • Description updated (diff)

Also available in: Atom PDF