Task #3093

Feature #2816: GPU offload / optimization for update&constraits, buffer ops and multi-gpu communication

Feature #2890: GPU Halo Exchange

rework GPU direct halo-exchange related force reduction complexities

Added by Szilárd Páll 2 months ago. Updated 18 days ago.

In Progress
Target version:


Force reduction is now done in two stages, if there is halo exchange CPU contribution is already reduced with communicated data early, while in other cases the transfer of force this happens later. The current mechanism also relies on position-dependent code with leading to implicit dependencies rather than explicit event-based sync with a record closely succeeding the producer and an eventWait enqueue at preceding the consumer task.

Associated revisions

Revision 54c24729 (diff)
Added by Alan Gray 2 months ago

GPU Force Halo Exchange

Activate with GMX_GPU_DD_COMMS environment variable.

Extends GPU Halo exchange feature to provide GPU Force halo exchange
functionality. Does not yet support virial steps, which require an
extra shift force reduction - these are currently performed on the
non-buffer ops / non direct-comm path. Also has same limitations as
coordinate halo exchange.

Performs part of #2890. Future work to improve synchronization towards
a more one-sided scheme (#3092) and to make depenencies more
explicit (#3093)

Change-Id: Ifc23cc8db2655f7258e68b34e7cdc7b71994e1e8

Revision 8a0d4d97 (diff)
Added by Szilárd Páll about 1 month ago

Enable StatePropagatorGpuData for force transfers

Force transfers have been switched to use StatePropagatorGpuData already
before. This change updates the synchronization mechanisms as:
- replaces the previous stream sync after GPU buffer/ops reduction with
a waitForcesReadyOnHost call;
- removes the barriers in copyForces[From|To]Gpu() as dependencies
are now satisfied: most dependencies are intra-stream and therefore
implicit, the exception being the halo exchange that uses its own
mechanism to sync H2D in the local stream with the nonlocal stream
(which is yet to be replaces Refs #3093).

Refs. #3126.

Change-Id: I8bfd39f79c87f20492c4ae287d6f19261724f806


#1 Updated by Szilárd Páll 2 months ago

  • Private changed from Yes to No

#2 Updated by Szilárd Páll about 1 month ago

  • Subject changed from rework GPU direct halo-ecxhange related force reduction complexities to rework GPU direct halo-exchange related force reduction complexities

#3 Updated by Szilárd Páll about 1 month ago

  • Priority changed from Normal to High

what is the

#4 Updated by Alan Gray 18 days ago

  • Status changed from New to In Progress
  • Target version changed from 2020 to 2020-beta3

#5 Updated by Szilárd Páll 18 days ago

Alan Gray wrote:

Addressed in pending changes

Thanks for the update. I've just flagged the former which shows 3 additional tests failing on the gpucomm matrix compared to previous triggers.

I think we need to fix correctness issues of the code before we can really move forward with new changes.

Also available in: Atom PDF