GROMACS

Yes, you can get console log for individual builds if you look through the "Pipeline steps" but it's not exactly convenient. Thus
http://jenkins.gromacs.org/job/Release_workflow_master/173/execution/node/107/log/?consoleFull shows that the new AMD GPU slave also has an issue. That issue seems more like OpenCL builds not being able to jit, but we'll need to inspect the mdrun.out files.

Gerrit received a related patchset '1' for Issue #2737.
Uploader: Berk Hess (hess@kth.se)
Change-Id: gromacs~release-2019~I45068c9187873548dff585044d2c8541444e385c
Gerrit URL: https://gerrit.gromacs.org/8802

I've been running tests 5 days straight (both with and without PME offload) without any errors. I've no idea what is wrong with the release builds and not being able to read the jenkins output doesn't help either.

Paul Bauer wrote:

The release build has an additional failure in the complex/dd121 case, with the errors in checkforce.out

This output doesn't seem to tell much, except that the errors do not resemble the previous ones. I can't repro this although what I ran for 5+ days both a dev machine and bs-gpu01 was the orientation restraints tests. Now I have all nbnxn tests running, for a few hours, but so far zero errors, so I'm puzzled.

But these errors are very weird: most coordinate have a massive mismatch in only x, y or z. Some have a mismatch in 2 dimensions. So the coordinate data seems to have gotten corrupted. It might be possible that a single, varying component of one force contribution gets very large values.

Force errors can not explain this, since some atoms that are constrained are affected, whereas atoms they are constrained to are not. So either the coordinate update in the constraints is buggy, or the copy-back of constraint coordinates to the state or the coordinates get corrupted in between constraining and trajectory writing.

As noted on gerrit, AFACIT the reason for the failures is that this verification has been buggy / not updated to the right branch and has been running master code.

Assuming the underlying issue is still the already fixed rare OpenCL gather reduction (which was made less rare by me running two concurrent regressiontests in an infinite loop on GPU0 on the AMD slave) it's still intriguing the kind of errors that the above output file shows, but given that I've been unable to reproduce any of these with code other than with master (without the fix), I think we can assume the issue is a false positive until proven otherwise.

Possibly related to #2897 not related

The orientation restraints test fails on my laptop with Intel iGPU and only nobondeds offloaded with some subtle checkforce (xfv) errors, so quite likely not PME OpenCL related (possibly not even OpenCL-related?). This however did not reproduce on either NVIDIA or AMD with or without PME offload. The tip4p error I've still not managed to repro so far.

Can you please retrigger these -- as I noted above, this may not be related to OpenCL code at all (or at least not to OpenCL code new in 2019).

I retriggered this and it works now

Paul Bauer wrote:

this seems to have been resolved some time in the past. Can be kept open if we need to investigate further in the future.

I do not thin it has. I've retriggered your change (PS11 when it was still based on the OpenCL gather fix) and had 1 or 3 failures; here's the failing case on both complex/dd121 and complex/nbnxn-ljpme-geometric:
http://jenkins.gromacs.org/job/Release_workflow_master/457/

Note again that AFAICT these runs do DD so they do not use PME OpenCL.

Note again that AFAICT these runs do DD so they do not use PME OpenCL.

Confirmed:
- dd121 uses 12 ranks, 3x4x1 decomposition
- nbnxn-ljpme-geometric uses 2 ranks
hence only nonbonded offload in the configs in question that do fail intermittently on bs-gpu01.

Update: I now have one instance where manually run test failed on bs-gpu01; I've however not been able to repro it on another dev machine with ROCm yet; started a long loop of test and see what that gives.

Could it be that this is the same issue that I fixed here?
https://gerrit.gromacs.org/c/gromacs/+/13560

Berk Hess wrote:

Could it be that this is the same issue that I fixed here?
https://gerrit.gromacs.org/c/gromacs/+/13560

The issue I fixed only fixed issue with PME on GPU with a PME only rank. That is not the case here.

The output in nbnxn_vsite.tar.gz shows that the LJ and Coulomb pair energies are smaller by 2% and 20% at step 0 already. So likely pair interactions are missing.

Berk Hess wrote:

The output in nbnxn_vsite.tar.gz shows that the LJ and Coulomb pair energies are smaller by 2% and 20% at step 0 already. So likely pair interactions are missing.

Maybe some non-local interactions are missing due to a synchronization issue?

One failure on jenkins has a massive difference in LJ energy at step 0:
LJ (SR) step 0: 1944.39, step 0: 1113.65
Coulomb (SR) step 0: -15234.3, step 0: -15265.1

This difference corresponds to 99.6% of the local LJ energy of one of the two zones. This could be coincidence. But it does show that the difference must come partially, or maybe fully, from missing local interactions. My suspicion is that the local energies, and probably forces, are used before the local kernel has finished.

no progress, suggest bump before beta2