GROMACS

Your .tpr works fine for me with intel 2018 compilers (+ mkl,openmpi,hwloc) on our dev-knl01 node.

The timing suggests that the module(?) for gromacs 2018.1 is failing to permit mdrun resolve hwloc library at run time. You might try ldd mdrun_knl to see if that is the case, or explictly loading an hwloc module (though I do not see such a module on marconi)

However an otherwise identical build with icc 17.0.4.20170411 behaves very strangely, apparently starting 32 independent processes each of 272 threads. That's also consistent with Carlo's observations. I suggest Carlo use/request intel 2018 (I used 18.0.1.20171018)

For me it's fine with 17.0.4. You might want to compile without HWLOC (-DGMX_HWLOC=no) to check whether that works. Also you should get an error before the MPI error. Please check whether there is any output before the MPI error and copy it here. BTW: You don't want to run with default number of OpenMP on KNL. You want to use "-ntomp 4".

There is not much text before the mpi error

GROMACS: mdrun_knl, version 2018.1
Executable: /marconi/home/userexternal/ccamillo/opt/gromacs-2018.1/bin/mdrun_knl
Data prefix: /marconi/home/userexternal/ccamillo/opt/gromacs-2018.1
Working dir: /marconi_scratch/userexternal/ccamillo/test
Command line:
mdrun_knl -s topol0 -nb cpu -v -maxh 23.9 -nsteps -1

Back Off! I just backed up md.log to ./#md.log.9#

=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 25280 RUNNING AT r065c04s03 = EXIT CODE: 132 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================

=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 25280 RUNNING AT r065c04s03 = EXIT CODE: 4 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================
Intel(R) MPI Library troubleshooting guide:
https://software.intel.com/node/561764 ===================================================================================

I will do some more test and report back

Test 1:
(I have added hwloc in the module I am loading explicitly in the script and added explicitly -ntomp 2, but btw I am anyway setting the number of openmp thread in the slurm script, and indeed this is not an issue with GMX2016.5)

module load intel intelmpi mkl hwloc
mpiexec -np 32 mdrun_knl -s topol0 -nb cpu -v -maxh 23.9 -nsteps -1 -ntomp 2 >& log

Same error messages

Test 2:
Recompiled with HWLOC OFF

GROMACS version: 2018.1
Precision: single
Memory model: 64 bit
MPI library: MPI
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support: disabled
SIMD instructions: AVX_512_KNL
FFT library: Intel MKL
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
Built on: 2018-05-17 11:33:17
Built by: ccamillo@r000u06l01 [CMAKE]
Build OS/arch: Linux 3.10.0-327.36.3.el7.x86_64 x86_64
Build CPU vendor: Intel
Build CPU brand: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
Build CPU family: 6 Model: 79 Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /cineca/prod/opt/compilers/intel/pe-xe-2017/binary/bin/icc Intel 17.0.4.20170411
C compiler flags: -xMIC-AVX512 -mkl=sequential -std=gnu99 -O3 -DNDEBUG -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits
C++ compiler: /cineca/prod/opt/compilers/intel/pe-xe-2017/binary/bin/icpc Intel 17.0.4.20170411
C++ compiler flags: -xMIC-AVX512 -mkl=sequential -std=c++11 -O3 -DNDEBUG -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits

Same error message as before

I will test the Intel2018 available on Marconi

Unfortunately it doesn't work also with the intel 2018 available

GROMACS version: 2018.1
Precision: single
Memory model: 64 bit
MPI library: MPI
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64)
GPU support: disabled
SIMD instructions: AVX_512_KNL
FFT library: Intel MKL
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
Built on: 2018-05-18 13:21:43
Built by: ccamillo@r000u06l01 [CMAKE]
Build OS/arch: Linux 3.10.0-327.36.3.el7.x86_64 x86_64
Build CPU vendor: Intel
Build CPU brand: Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
Build CPU family: 6 Model: 79 Stepping: 1
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /cineca/prod/opt/compilers/intel/pe-xe-2018/binary/bin/icc Intel 18.0.2.20180210
C compiler flags: -xMIC-AVX512 -mkl=sequential -std=gnu99 -O3 -DNDEBUG -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits
C++ compiler: /cineca/prod/opt/compilers/intel/pe-xe-2018/binary/bin/icpc Intel 18.0.2.20180210
C++ compiler flags: -xMIC-AVX512 -mkl=sequential -std=c++11 -O3 -DNDEBUG -ip -funroll-all-loops -alias-const -ansi-alias -no-prec-div -fimf-domain-exclusion=14 -qoverride-limits

log:

Back Off! I just backed up md.log to ./#md.log.13#

=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 2883 RUNNING AT r065c01s03 = EXIT CODE: 132 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================

=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 2883 RUNNING AT r065c01s03 = EXIT CODE: 4 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES ===================================================================================
Intel(R) MPI Library troubleshooting guide:
https://software.intel.com/node/561764 ===================================================================================

• Try to run mdrun from an interactive slurm shell rather than starting it through a script. My hope is it shows more info in that case.
• Build with "-g" (CXXFLAGS and CFLAGS) and run inside debugger.

Have you asked the Marconi admins?

I know, anyway for completeness

Compiled in debug mode it works

Compiled in RelWithDebInfo it doesn’t and unfortunately the interactive mode doesn’t give any additional information about the error

It is likely a compiler issue but a unfortunate one..

Or trying using an Intel compiler with another MPI library.

I don't know if it is of any help, but recompiling with openmpi and RelWithDebInfo and running it inside GDB I get

Tracepoint 1 at 0x63fc7b: file /marconi/home/userexternal/ccamillo/Codes/gromacs-2018.1/src/gromacs/hardware/identifyavx512fmaunits.cpp, line 244.

Could you try the Release build and change the SIMD_AVX_512_CXX_SUPPORTED in the CMakeCache.txt file to 0?

Could you also paste all (/much more) of your output from GDB? Did you set a tracepoint? In what context did you get that message?

Changing SIMD_AVX_512_CXX_SUPPORTED in the CMakeCache.txt file to 0 makes it work!

About the gdb output it was not much more than what I reported:

Reading symbols from /marconi/home/userexternal/ccamillo/opt/gromacs-2018.1/bin/mdrun_knl...done.
(gdb) run
Starting program: /marconi/home/userexternal/ccamillo/opt/gromacs-2018.1/bin/mdrun_knl
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /cineca/prod/opt/compilers/intel/pe-xe-2017/binary/inspector/lib64/libstdc++.so.6
Missing separate debuginfo for /cineca/prod/opt/compilers/intel/pe-xe-2017/binary/inspector/lib64/libgcc_s.so.1
[New Thread 0x2aaab4359700 (LWP 38979)]
[New Thread 0x2aaab497d700 (LWP 39012)]
:-) GROMACS - mdrun_knl, 2018.1 (-:

GROMACS is written by:
[...]
Back Off! I just backed up md.log to ./#md.log.9#

Program received signal SIGILL, Illegal instruction.
checkDualAvx512FmaUnits () at /marconi/home/userexternal/ccamillo/Codes/gromacs-2018.1/src/gromacs/hardware/identifyavx512fmaunits.cpp:244
244 return (timeFmaAndShuf > 1.5 * timeFmaOnly);
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.5.x86_64 infinipath-psm-3.3-0.g6f42cdb1bb8.2.el7.x86_64 libhfi1-0.5-27.el7.x86_64 libibumad-1.3.10.2-1.el7.x86_64 libibverbs-1.1.8-8.el7.x86_64 libipathverbs-1.3-2.el7.x86_64 libnl3-3.2.21-10.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64 libpsm2-10.2.235-1.x86_64 librdmacm-1.0.21-1.el7.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 opensm-libs-3.3.19-1.el7.x86_64 sssd-client-1.13.0-40.el7_2.12.x86_64

(gdb) trace
Tracepoint 1 at 0x63fc7b: file /marconi/home/userexternal/ccamillo/Codes/gromacs-2018.1/src/gromacs/hardware/identifyavx512fmaunits.cpp, line 244.

Gerrit received a related patchset '1' for Issue #2504.
Uploader: Roland Schulz (roland.schulz@intel.com)
Change-Id: gromacs~release-2018~Ie2f55718f98d3dfbf3c312afa5141c77ead77a6d
Gerrit URL: https://gerrit.gromacs.org/7926

PS: Please undo the change to the CMakeCache.txt for the test.

yes your fix (patch set 2) works

Thanks. Until we release 2018.2 you can use the CMakeCache work-around. That shouldn't have any side-effects.

Great, thanks!