Project

General

Profile

Bug #2507

MSAN error in TNG

Added by Roland Schulz 4 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Affected version - extra info:
Affected version:
Difficulty:
uncategorized
Close

Description

./bin/mdrun-test --gtest_filter=MdrunCanWrite/Trajectories.ThatDifferInNstxout/2
gives:

==27792==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f4edb416d7b in deflate_slow /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:1757:13
    #1 0x7f4edb40c05b in deflate /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:905:25
    #2 0x7f4edb401e6a in compress2 /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/compress.c:49:11
    #3 0x7f4edb3f7762 in tng_gzip_compress /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:4646:12
    #4 0x7f4edb3b287f in tng_data_block_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:6146:20
    #5 0x7f4edb39be07 in tng_frame_set_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12161:9
    #6 0x7f4edb3c1e64 in tng_frame_set_premature_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12226:12
    #7 0x7f4edc8f517c in mdoutf_write_to_trajectory_files(_IO_FILE*, t_commrec const*, gmx_mdoutf*, int, gmx_mtop_t*, long, double, t_state*, t_state*, ObservablesHistory*, gmx::ArrayRef<gmx::BasicVector<float> >) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/mdoutf.cpp:296:13
    #8 0x7f4edc9e74c9 in do_md_trajectory_writing /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/trajectory_writing.cpp:148:9
    #9 0x7f4edca99572 in gmx::Integrator::do_md() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/md.cpp:1377:9
    #10 0x7f4edcad39ed in gmx::Mdrunner::mdrunner() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/runner.cpp:1315:20
    #11 0x4cf60d in gmx::Mdrunner::mainFunction(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:511:10
    #12 0x4cc111 in gmx_mdrun(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:95:19
    #13 0x4bf88f in gmx::test::SimulationRunner::callMdrun(gmx::test::CommandLine const&) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:248:12
    #14 0x4bf9e8 in gmx::test::SimulationRunner::callMdrun() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:254:12
    #15 0x4f10f8 in (anonymous namespace)::TrajectoryWritingTest::runTest() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:78:13
    #16 0x4f0782 in (anonymous namespace)::Trajectories_ThatDifferInNstxout_Test::TestBody() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:107:5
    #17 0x5ebeba in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2438:14
    #18 0x5ac88d in testing::Test::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2474:5
    #19 0x5ae61f in testing::TestInfo::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2656:11
    #20 0x5af85a in testing::TestCase::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2774:28
    #21 0x5c3c24 in testing::internal::UnitTestImpl::RunAllTests() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:4649:43
    #22 0x5f02aa in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2438:14
    #23 0x5c321c in testing::UnitTest::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:4257:10
    #24 0x516377 in main /home/schulzro/gromacs/clang6.hsw.msan/../src/testutils/unittest_main.cpp:85:19
    #25 0x7f4ed9f90c04 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
    #26 0x431351 in _start (/home/schulzro/gromacs/clang6.hsw.msan/bin/mdrun-test+0x431351)

  Uninitialized value was stored to memory at
    #0 0x43708b in __msan_memcpy /local/mnt/workspace/bcain_clang_aus-pallas-04_35715/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/msan/msan_interceptors.cc:1520:3
    #1 0x7f4edb41803b in read_buf /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:1088:5
    #2 0x7f4edb406ebf in fill_window /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:1467:13
    #3 0x7f4edb414f0e in deflate_slow /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:1745:13
    #4 0x7f4edb40c05b in deflate /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/deflate.c:905:25
    #5 0x7f4edb401e6a in compress2 /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/external/zlib/compress.c:49:11
    #6 0x7f4edb3f7762 in tng_gzip_compress /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:4646:12
    #7 0x7f4edb3b287f in tng_data_block_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:6146:20
    #8 0x7f4edb39be07 in tng_frame_set_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12161:9
    #9 0x7f4edb3c1e64 in tng_frame_set_premature_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12226:12
    #10 0x7f4edc8f517c in mdoutf_write_to_trajectory_files(_IO_FILE*, t_commrec const*, gmx_mdoutf*, int, gmx_mtop_t*, long, double, t_state*, t_state*, ObservablesHistory*, gmx::ArrayRef<gmx::BasicVector<float> >) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/mdoutf.cpp:296:13
    #11 0x7f4edc9e74c9 in do_md_trajectory_writing /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/trajectory_writing.cpp:148:9
    #12 0x7f4edca99572 in gmx::Integrator::do_md() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/md.cpp:1377:9
    #13 0x7f4edcad39ed in gmx::Mdrunner::mdrunner() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/runner.cpp:1315:20
    #14 0x4cf60d in gmx::Mdrunner::mainFunction(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:511:10
    #15 0x4cc111 in gmx_mdrun(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:95:19
    #16 0x4bf88f in gmx::test::SimulationRunner::callMdrun(gmx::test::CommandLine const&) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:248:12
    #17 0x4bf9e8 in gmx::test::SimulationRunner::callMdrun() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:254:12
    #18 0x4f10f8 in (anonymous namespace)::TrajectoryWritingTest::runTest() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:78:13
    #19 0x4f0782 in (anonymous namespace)::Trajectories_ThatDifferInNstxout_Test::TestBody() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:107:5

  Uninitialized value was stored to memory at
    #0 0x43708b in __msan_memcpy /local/mnt/workspace/bcain_clang_aus-pallas-04_35715/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/msan/msan_interceptors.cc:1520:3
    #1 0x7f4edb3b229e in tng_data_block_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:6006:13
    #2 0x7f4edb39be07 in tng_frame_set_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12161:9
    #3 0x7f4edb3c1e64 in tng_frame_set_premature_write /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:12226:12
    #4 0x7f4edc8f517c in mdoutf_write_to_trajectory_files(_IO_FILE*, t_commrec const*, gmx_mdoutf*, int, gmx_mtop_t*, long, double, t_state*, t_state*, ObservablesHistory*, gmx::ArrayRef<gmx::BasicVector<float> >) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/mdoutf.cpp:296:13
    #5 0x7f4edc9e74c9 in do_md_trajectory_writing /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/trajectory_writing.cpp:148:9
    #6 0x7f4edca99572 in gmx::Integrator::do_md() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/md.cpp:1377:9
    #7 0x7f4edcad39ed in gmx::Mdrunner::mdrunner() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/runner.cpp:1315:20
    #8 0x4cf60d in gmx::Mdrunner::mainFunction(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:511:10
    #9 0x4cc111 in gmx_mdrun(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:95:19
    #10 0x4bf88f in gmx::test::SimulationRunner::callMdrun(gmx::test::CommandLine const&) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:248:12
    #11 0x4bf9e8 in gmx::test::SimulationRunner::callMdrun() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:254:12
    #12 0x4f10f8 in (anonymous namespace)::TrajectoryWritingTest::runTest() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:78:13
    #13 0x4f0782 in (anonymous namespace)::Trajectories_ThatDifferInNstxout_Test::TestBody() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:107:5
    #14 0x5ebeba in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2438:14
    #15 0x5ac88d in testing::Test::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2474:5
    #16 0x5ae61f in testing::TestInfo::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2656:11
    #17 0x5af85a in testing::TestCase::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2774:28
    #18 0x5c3c24 in testing::internal::UnitTestImpl::RunAllTests() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:4649:43
    #19 0x5f02aa in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2438:14

  Uninitialized value was created by a heap allocation
    #0 0x43d642 in realloc /local/mnt/workspace/bcain_clang_aus-pallas-04_35715/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/msan/msan_interceptors.cc:921:3
    #1 0x7f4edb3e4e01 in tng_allocate_data_mem /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:5191:27
    #2 0x7f4edb3e2de6 in tng_util_generic_write_interval_set /home/schulzro/gromacs/clang6.hsw.msan/../src/external/tng_io/src/lib/tng_io.c:16194:20
    #3 0x7f4edb998b3d in set_writing_intervals(gmx_tng_trajectory*, int, t_inputrec const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/fileio/tngio.cpp:538:9
    #4 0x7f4edb998425 in gmx_tng_prepare_md_writing(gmx_tng_trajectory*, gmx_mtop_t const*, t_inputrec const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/fileio/tngio.cpp:561:5
    #5 0x7f4edc8f3950 in init_mdoutf(_IO_FILE*, int, t_filenm const*, MdrunOptions const&, t_commrec const*, gmx::IMDOutputProvider*, t_inputrec const*, gmx_mtop_t*, gmx_output_env_t const*, gmx_wallcycle*) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/mdoutf.cpp:174:25
    #6 0x7f4edc9d4178 in init_md(_IO_FILE*, t_commrec const*, gmx::IMDOutputProvider*, t_inputrec*, gmx_output_env_t const*, MdrunOptions const&, double*, double*, t_state*, double*, t_nrnb*, gmx_mtop_t*, gmx_update_t**, int, t_filenm const*, gmx_mdoutf**, t_mdebin**, float (*) [3], float (*) [3], float*, int*, t_vcm**, gmx_wallcycle*) /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdlib/sim_util.cpp:2968:17
    #7 0x7f4edca8cf98 in gmx::Integrator::do_md() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/md.cpp:418:5
    #8 0x7f4edcad39ed in gmx::Mdrunner::mdrunner() /home/schulzro/gromacs/clang6.hsw.msan/../src/gromacs/mdrun/runner.cpp:1315:20
    #9 0x4cf60d in gmx::Mdrunner::mainFunction(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:511:10
    #10 0x4cc111 in gmx_mdrun(int, char**) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/mdrun.cpp:95:19
    #11 0x4bf88f in gmx::test::SimulationRunner::callMdrun(gmx::test::CommandLine const&) /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:248:12
    #12 0x4bf9e8 in gmx::test::SimulationRunner::callMdrun() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/moduletest.cpp:254:12
    #13 0x4f10f8 in (anonymous namespace)::TrajectoryWritingTest::runTest() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:78:13
    #14 0x4f0782 in (anonymous namespace)::Trajectories_ThatDifferInNstxout_Test::TestBody() /home/schulzro/gromacs/clang6.hsw.msan/../src/programs/mdrun/tests/trajectory_writing.cpp:107:5
    #15 0x5ebeba in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2438:14
    #16 0x5ac88d in testing::Test::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2474:5
    #17 0x5ae61f in testing::TestInfo::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2656:11
    #18 0x5af85a in testing::TestCase::Run() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:2774:28
    #19 0x5c3c24 in testing::internal::UnitTestImpl::RunAllTests() /home/schulzro/gromacs/clang6.hsw.msan/../src/external/googletest/googletest/src/gtest.cc:4649:43

All other unit tests are fine.
Version: https://gerrit.gromacs.org/7920/1
clang: 6.0
cmake cmd: FLAGS="-g" ; CFLAGS=$FLAGS CXXFLAGS=$FLAGS CC=clang CXX=clang++ cmake -GNinja -DGMX_BUILD_HELP=OFF -DCMAKE_BUILD_TYPE=MSAN -DGMX_MSAN_PATH=$LIBCXX_BUILD -DGMX_HWLOC=no -DGMX_OPENMP=no -DGMX_EXTERNAL_BLAS=no -DGMX_EXTERNAL_LAPACK=no ..

Associated revisions

Revision 0ffc06b3 (diff)
Added by Magnus Lundborg 4 months ago

Write lambdas and box to TNG at correct intervals

Lambdas and box sizes are supposed to be written at the lowest common
denominator of the interval of all outputs to the TNG file(s). Before
this commit it was only written when other output was also written.

Fixes #2507

Change-Id: If9c7b063deccbc3a94b5c5df4b59e06ffdbeac56

History

#1 Updated by Aleksei Iupinov 4 months ago

Oh, so there is something here already.

I've seen the trajectory tests fail twice recently, both with https://gerrit.gromacs.org/#/c/7865/, which changes the way PME CUDA timings are compiled. PME CUDA is nowhere near this test, so it must be same bug.

First failure from Jenkins gcc-6 cuda-9.1 presubmit


Error Message

NUMERICAL

...

[----------] 3 tests from MdrunCanWrite/Trajectories
[ RUN      ] MdrunCanWrite/Trajectories.ThatDifferInNstxout/0

NOTE 1 [file /mnt/workspace/Matrix_PreSubmit_master/30436e55/gromacs/src/programs/mdrun/tests/Testing/Temporary/MdrunCanWrite_Trajectories_ThatDifferInNstxout_0_input.mdp, line 12]:
  /mnt/workspace/Matrix_PreSubmit_master/30436e55/gromacs/src/programs/mdrun/tests/Testing/Temporary/MdrunCanWrite_Trajectories_ThatDifferInNstxout_0_input.mdp did not specify a value for the .mdp option "cutoff-scheme". Probably it
  was first intended for use with GROMACS before 4.6. In 4.6, the Verlet
  scheme was introduced, but the group scheme was still the default. The
  default is now the Verlet scheme, so you will observe different behaviour.

Setting the LD random seed to -473675791
Generated 279 of the 1225 non-bonded parameter combinations
Excluding 2 bonded neighbours molecule type 'Methanol'
Excluding 2 bonded neighbours molecule type 'SOL'
Removing all charge groups because cutoff-scheme=Verlet
Number of degrees of freedom in T-Coupling group System is 12.00

NOTE 2 [file /mnt/workspace/Matrix_PreSubmit_master/30436e55/gromacs/src/programs/mdrun/tests/Testing/Temporary/MdrunCanWrite_Trajectories_ThatDifferInNstxout_0_input.mdp]:
  You are using a plain Coulomb cut-off, which might produce artifacts.
  You might want to consider using PME electrostatics.

There were 2 notes
The current CPU can measure timings more accurately than the code in
mdrun-test was configured to use. This might affect your simulation
speed as accurate timings are needed for load-balancing.
Please consider rebuilding mdrun-test with the GMX_USE_RDTSCP=ON CMake option.
Reading file /mnt/workspace/Matrix_PreSubmit_master/30436e55/gromacs/src/programs/mdrun/tests/Testing/Temporary/MdrunCanWrite_Trajectories_ThatDifferInNstxout_0.tpr, VERSION 2019-dev-20180523-132c3dc-local (single precision)
Changing nstlist from 10 to 100, rlist from 1.022 to 1.373

Using 1 MPI thread
Using 1 OpenMP thread 

1 GPU auto-selected for this run.
Mapping of GPU IDs to the 1 GPU task in the 1 rank on this node:
  PP:0

NOTE: Thread affinity was not set.
starting mdrun 'spc-and-methanol'
6 steps,      0.0 ps.

The second one - NVIDIA OpenCL debug build on my machine, MdrunCanWrite/NptTrajectories.WithDifferentPcoupl/2, same symptoms.

#2 Updated by Gerrit Code Review Bot 4 months ago

Gerrit received a related patchset '1' for Issue #2507.
Uploader: Magnus Lundborg ()
Change-Id: gromacs~master~If9c7b063deccbc3a94b5c5df4b59e06ffdbeac56
Gerrit URL: https://gerrit.gromacs.org/7961

#3 Updated by Magnus Lundborg 4 months ago

  • Assignee set to Magnus Lundborg

The problem was that lambdas and box sizes were not written as often as expected. Hopefully everything works with the patch I just submitted.

#4 Updated by Mark Abraham 4 months ago

  • Status changed from New to Fix uploaded
  • Target version set to 2019

#5 Updated by Magnus Lundborg 4 months ago

  • Status changed from Fix uploaded to Resolved

#6 Updated by Mark Abraham 4 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF