Project

General

Profile

Feature #2282

Density map toolset

Added by Christian Blau almost 2 years ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Category:
-
Target version:
Difficulty:
uncategorized
Close

Description

Assemble functionality to deal with density maps as e.g. from cryo electron microscopy.

  • i/o routines for density maps
  • a module to allow fitting to cryo-EM densities during the simulation
  • analysis tool to compute average densities from a trajectory.

Related issues

Related to GROMACS - Task #2281: Represent data on regularly spaced N-dimensional grids.In Progress
Related to GROMACS - Feature #1877: Support reading 3d-grid data from cryo-electron microscopy tomography and X-ray crystallographyClosed
Related to GROMACS - Task #2461: Expose more info from t_atom to trajectory analysis frameworkClosed

Associated revisions

Revision 6475ce8b (diff)
Added by Christian Blau 9 months ago

Multidimensional data in canonical layout.

Introduces a multidimensional array that holds its own data and makes it
accessible via the canonical right-aligned layout for indexing the data.

refs: #2282

Change-Id: I883cdb6a41d7277669e100320e230904856d1318

Revision c94c85dc (diff)
Added by Christian Blau 7 months ago

densityfitting - density file format (mrc) header.

Implements mrc/cc4/map format file headers as decribed in
"EMDB Map Distribution Format Description Version 1.01 (c)
emdatabank.org 2014"

Reading and writing is part of a follow-up patch.

refs #2282 and #1877

Change-Id: Ifd43cedb137c0beeb320e1299df8bae9916c8ea8

Revision 75a273b0 (diff)
Added by Christian Blau 6 months ago

densityfitting - MDModule declaration

Declaring the infrastructure for running molecular dynamics simulations with
addional forces that are derived from densities.

Adds a IForceProvider for density fitting simulation that is set up with
its DensityFittingParameters that are in turn built from DensityFittingOptions.

refs #2282 #2901

Change-Id: I0732a78747582a6e23bba1e141d73c4cda421011

Revision a1b56efd (diff)
Added by Christian Blau 6 months ago

One dimensional Gaussian function on lattice.

Evaluate a Gaussian of width sigma in one dimension on lattice points with unit spacing.

refs #2282

Change-Id: I57a513f05af44323e7109c4560438dc023793c4c

Revision b37274d9 (diff)
Added by Christian Blau 6 months ago

Three dimensional gauss transform

Added class that adds gaussians functions with specified
amplitudes and coordinates onto a three-dimensional scalar field.

Change-Id: I07ea2cdd219b34afc402975a0b0e8a52eb71acaf
refs: #2282

Revision f97e5a8c (diff)
Added by Christian Blau 6 months ago

density fitting forces

Evaluate forces on atoms that increase a measure of the goodness of fit to a reference density.

The density fitting forces are evaluated as the gradient of the goodness of fit of a comparison density is generated from atoms by spreading on a grid via a Gaussian kernel to the reference density.

Applying the chain rule for this gradient yields the density fitting force as the inner product of the derivative of the comparison density to reference density goodness-of-fit function at all density values and the gradient of the spreading kernel used to simulate the density.

This patch assumes that the comparison density to reference density goodness of fit derivative density is given, methods that aid the evaluation of this derivative density are part of a child patch.

refs #2282

Change-Id: I4d93fffbdb41753f9a0a4af921011aca02ade69c

Revision 990f4992 (diff)
Added by Christian Blau 6 months ago

mrc density map reading/writing

Add a class that combines header and mrc data handling for float values
to enable reading and writing of mrc files.

refs #2282

Change-Id: Idd84ab2628a52a2fcafb6121ad66f62e88235c74

Revision e559e6f4 (diff)
Added by Christian Blau 4 months ago

Density similarity measure and gradient

Adding a class that allows evaluating the similarity of three dimensional
densities and their gradient given a comparison method, along with a
concrete implementation of the inner-product as a comparison method.

refs #2282

Change-Id: Id73f237e123bed14241227b84cbf83970a09ff49

Revision 0293bd9c (diff)
Added by Christian Blau 4 months ago

Mrc format header to lattice coordinate transform

Extracts a coordinate transformation into lattice coordinates from the mrc density
header, as well as the three dimensional grid extents.

refs #2282

Change-Id: I8c01a370e3010add29b8810e2f3320b9ab428c02

Revision 5bbf73ed (diff)
Added by Christian Blau 4 months ago

Allow non-uniform Gaussians

For three-dimensional Gaussian spreading, as well as density guided force
evaluation, allow spreading with different widths along x, y, and z-direction.

This required to account for the common, slightly non-uniform grid
spacing in input densities.

Added some simplifactions for the vector operations based on newly
implemented functions in for BasicVector in a previous patch.

refs #2282

Change-Id: I730897fe672d2444a04ea68fea83e2273b359b27

Revision 1f0c6081 (diff)
Added by Christian Blau 4 months ago

Add coordinate scaling class and extend TranslateAndScale

- Make TranslateAndScale copyable and movable.
- Make coordinate transformation application const on the
TranslateAndScale object itself
- Add a function that returns the scale operation only without translation.
- Add a coordinate scaling class that also allows to inverse the scaling, ignoring zero input.

refs #2282

Change-Id: I73f9bd892f1f81692e05ac6b2fece2e7be9e8e69

Revision f7ac60a1 (diff)
Added by Christian Blau 4 months ago

Add class to look up density fitting amplitudes

Adds a look-up for the weight used for spreading atoms on a grid and
calculate forces on them.

refs #2282

Change-Id: Iadd26f04265aa85ad30a827c92f2c5d5fc48ef85

Revision 8e7615a8 (diff)
Added by Test User 2 months ago

Move densityfitting options and output into seperate file

Exposes density fitting options and output interface for testing.

Added test for density fitting options class.

refs: #2282

Change-Id: I0107217b5af86abf1ed298dd77553c4c24b1d3f2

Revision c3d15cd8 (diff)
Added by Test User about 2 months ago

Index groups for MdModules from grompp

Make index groups available for mdrun modules during preprocessing.
Notify mdrun modules with a DefaultIndexGroupsAndNames class that
translates index group names into atom indices.

refs #2282

Change-Id: Idc1bc3a5840e7b7ec3c82904416887b02417c92c

Revision 8bec4630 (diff)
Added by Test User about 2 months ago

Add index group option to density fitting

Adds the option to select a specific group for density fitting, stores
the indices in the tpr file and restores them after reading.

Adapted the densityfittingoptions test to test fixture to better reflect
the increased required complexity for testing the storage of indices.

refs #2282

Change-Id: I3e33508bbcadf968b0ddf978a73e7f6cf5753b5b

Revision ecd9ba25 (diff)
Added by Test User about 2 months ago

Access to GaussTransform3D data

Return a non-constant view to the GaussTransform3D to allow
sum reduction of Gauss transformation data via MPI. The alternative
to have the GaussTransform3D take a commrec to perform the
sum reduction itself leads to a cyclic dependency between
math and mdtypes modules.

refs #2282

Change-Id: I8b786965c3562ed84fe4e97fa0f68cf89ea0f03a

Revision 87630dc4 (diff)
Added by Test User about 2 months ago

Add density fitting parameters

Adds the mdp parameters that are required for running a
density fitting simulation.

refs #2282

Change-Id: Id22ceee0f0036450173522bd5e85af06542a2c39

Revision 22fcc125 (diff)
Added by Test User about 2 months ago

Implementing the density fitting force provider

refs #2282

Change-Id: Ib65679a82168f18992920f4529596db2494cccc3

Revision d7d84c60 (diff)
Added by Test User about 2 months ago

Constructing local atom sets for density fitting

Constructs local atom sets for density fitting during simulation
setup.

refs #2282

Change-Id: I0300376a6818f3ad8bb6474b09a9b0f1cbb46eae

Revision b2c7e349 (diff)
Added by Test User about 1 month ago

Add checkpointing for MdModules

Adds the possiblity to write and read from checkpoint files for MdModules.

refs #2282

Change-Id: Id3ab57d1c9a40b6e1fb0193430e5b66ffa87d787

Revision 65c6f53a (diff)
Added by Test User about 1 month ago

Add reference density file option to densityfitting

Add option to density fitting to read reference density for
density fitting code.

refs #2282

Change-Id: I598b679a3c1c63b913a104d7ccff8e1643b3946d

Revision 4db1c761 (diff)
Added by Test User about 1 month ago

Periodic boundary correction for density fitting

Corrects the periodic boundaries for density fittinng
such that always the periodic image closest to the center
of the density is chosen.

ref #2282

Change-Id: I46d9546d7dbb659e03070b4ba6ffd401eb7368c4

Revision 9ccad13e (diff)
Added by Test User about 1 month ago

Adding energy output field for density fitting

Added a new energy output field for energies that stem from the density
fitting code.

refs #2282

Change-Id: I78c64213f4b958f00a938f0a67c586c2f4e9033e

Revision ff37dc30 (diff)
Added by Test User about 1 month ago

Adding every N-steps option to densityfitting

Adds an option to the density fitting module to apply forces only every
N steps.

refs #2282

Change-Id: I8d264734e1d9a82ff9717d270552e38a28b34729

Revision 1b69a440 (diff)
Added by Test User about 1 month ago

Add option to normalize densities prior to density fitting

Adds an mdp option that makes sure that the sum of voxel values
for both input densities is unity.

refs #2282

Change-Id: I9d3c63d7548bca72a083cdafa6818ded4fcdd400

Revision eb590a14 (diff)
Added by Test User about 1 month ago

Relative entropy as density similarity measure

Adds a new density similarity measure that uses the relative entropy of
two densities to compare them to one another. Also adds the gradient of
this measure.

refs #2282

Change-Id: I3be18ec3fad3c2551ce7c9fe896dc257d5789cdf

Revision aa666e87 (diff)
Added by Test User about 1 month ago

Account for reference density origin shift in pbc correction

When the refernce density is shifted from the origin, this shift has to
be added to the reference density center. This patch fixes incorrect pbc
correction behaviour, due to not taking this shift into account.

refs #2282

Change-Id: I953f717ecf6ab53985a59195eab896d50965039e

Revision bee41569 (diff)
Added by Test User about 1 month ago

Use the current mass instead of massA in density fitting code.

When using masses to determine the weights of the spread amplitudes,
massA instead of the correct massT was used. This commit fixes this
behavior.

refs #2282

Change-Id: I58deb14b4b45c122ab3c9caf3f6b76c89f9ae739

Revision c6a5224c (diff)
Added by Test User about 1 month ago

Densityfitting write int64_t indices instead of gmx::index

When serializing atom indices to a key-value-tree that is subsequently
written to file, the type of the indices needs to be independent of
gmx::index (aka ptrdiff_t) type definition.

This patch introduces this behaviour.

refs #2282

Change-Id: I1384158204354e2267674c266f91c133a01bc5f7

Revision 6857a910 (diff)
Added by Test User about 1 month ago

End to end test for density fitting

Implement an end to end test for the density fitting module.

refs #2282

Change-Id: Ic94013cae6820ccf6fa7415887b280daff25d83c

Revision dd779369 (diff)
Added by Test User about 1 month ago

Add comments to grompp generated .mdp output for densityfitting

Adds additional explanatory comments to the grompp generated mdout.mdp
file for densityfitting.

refs #2282

Change-Id: I496cccced2cc87cb2434f877c0caf04cb29c1903

Revision 27c22069 (diff)
Added by Test User about 1 month ago

Add release note for density fitting.

Adds a release note that density-guided simulations are now available.

refs #2282

Change-Id: I2ae6896ba32cadfb2029a5c923db6fd76ddbf620

Revision 57b52ad3 (diff)
Added by Test User about 1 month ago

Manual entry for density guided simulations

Extends the reference manual with a section for density guided
simulations.

refs #2282

Change-Id: Ie3017fa5e75fcd2c809c20782300f759182486a8

Revision 8d8cd485 (diff)
Added by Test User 28 days ago

Add mdp option documentation for density fitting.

Documents the .mdp options for the density fitting module.

refs #2282

Change-Id: Ic2cb616ca13627553a36d739d86192b0fcb0e239

Revision ec5a5533 (diff)
Added by Test User 27 days ago

Consistently change densityfitting spreading weight to charge

In the mdp options, the density-guided-simulation-atom-spreading weight
has been changed from charges to "charge" but within the code and in an
.mdp file comment the plural charges was still used. Changed now all
over to make this consistent.

refs #2282

Change-Id: I055743c8301e4ad18ceed5670c62be0a93c2d5f1

Revision 57466565 (diff)
Added by Test User 23 days ago

Add cross-correlation as density simlarity measure

Add the canonical cross-correlation as similarity measure for densities

refs #2282

Change-Id: Idec820f38dab07c5ee8281944855a532a587a2ff

Revision 69f923e5 (diff)
Added by Test User 22 days ago

Amendments to density fitting manual section

- Fixed a sign error in the energy and force definition
- Added performance considerations
- Fixed whitespace
- Changed vector notation to mathbf as in the other parts of the manual
- Added pressure-coupling considerations
- Added considerations when using multiple-time-stepping

refs #2282

Change-Id: I8421ccf09ac960fa04508234e738967f51a27fab

History

#1 Updated by Christian Blau almost 2 years ago

Current functionality summary:
  • The module that does on-the-fly fitting to density maps is implemented using the correlation-based density fitting potential V = k*(1 - c.c.) as described by Orzechowski and Tama in Biophys. J. 2008 "Flexible Fitting of High-Resolution X-Ray Structures into Cryoelectron Microscopy Maps Using Biased Molecular Dynamics Simulations". Other alternatives can/will also be added.
  • deal with CCP4 density maps (read/write/embed to .tpr)
  • spread out discrete atomic positions on a density grid using Gaussian functions
  • calculate the correlation coefficient (c.c.) between the experimental and the simulated density and derive the corresponding atomic forces
  • read in density fitting .mdp parameters (which index group to submit to the potential, what k and sigma to use)
  • allow the user to implement time-dependent fitting protocols by giving k and sigma as a function of time (similar to the simulated annealing protocol)
Todo
  • get rid of the dependency on the CCP4 library cmaplib. Christian already has code to do that
  • restructure the code, move it to more adequate places in the GROMACS source tree
  • implement the mdrun part of the fitting as a proper force provider module
  • join the g_map and the "tide" tools into a single trajectory analysis module
  • implement a smarter way to do the calculation in parallel, currently the grid reduction limits the parallel scaling for large grids
  • implement more tests
  • properly document everything
  • incorporate scaling factors for the different atom types

#2 Updated by Christian Blau almost 2 years ago

  • Related to Task #2281: Represent data on regularly spaced N-dimensional grids. added

#3 Updated by Christian Blau almost 2 years ago

  • Related to Feature #1877: Support reading 3d-grid data from cryo-electron microscopy tomography and X-ray crystallography added

#4 Updated by Gerrit Code Review Bot almost 2 years ago

Gerrit received a related patchset '11' for Issue #2282.
Uploader: Carsten Kutzner ()
Change-Id: gromacs~master~I5f18c33e77d68554456c55c6eedfe6a16d7b3ff0
Gerrit URL: https://gerrit.gromacs.org/6970

#5 Updated by Camilo Aponte almost 2 years ago

Christian Blau wrote:

Current functionality summary:
  • The module that does on-the-fly fitting to density maps is implemented using the correlation-based density fitting potential V = k*(1 - c.c.) as described by Orzechowski and Tama in Biophys. J. 2008 "Flexible Fitting of High-Resolution X-Ray Structures into Cryoelectron Microscopy Maps Using Biased Molecular Dynamics Simulations". Other alternatives can/will also be added.
  • deal with CCP4 density maps (read/write/embed to .tpr)
  • spread out discrete atomic positions on a density grid using Gaussian functions
  • calculate the correlation coefficient (c.c.) between the experimental and the simulated density and derive the corresponding atomic forces
  • read in density fitting .mdp parameters (which index group to submit to the potential, what k and sigma to use)
  • allow the user to implement time-dependent fitting protocols by giving k and sigma as a function of time (similar to the simulated annealing protocol)
Todo
  • get rid of the dependency on the CCP4 library cmaplib. Christian already has code to do that
  • restructure the code, move it to more adequate places in the GROMACS source tree
  • implement the mdrun part of the fitting as a proper force provider module
  • join the g_map and the "tide" tools into a single trajectory analysis module
  • implement a smarter way to do the calculation in parallel, currently the grid reduction limits the parallel scaling for large grids
  • implement more tests
  • properly document everything
  • incorporate scaling factors for the different atom types.
  • implement masking, subtraction, and division of reference and computed maps.
  • tide: implement output option with the global correlation between the reference and the computed map (xvg format).
  • tide: implement output option with the local correlation between the reference and the computed map (map format).
  • tide: implement output option with the the global correlation as a function of time (xvg format).
  • tide: implement scaling option of output maps in units of sigma and removing average value over the entire lattice.
  • tide: implement input file with Gaussian coefficients based on the atom types and some internal default values, in case the input file is not provided.

#6 Updated by Joe Jordan over 1 year ago

  • Related to Task #2461: Expose more info from t_atom to trajectory analysis framework added

#7 Updated by Gerrit Code Review Bot 9 months ago

Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~I883cdb6a41d7277669e100320e230904856d1318
Gerrit URL: https://gerrit.gromacs.org/8919

#8 Updated by Gerrit Code Review Bot 8 months ago

Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~Ia4d63faadeb595d7a2eeb4ac4760c211c0d2aac6
Gerrit URL: https://gerrit.gromacs.org/9089

#9 Updated by Gerrit Code Review Bot 8 months ago

Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~I5f0c839cd2f516e18dd6d5e5f76fb2f06fb40eef
Gerrit URL: https://gerrit.gromacs.org/9203

#10 Updated by Gerrit Code Review Bot 8 months ago

Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~Ib63ba97edf34928e47b7a9d8461f9817d2cc7e59
Gerrit URL: https://gerrit.gromacs.org/9208

#11 Updated by Gerrit Code Review Bot 7 months ago

Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~I0732a78747582a6e23bba1e141d73c4cda421011
Gerrit URL: https://gerrit.gromacs.org/9280

#12 Updated by Gerrit Code Review Bot 7 months ago

Gerrit received a related patchset '2' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~Ifd43cedb137c0beeb320e1299df8bae9916c8ea8
Gerrit URL: https://gerrit.gromacs.org/9281

#13 Updated by Gerrit Code Review Bot 7 months ago

Gerrit received a related DRAFT patchset '1' for Issue #2282.
Uploader: Christian Blau ()
Change-Id: gromacs~master~If8bafcc10b2720857b2fceca311cf6ba3ea98a5c
Gerrit URL: https://gerrit.gromacs.org/9344

Also available in: Atom PDF