Feature #2282
Density map toolset
Description
Assemble functionality to deal with density maps as e.g. from cryo electron microscopy.
- i/o routines for density maps
- a module to allow fitting to cryo-EM densities during the simulation
- analysis tool to compute average densities from a trajectory.
Related issues
Associated revisions
densityfitting - density file format (mrc) header.
Implements mrc/cc4/map format file headers as decribed in
"EMDB Map Distribution Format Description Version 1.01 (c)
emdatabank.org 2014"
Reading and writing is part of a follow-up patch.
Change-Id: Ifd43cedb137c0beeb320e1299df8bae9916c8ea8
densityfitting - MDModule declaration
Declaring the infrastructure for running molecular dynamics simulations with
addional forces that are derived from densities.
Adds a IForceProvider for density fitting simulation that is set up with
its DensityFittingParameters that are in turn built from DensityFittingOptions.
Change-Id: I0732a78747582a6e23bba1e141d73c4cda421011
One dimensional Gaussian function on lattice.
Evaluate a Gaussian of width sigma in one dimension on lattice points with unit spacing.
refs #2282
Change-Id: I57a513f05af44323e7109c4560438dc023793c4c
Three dimensional gauss transform
Added class that adds gaussians functions with specified
amplitudes and coordinates onto a three-dimensional scalar field.
Change-Id: I07ea2cdd219b34afc402975a0b0e8a52eb71acaf
refs: #2282
density fitting forces
Evaluate forces on atoms that increase a measure of the goodness of fit to a reference density.
The density fitting forces are evaluated as the gradient of the goodness of fit of a comparison density is generated from atoms by spreading on a grid via a Gaussian kernel to the reference density.
Applying the chain rule for this gradient yields the density fitting force as the inner product of the derivative of the comparison density to reference density goodness-of-fit function at all density values and the gradient of the spreading kernel used to simulate the density.
This patch assumes that the comparison density to reference density goodness of fit derivative density is given, methods that aid the evaluation of this derivative density are part of a child patch.
refs #2282
Change-Id: I4d93fffbdb41753f9a0a4af921011aca02ade69c
mrc density map reading/writing
Add a class that combines header and mrc data handling for float values
to enable reading and writing of mrc files.
refs #2282
Change-Id: Idd84ab2628a52a2fcafb6121ad66f62e88235c74
Density similarity measure and gradient
Adding a class that allows evaluating the similarity of three dimensional
densities and their gradient given a comparison method, along with a
concrete implementation of the inner-product as a comparison method.
refs #2282
Change-Id: Id73f237e123bed14241227b84cbf83970a09ff49
Mrc format header to lattice coordinate transform
Extracts a coordinate transformation into lattice coordinates from the mrc density
header, as well as the three dimensional grid extents.
refs #2282
Change-Id: I8c01a370e3010add29b8810e2f3320b9ab428c02
Allow non-uniform Gaussians
For three-dimensional Gaussian spreading, as well as density guided force
evaluation, allow spreading with different widths along x, y, and z-direction.
This required to account for the common, slightly non-uniform grid
spacing in input densities.
Added some simplifactions for the vector operations based on newly
implemented functions in for BasicVector in a previous patch.
refs #2282
Change-Id: I730897fe672d2444a04ea68fea83e2273b359b27
Add coordinate scaling class and extend TranslateAndScale
- Make TranslateAndScale copyable and movable.
- Make coordinate transformation application const on the
TranslateAndScale object itself
- Add a function that returns the scale operation only without translation.
- Add a coordinate scaling class that also allows to inverse the scaling, ignoring zero input.
refs #2282
Change-Id: I73f9bd892f1f81692e05ac6b2fece2e7be9e8e69
Add class to look up density fitting amplitudes
Adds a look-up for the weight used for spreading atoms on a grid and
calculate forces on them.
refs #2282
Change-Id: Iadd26f04265aa85ad30a827c92f2c5d5fc48ef85
Move densityfitting options and output into seperate file
Exposes density fitting options and output interface for testing.
Added test for density fitting options class.
refs: #2282
Change-Id: I0107217b5af86abf1ed298dd77553c4c24b1d3f2
Index groups for MdModules from grompp
Make index groups available for mdrun modules during preprocessing.
Notify mdrun modules with a DefaultIndexGroupsAndNames class that
translates index group names into atom indices.
refs #2282
Change-Id: Idc1bc3a5840e7b7ec3c82904416887b02417c92c
Add index group option to density fitting
Adds the option to select a specific group for density fitting, stores
the indices in the tpr file and restores them after reading.
Adapted the densityfittingoptions test to test fixture to better reflect
the increased required complexity for testing the storage of indices.
refs #2282
Change-Id: I3e33508bbcadf968b0ddf978a73e7f6cf5753b5b
Access to GaussTransform3D data
Return a non-constant view to the GaussTransform3D to allow
sum reduction of Gauss transformation data via MPI. The alternative
to have the GaussTransform3D take a commrec to perform the
sum reduction itself leads to a cyclic dependency between
math and mdtypes modules.
refs #2282
Change-Id: I8b786965c3562ed84fe4e97fa0f68cf89ea0f03a
Add density fitting parameters
Adds the mdp parameters that are required for running a
density fitting simulation.
refs #2282
Change-Id: Id22ceee0f0036450173522bd5e85af06542a2c39
Implementing the density fitting force provider
refs #2282
Change-Id: Ib65679a82168f18992920f4529596db2494cccc3
Constructing local atom sets for density fitting
Constructs local atom sets for density fitting during simulation
setup.
refs #2282
Change-Id: I0300376a6818f3ad8bb6474b09a9b0f1cbb46eae
Add checkpointing for MdModules
Adds the possiblity to write and read from checkpoint files for MdModules.
refs #2282
Change-Id: Id3ab57d1c9a40b6e1fb0193430e5b66ffa87d787
Add reference density file option to densityfitting
Add option to density fitting to read reference density for
density fitting code.
refs #2282
Change-Id: I598b679a3c1c63b913a104d7ccff8e1643b3946d
Periodic boundary correction for density fitting
Corrects the periodic boundaries for density fittinng
such that always the periodic image closest to the center
of the density is chosen.
ref #2282
Change-Id: I46d9546d7dbb659e03070b4ba6ffd401eb7368c4
Adding energy output field for density fitting
Added a new energy output field for energies that stem from the density
fitting code.
refs #2282
Change-Id: I78c64213f4b958f00a938f0a67c586c2f4e9033e
Adding every N-steps option to densityfitting
Adds an option to the density fitting module to apply forces only every
N steps.
refs #2282
Change-Id: I8d264734e1d9a82ff9717d270552e38a28b34729
Add option to normalize densities prior to density fitting
Adds an mdp option that makes sure that the sum of voxel values
for both input densities is unity.
refs #2282
Change-Id: I9d3c63d7548bca72a083cdafa6818ded4fcdd400
Relative entropy as density similarity measure
Adds a new density similarity measure that uses the relative entropy of
two densities to compare them to one another. Also adds the gradient of
this measure.
refs #2282
Change-Id: I3be18ec3fad3c2551ce7c9fe896dc257d5789cdf
Account for reference density origin shift in pbc correction
When the refernce density is shifted from the origin, this shift has to
be added to the reference density center. This patch fixes incorrect pbc
correction behaviour, due to not taking this shift into account.
refs #2282
Change-Id: I953f717ecf6ab53985a59195eab896d50965039e
Use the current mass instead of massA in density fitting code.
When using masses to determine the weights of the spread amplitudes,
massA instead of the correct massT was used. This commit fixes this
behavior.
refs #2282
Change-Id: I58deb14b4b45c122ab3c9caf3f6b76c89f9ae739
Densityfitting write int64_t indices instead of gmx::index
When serializing atom indices to a key-value-tree that is subsequently
written to file, the type of the indices needs to be independent of
gmx::index (aka ptrdiff_t) type definition.
This patch introduces this behaviour.
refs #2282
Change-Id: I1384158204354e2267674c266f91c133a01bc5f7
End to end test for density fitting
Implement an end to end test for the density fitting module.
refs #2282
Change-Id: Ic94013cae6820ccf6fa7415887b280daff25d83c
Add comments to grompp generated .mdp output for densityfitting
Adds additional explanatory comments to the grompp generated mdout.mdp
file for densityfitting.
refs #2282
Change-Id: I496cccced2cc87cb2434f877c0caf04cb29c1903
Add release note for density fitting.
Adds a release note that density-guided simulations are now available.
refs #2282
Change-Id: I2ae6896ba32cadfb2029a5c923db6fd76ddbf620
Manual entry for density guided simulations
Extends the reference manual with a section for density guided
simulations.
refs #2282
Change-Id: Ie3017fa5e75fcd2c809c20782300f759182486a8
Add mdp option documentation for density fitting.
Documents the .mdp options for the density fitting module.
refs #2282
Change-Id: Ic2cb616ca13627553a36d739d86192b0fcb0e239
Consistently change densityfitting spreading weight to charge
In the mdp options, the density-guided-simulation-atom-spreading weight
has been changed from charges to "charge" but within the code and in an
.mdp file comment the plural charges was still used. Changed now all
over to make this consistent.
refs #2282
Change-Id: I055743c8301e4ad18ceed5670c62be0a93c2d5f1
Add cross-correlation as density simlarity measure
Add the canonical cross-correlation as similarity measure for densities
refs #2282
Change-Id: Idec820f38dab07c5ee8281944855a532a587a2ff
Amendments to density fitting manual section
- Fixed a sign error in the energy and force definition
- Added performance considerations
- Fixed whitespace
- Changed vector notation to mathbf as in the other parts of the manual
- Added pressure-coupling considerations
- Added considerations when using multiple-time-stepping
refs #2282
Change-Id: I8421ccf09ac960fa04508234e738967f51a27fab
Adaptive force scaling for densityfitting
Scales the force adaptively for density-guided-simulations
refs #2282
Change-Id: I96310f498cf2fae9f7385a9396f1253b760d135e
Manual entry and .mdp option for adaptive force scaling
Adds a manual entry and extends the mdp option section to describe the
adpative force scaling parameters.
refs #2282
Change-Id: Ifb2e4af256282340f1111262f91e5d73d93c3208
Fix nst in density-fitting module
An error in the nst variable led to forces from density-guided
simulations to be applied every step, because its internal step
counter would stay at 0.
Change-Id: Ib8b33b407bfe9c56bab497f7e2d505843675afca
refs: #2282
Fix cross-correlation calculation instability
Fixes a fail in calculating the cross-correlation for density guided
simulations in cases where the covariance between densities values had
been negative.
refs #2282
Change-Id: Ibd2a88768eba25c4836b8f692fd9d61e8960e073
InMemorySerializer with endianess swap.
Adds an InMemorySerializer that swaps endianess. Required for reading
data that is not handled with XDR serializers and was written with
different endianess, e.g., mrc files for density guided simulations.
refs #2282
Change-Id: Ice96fe8684bb0b920c22dc7d3c2b2087260eb9ad
Read mrc density map format with all endianesses.
Swaps the endianess of reading mrc density maps if header is not sane.
refs: #2282
Change-Id: I417e402b2ab801c22d9963fabb0e60b186288651
Fix check-pointing for density guided simulations
Previously, checkpointing for density-guided simulations did not take
into account that the checkpointing occurs in the middle of the md-loop.
Now, the state is stored before the force-calculations, so that
simulations are restarted correctly.
Change-Id: Ide4ebf94f2801acbe476a5cd348af66d0a6f28ac
History
#1 Updated by Christian Blau over 3 years ago
- The module that does on-the-fly fitting to density maps is implemented using the correlation-based density fitting potential V = k*(1 - c.c.) as described by Orzechowski and Tama in Biophys. J. 2008 "Flexible Fitting of High-Resolution X-Ray Structures into Cryoelectron Microscopy Maps Using Biased Molecular Dynamics Simulations". Other alternatives can/will also be added.
- deal with CCP4 density maps (read/write/embed to .tpr)
- spread out discrete atomic positions on a density grid using Gaussian functions
- calculate the correlation coefficient (c.c.) between the experimental and the simulated density and derive the corresponding atomic forces
- read in density fitting .mdp parameters (which index group to submit to the potential, what k and sigma to use)
- allow the user to implement time-dependent fitting protocols by giving k and sigma as a function of time (similar to the simulated annealing protocol)
- get rid of the dependency on the CCP4 library cmaplib. Christian already has code to do that
- restructure the code, move it to more adequate places in the GROMACS source tree
- implement the mdrun part of the fitting as a proper force provider module
- join the g_map and the "tide" tools into a single trajectory analysis module
- implement a smarter way to do the calculation in parallel, currently the grid reduction limits the parallel scaling for large grids
- implement more tests
- properly document everything
- incorporate scaling factors for the different atom types
#2 Updated by Christian Blau over 3 years ago
- Related to Task #2281: Represent data on regularly spaced N-dimensional grids. added
#3 Updated by Christian Blau over 3 years ago
- Related to Feature #1877: Support reading 3d-grid data from cryo-electron microscopy tomography and X-ray crystallography added
#4 Updated by Gerrit Code Review Bot over 3 years ago
Gerrit received a related patchset '11' for Issue #2282.
Uploader: Carsten Kutzner (ckutzne@gwdg.de)
Change-Id: gromacs~master~I5f18c33e77d68554456c55c6eedfe6a16d7b3ff0
Gerrit URL: https://gerrit.gromacs.org/6970
#5 Updated by Camilo Aponte over 3 years ago
Christian Blau wrote:
Current functionality summary:Todo
- The module that does on-the-fly fitting to density maps is implemented using the correlation-based density fitting potential V = k*(1 - c.c.) as described by Orzechowski and Tama in Biophys. J. 2008 "Flexible Fitting of High-Resolution X-Ray Structures into Cryoelectron Microscopy Maps Using Biased Molecular Dynamics Simulations". Other alternatives can/will also be added.
- deal with CCP4 density maps (read/write/embed to .tpr)
- spread out discrete atomic positions on a density grid using Gaussian functions
- calculate the correlation coefficient (c.c.) between the experimental and the simulated density and derive the corresponding atomic forces
- read in density fitting .mdp parameters (which index group to submit to the potential, what k and sigma to use)
- allow the user to implement time-dependent fitting protocols by giving k and sigma as a function of time (similar to the simulated annealing protocol)
- get rid of the dependency on the CCP4 library cmaplib. Christian already has code to do that
- restructure the code, move it to more adequate places in the GROMACS source tree
- implement the mdrun part of the fitting as a proper force provider module
- join the g_map and the "tide" tools into a single trajectory analysis module
- implement a smarter way to do the calculation in parallel, currently the grid reduction limits the parallel scaling for large grids
- implement more tests
- properly document everything
- incorporate scaling factors for the different atom types.
- implement masking, subtraction, and division of reference and computed maps.
- tide: implement output option with the global correlation between the reference and the computed map (xvg format).
- tide: implement output option with the local correlation between the reference and the computed map (map format).
- tide: implement output option with the the global correlation as a function of time (xvg format).
- tide: implement scaling option of output maps in units of sigma and removing average value over the entire lattice.
- tide: implement input file with Gaussian coefficients based on the atom types and some internal default values, in case the input file is not provided.
#6 Updated by Joe Jordan almost 3 years ago
- Related to Task #2461: Expose more info from t_atom to trajectory analysis framework added
#7 Updated by Gerrit Code Review Bot about 2 years ago
Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~I883cdb6a41d7277669e100320e230904856d1318
Gerrit URL: https://gerrit.gromacs.org/8919
#8 Updated by Gerrit Code Review Bot about 2 years ago
Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~Ia4d63faadeb595d7a2eeb4ac4760c211c0d2aac6
Gerrit URL: https://gerrit.gromacs.org/9089
#9 Updated by Gerrit Code Review Bot about 2 years ago
Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~I5f0c839cd2f516e18dd6d5e5f76fb2f06fb40eef
Gerrit URL: https://gerrit.gromacs.org/9203
#10 Updated by Gerrit Code Review Bot about 2 years ago
Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~Ib63ba97edf34928e47b7a9d8461f9817d2cc7e59
Gerrit URL: https://gerrit.gromacs.org/9208
#11 Updated by Gerrit Code Review Bot almost 2 years ago
Gerrit received a related patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~I0732a78747582a6e23bba1e141d73c4cda421011
Gerrit URL: https://gerrit.gromacs.org/9280
#12 Updated by Gerrit Code Review Bot almost 2 years ago
Gerrit received a related patchset '2' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~Ifd43cedb137c0beeb320e1299df8bae9916c8ea8
Gerrit URL: https://gerrit.gromacs.org/9281
#13 Updated by Gerrit Code Review Bot almost 2 years ago
Gerrit received a related DRAFT patchset '1' for Issue #2282.
Uploader: Christian Blau (cblau@gwdg.de)
Change-Id: gromacs~master~If8bafcc10b2720857b2fceca311cf6ba3ea98a5c
Gerrit URL: https://gerrit.gromacs.org/9344
Multidimensional data in canonical layout.
Introduces a multidimensional array that holds its own data and makes it
accessible via the canonical right-aligned layout for indexing the data.
refs: #2282
Change-Id: I883cdb6a41d7277669e100320e230904856d1318