Feature #2615

Switch to Python3

Added by Pascal Merz 8 months ago. Updated 23 days ago.

core library
Target version:


Currently, all python scripts in GROMACS are written in Python2. Python2 support ends 01/01/2020 (,, so a move to Python3 becomes necessary.

Python3 versions

Python versions generally have a 5 year lifetime after release. The oldest currently supported Python3 version is 3.4.

This probably excludes
  • Python <= 3.2 Compatibility with Python2.7 is difficult, and has been out-of-life since March 2016.
  • Python3.3 Compatibility with Python2.7 is easy, but has been out-of-life since September 2017. Also, recent Sphinx releases (>= 1.7.0) don't support Python3.3.

We could therefore decide to support 3.4+, or 3.3+ and move to 3.4+ once we require a newer Sphinx version (currently, we require 1.6.1).

Possible strategies

Python2/3 compatibility

Translate code to valid Python3, use `__future__` imports to guarantee compatibility to Python2.7. Accept possibly slightly lower performance in py2 when moving from `xrange` to `range`, `iterkeys()` to `keys()`, etc. Keep Python2.7 as required version in CMake (possibly first trying to find Python3.3/3.4+, and falling back to 2.7 if necessary - would need to check if that works reliably).

Move to Python3

Translate code to valid Python3 without keeping Python2.7 compatibility. Move required version to 3.3/3.4+. (Obviously, dropping Python2 could be done at a later stage.)

Required changes

The python scripts can be divided in different categories having different scope and different amount of required changes to be python3-compatible: (The assessment of what needs to be changed largely relies on help of tools like `futurize`, but with manual changes allowing for compatibility without the introduction of additional dependencies. Obviously, the scripts with "No changes needed" need to be thoroughly checked with Python3.)

  • Documentation scripts
    - docs/*.py
    - docs/doxygen/*.py
    These scripts are needed to generate the documentation and have a fair amount of incompatibilities with Python 3. The main changes needed involve print statements, divisions, and some changes related to the move from lists (in py2) to iterators (in py3). Additionally, these scripts rely on the `cmp` function not present in python3. This needs to be replaced by rich comparisons.
  • Admin scripts
    - admin/
    - admin/builds/*.py
    Probably no changes needed.
  • The nonbonded kernel scripts
    - src/gromacs/mdlib/nbnxn_kernels/nbnxn_kernel_file_generator/
    Probably no changes needed.
    - src/gromacs/gmxlib/nonbonded/nb_kernel_*/*.py
    - src/gromacs/gmxlib/nonbonded/preprocessor/
    These scripts can probably be left untouched, as per Erik Lindahl's comment in

You can leave out the scripts in the nonbonded subdirectory. They are
only used to regenerate new group-style kernels, and I don't expect
we'll ever do that again since the present ones are scheduled to be
removed and fully replaced by the verlet-style kernels.

  • Physical validation scripts
    - tests/physicalvalidation/ and subfolders
    The physical validation tests are already python2/3 compatible.
  • Misc
    - src/gromacs/selection/tests/
    Has divisions, but none of them behaves differently under python3 -> no changes needed.
    - scripts/
    This script seems very old, it is not even valid python 2.7. Probably best left unchanged or deleted.
  • CMake files
    - cmake/FindPythonModule.cmake
    Small change is needed to ensure compatibility of the `find_python_module` function with python3.
    - cmake/FindPythonModule.cmake
    - docs/CMakeLists.txt
    These files explicitly require Python 2.7 via `find_package()` - needs to be changes once Python 3 is required.


How can we test that no errors are introduced? For the documentation scripts updated in, the output of `make webpage` was compared. Is that sufficient? What about the other scripts?

Default future imports

As proposed by Eric Irrgang in, if Python2 compatibility should be retained for a longer time, it might be worth to include a set of imports to all Python files to avoid future incompatibilities, e.g.

    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    from __future__ import unicode_literals

Related issues

Related to GROMACS - Task #2831: Bump required version numbers of infrastructure for 2020New

Associated revisions

Revision ad2ac649 (diff)
Added by Eric Irrgang 1 day ago

Add Flake8 linting to gmxapi tests.

Add the flake8 package to requirements-test.txt and a run_flake8
docker entry point. This change does not require Python source
checking, but facilitates it.

Refs: #2896
Refs: #2615

Gerrit patch set 10419/1


#1 Updated by Gerrit Code Review Bot 8 months ago

Gerrit received a related patchset '8' for Issue #2615.
Uploader: Pascal Merz ()
Change-Id: gromacs~master~I2edbe213bc6401563934ce56f5fd6a39886d3095
Gerrit URL:

#2 Updated by Roland Schulz 8 months ago

Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems ( What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?

#3 Updated by Pascal Merz 8 months ago

Roland Schulz wrote:

Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems (

I would suggest to avoid any new dependencies. There are ways to keep the code py2/py3 compatible without adding dependencies. For example, I would suggest to use items(), keys(), values(), range(), etc - with an explicit list(...) or iter(...) around if necessary. This reduces the efficiency using Python2, but I wouldn't expect this to be noticeable in our case based on some tests I ran, and the documentation generation is probably not the most performance-critical part of GROMACS. This is the approach I used in

What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?

No thoughts / preferences on this, but having a quick look at the scripts, compatibility seems doable if needed.

#4 Updated by Mark Abraham 7 months ago

  • Category set to core library
  • Target version changed from 2019 to 2020

Paul had a look at whether we could e.g. build the docs or API support with v3. Currently releng is implemented in Python v2, and runs the build script from the repo in the same python process. But anything called via cmake --build could be arranged to start in a v3 interpreter. But since anyway we aren't targeting v3 for anything in GROMACS 2019, we can leave all of this on the table for 2019, and go all in on v3 (including for gmxapi) for 2020.

Pascal's work is still up in Gerrit when the time comes.

#5 Updated by Eric Irrgang 24 days ago

  • Related to Task #2831: Bump required version numbers of infrastructure for 2020 added

#6 Updated by Eric Irrgang 24 days ago

Note: Python 3.4 has reached end-of-life.

#7 Updated by Eric Irrgang 23 days ago

We should consider how best to apply linting / PEP-8 compliance checking during the submission / code-review process.

Also available in: Atom PDF