Project

General

Profile

Feature #2615

Switch to Python3

Added by Pascal Merz about 1 month ago. Updated 18 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Difficulty:
uncategorized
Close

Description

Currently, all python scripts in GROMACS are written in Python2. Python2 support ends 01/01/2020 (https://www.python.org/dev/peps/pep-0373/#id4, https://pythonclock.org/), so a move to Python3 becomes necessary.

Python3 versions

https://devguide.python.org/#status-of-python-branches

Python versions generally have a 5 year lifetime after release. The oldest currently supported Python3 version is 3.4.

This probably excludes
  • Python <= 3.2 Compatibility with Python2.7 is difficult, and has been out-of-life since March 2016.
  • Python3.3 Compatibility with Python2.7 is easy, but has been out-of-life since September 2017. Also, recent Sphinx releases (>= 1.7.0) don't support Python3.3.

We could therefore decide to support 3.4+, or 3.3+ and move to 3.4+ once we require a newer Sphinx version (currently, we require 1.6.1).

Possible strategies

Python2/3 compatibility

Translate code to valid Python3, use `__future__` imports to guarantee compatibility to Python2.7. Accept possibly slightly lower performance in py2 when moving from `xrange` to `range`, `iterkeys()` to `keys()`, etc. Keep Python2.7 as required version in CMake (possibly first trying to find Python3.3/3.4+, and falling back to 2.7 if necessary - would need to check if that works reliably).

Move to Python3

Translate code to valid Python3 without keeping Python2.7 compatibility. Move required version to 3.3/3.4+. (Obviously, dropping Python2 could be done at a later stage.)

Required changes

The python scripts can be divided in different categories having different scope and different amount of required changes to be python3-compatible: (The assessment of what needs to be changed largely relies on help of tools like `futurize`, but with manual changes allowing for compatibility without the introduction of additional dependencies. Obviously, the scripts with "No changes needed" need to be thoroughly checked with Python3.)

  • Documentation scripts
    - docs/*.py
    - docs/doxygen/*.py
    These scripts are needed to generate the documentation and have a fair amount of incompatibilities with Python 3. The main changes needed involve print statements, divisions, and some changes related to the move from lists (in py2) to iterators (in py3). Additionally, these scripts rely on the `cmp` function not present in python3. This needs to be replaced by rich comparisons.
  • Admin scripts
    - admin/copyright.py
    - admin/builds/*.py
    Probably no changes needed.
  • The nonbonded kernel scripts
    - src/gromacs/mdlib/nbnxn_kernels/nbnxn_kernel_file_generator/make_verlet_simd_kernel_files.py
    Probably no changes needed.
    - src/gromacs/gmxlib/nonbonded/nb_kernel_*/*.py
    - src/gromacs/gmxlib/nonbonded/preprocessor/gmxpreprocess.py
    These scripts can probably be left untouched, as per Erik Lindahl's comment in https://gerrit.gromacs.org/#/c/6621/:

You can leave out the scripts in the nonbonded subdirectory. They are
only used to regenerate new group-style kernels, and I don't expect
we'll ever do that again since the present ones are scheduled to be
removed and fully replaced by the verlet-style kernels.

  • Physical validation scripts
    - tests/physicalvalidation/ and subfolders
    The physical validation tests are already python2/3 compatible.
  • Misc
    - src/gromacs/selection/tests/gensphere.py
    Has divisions, but none of them behaves differently under python3 -> no changes needed.
    - scripts/make_gromos_rtp.py
    This script seems very old, it is not even valid python 2.7. Probably best left unchanged or deleted.
  • CMake files
    - cmake/FindPythonModule.cmake
    Small change is needed to ensure compatibility of the `find_python_module` function with python3.
    - cmake/FindPythonModule.cmake
    - docs/CMakeLists.txt
    These files explicitly require Python 2.7 via `find_package()` - needs to be changes once Python 3 is required.

Testing

How can we test that no errors are introduced? For the documentation scripts updated in https://gerrit.gromacs.org/#/c/6621/, the output of `make webpage` was compared. Is that sufficient? What about the other scripts?

Default future imports

As proposed by Eric Irrgang in https://gerrit.gromacs.org/#/c/6621/, if Python2 compatibility should be retained for a longer time, it might be worth to include a set of imports to all Python files to avoid future incompatibilities, e.g.

    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    from __future__ import unicode_literals

History

#1 Updated by Gerrit Code Review Bot about 1 month ago

Gerrit received a related patchset '8' for Issue #2615.
Uploader: Pascal Merz ()
Change-Id: gromacs~master~I2edbe213bc6401563934ce56f5fd6a39886d3095
Gerrit URL: https://gerrit.gromacs.org/6621

#2 Updated by Roland Schulz 19 days ago

Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems (http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items)? What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?

#3 Updated by Pascal Merz 18 days ago

Roland Schulz wrote:

Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems (http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items)?

I would suggest to avoid any new dependencies. There are ways to keep the code py2/py3 compatible without adding dependencies. For example, I would suggest to use items(), keys(), values(), range(), etc - with an explicit list(...) or iter(...) around if necessary. This reduces the efficiency using Python2, but I wouldn't expect this to be noticeable in our case based on some tests I ran, and the documentation generation is probably not the most performance-critical part of GROMACS. This is the approach I used in https://gerrit.gromacs.org/c/6621/.

What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?

No thoughts / preferences on this, but having a quick look at the scripts, compatibility seems doable if needed.

Also available in: Atom PDF