Switch to Python3
Currently, all python scripts in GROMACS are written in Python2. Python2 support ends 01/01/2020 (https://www.python.org/dev/peps/pep-0373/#id4, https://pythonclock.org/), so a move to Python3 becomes necessary.
Python versions generally have a 5 year lifetime after release. The oldest currently supported Python3 version is 3.4.This probably excludes
- Python <= 3.2 Compatibility with Python2.7 is difficult, and has been out-of-life since March 2016.
- Python3.3 Compatibility with Python2.7 is easy, but has been out-of-life since September 2017. Also, recent Sphinx releases (>= 1.7.0) don't support Python3.3.
We could therefore decide to support 3.4+, or 3.3+ and move to 3.4+ once we require a newer Sphinx version (currently, we require 1.6.1).
Translate code to valid Python3, use `__future__` imports to guarantee compatibility to Python2.7. Accept possibly slightly lower performance in py2 when moving from `xrange` to `range`, `iterkeys()` to `keys()`, etc. Keep Python2.7 as required version in CMake (possibly first trying to find Python3.3/3.4+, and falling back to 2.7 if necessary - would need to check if that works reliably).
Move to Python3¶
Translate code to valid Python3 without keeping Python2.7 compatibility. Move required version to 3.3/3.4+. (Obviously, dropping Python2 could be done at a later stage.)
The python scripts can be divided in different categories having different scope and different amount of required changes to be python3-compatible: (The assessment of what needs to be changed largely relies on help of tools like `futurize`, but with manual changes allowing for compatibility without the introduction of additional dependencies. Obviously, the scripts with "No changes needed" need to be thoroughly checked with Python3.)
- Documentation scripts
These scripts are needed to generate the documentation and have a fair amount of incompatibilities with Python 3. The main changes needed involve print statements, divisions, and some changes related to the move from lists (in py2) to iterators (in py3). Additionally, these scripts rely on the `cmp` function not present in python3. This needs to be replaced by rich comparisons.
- Admin scripts
Probably no changes needed.
- The nonbonded kernel scripts
Probably no changes needed.
These scripts can probably be left untouched, as per Erik Lindahl's comment in https://gerrit.gromacs.org/#/c/6621/:
You can leave out the scripts in the nonbonded subdirectory. They are
only used to regenerate new group-style kernels, and I don't expect
we'll ever do that again since the present ones are scheduled to be
removed and fully replaced by the verlet-style kernels.
- Physical validation scripts
- tests/physicalvalidation/ and subfolders
The physical validation tests are already python2/3 compatible.
Has divisions, but none of them behaves differently under python3 -> no changes needed.
This script seems very old, it is not even valid python 2.7. Probably best left unchanged or deleted.
- CMake files
Small change is needed to ensure compatibility of the `find_python_module` function with python3.
These files explicitly require Python 2.7 via `find_package()` - needs to be changes once Python 3 is required.
How can we test that no errors are introduced? For the documentation scripts updated in https://gerrit.gromacs.org/#/c/6621/, the output of `make webpage` was compared. Is that sufficient? What about the other scripts?
Default future imports¶
As proposed by Eric Irrgang in https://gerrit.gromacs.org/#/c/6621/, if Python2 compatibility should be retained for a longer time, it might be worth to include a set of imports to all Python files to avoid future incompatibilities, e.g.
from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals
Use Python 3
This updates the Python scripts shipped with GROMACS to use Python 3.5+.
Most changes done are either due to `__cmp__` not being available in py3
(replaced by `functools.total_ordering`), py2-iterators being replaced by
py3-iterators, or print statements being replaced by functions.
Additionaly, some changes in the Cmake files were made to select the
different Python version.
All python scripts having an explicit `#!/usr/bin/env python2` as a first
line were checked and updated. Larger changes were only needed for the
documentation scripts, other scripts did not require any changes. Scripts
not explicitly using shebangs were also checked, but no changes were
needed. The following scripts were not touched:
- all scripts under tests/physicalvalidation/ (already py3 compatible)
- all scripts under python_packaging/
- scripts/make_gromos_rtp.py - this file is not valid py2.7
#2 Updated by Roland Schulz 10 months ago
Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems (http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items)? What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?
#3 Updated by Pascal Merz 10 months ago
Roland Schulz wrote:
Do we add a dependency on either the future or six package? If not how would I do e.g. iteritems (http://python-future.org/compatible_idioms.html#iterating-through-dict-keys-values-items)?
I would suggest to avoid any new dependencies. There are ways to keep the code py2/py3 compatible without adding dependencies. For example, I would suggest to use items(), keys(), values(), range(), etc - with an explicit list(...) or iter(...) around if necessary. This reduces the efficiency using Python2, but I wouldn't expect this to be noticeable in our case based on some tests I ran, and the documentation generation is probably not the most performance-critical part of GROMACS. This is the approach I used in https://gerrit.gromacs.org/c/6621/.
What do we plan to do with releng? Do we preserve 2 compatibility there or make it Python 3 only?
No thoughts / preferences on this, but having a quick look at the scripts, compatibility seems doable if needed.
#4 Updated by Mark Abraham 8 months ago
- Category set to core library
- Target version changed from 2019 to 2020
Paul had a look at whether we could e.g. build the docs or API support with v3. Currently releng is implemented in Python v2, and runs the build script from the repo in the same python process. But anything called via
cmake --build could be arranged to start in a v3 interpreter. But since anyway we aren't targeting v3 for anything in GROMACS 2019, we can leave all of this on the table for 2019, and go all in on v3 (including for gmxapi) for 2020.
Pascal's work is still up in Gerrit when the time comes.
#6 Updated by Eric Irrgang 3 months ago
Note: Python 3.4 has reached end-of-life. https://devguide.python.org/#status-of-python-branches
#8 Updated by Mark Abraham about 1 month ago
We mooted in the dev telco that python 3.5 sounds like a good minimum to require for GROMACS 2020. We want to balance what is supported by python devs, available on LTS desktops that they'll have in the timeframe of the next release from master branch, and available on HPC resources.
Python 3.5 is maintained until sep 2020
Python3 is 3.5 in 16.04 and 3.6 in 18.04.
Of HPC sites, archer, beskow, piz daint, juwels, bsc-power9, marconi all had python 3.6 (among others)?
So 3.5 seems like a good minimum requirement. Moving fowrard we'll test in Jenkins the oldest supported and the latest (at least).