Page MenuHome GnuPG

build: Python without distutils
Needs ReviewPublic

Authored by bnavigator on Dec 23 2021, 4:45 PM.

Details

Summary
  • m4/python.m4: Add python3.12 and 3.11 as valid interpreters, remove EOL 3.4, 3.5, 3.7 (keep 3.6 because some distributions still ship and maintain it)
  • lang/python/Makefile.am: switch from distutils to PEP517 build
  • lang/python/setup.py.in: switch from distutils to setuptools where possible, add classifiers up to 3.12, remove 3.4, 3.5, 3.7
  • lang/python/doc/src/gpgme-python-howto.org: replace distutils with setuptools
  • lang/python/examples/howto/advanced/cython/setup.py: replace distutils with setuptools
  • .gitignore: ignore dist-info
Test Plan

Tested on the openSUSE Tumbleweed rpm package for gpgme

Diff Detail

Repository
rM GPGME
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

bnavigator created this revision.
bnavigator retitled this revision from build: Support Python 3.10 version detection without distutils to build: Python without distutils.
bnavigator edited the summary of this revision. (Show Details)
bnavigator edited the test plan for this revision. (Show Details)
  • After two years, bump from serial 23 to 36
  • Remove more distutils usage

This version removes the usage of Python distutils. Python 3.10 deprecated distutils and started to print warnings that it will removed in Python 3.12. This has happened with the release of Python 3.12.

bnavigator edited the summary of this revision. (Show Details)

Also remove deprecated setup.py calls, use PEP517 style instead.

Requires setuptools, pip, wheel, build

What needs to be done that this gets merged?
Having to carry an increasingly large patch for NixOS is not ideal for us and it would be preferred if this could get merged.

I have updated m4/ax_python_devel.m4 to the current version and changed the call in configure.ac to set optional to true (which this patch didn't do causing the build to fail).

After my update of m4/ax_python_devel.m4 I retested the patch, but now the build fails for python 3.10:

Using pip 23.3.2 from /usr/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
ERROR: Directory '../../../../src/gpgme/lang/python' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

I think this patch needs to keep the old code for older versions of Python and only use the new code for 3.12+.

This revision now requires changes to proceed.Jan 16 2024, 10:47 AM

I have updated m4/ax_python_devel.m4 to the current version and changed the call in configure.ac to set optional to true (which this patch didn't do causing the build to fail).

The patch already updates to the current version + the GnuPG specific changes. Make a diff to http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_python_devel.m4;hb=df506ec920751087985f322e9b60d263c828661c and see for yourself.
What did you do additionally?

After my update of m4/ax_python_devel.m4 I retested the patch, but now the build fails for python 3.10:

I am sorry, I am not too familiar with this development platform. Where do I see your changes and what went wrong?

Using pip 23.3.2 from /usr/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
ERROR: Directory '../../../../src/gpgme/lang/python' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

But there *is* a setup.py in lang/python, The .in file is even part of the patch

I think this patch needs to keep the old code for older versions of Python and only use the new code for 3.12+.

With this patch applied, the openSUSE package for 1.23.2 builds fine with Python 3.9, 3.10, 3.11: https://build.opensuse.org/package/live_build_log/openSUSE:Factory/gpgme/standard/x86_64
... and with 3.12 in staging: https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:K/gpgme/standard/x86_64 (These logs are not permanently stored)

Wrong button? Didn't mean to abandon

This revision now requires changes to proceed.Jan 16 2024, 11:36 AM

The patch already updates to the current version + the GnuPG specific changes. Make a diff to http://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_python_devel.m4;hb=df506ec920751087985f322e9b60d263c828661c and see for yourself.
What did you do additionally?

The patch didn't make the necessary change to configure.ac which makes a missing Python a non-fatal warning instead of an error. See https://dev.gnupg.org/rMe3ae88267f03b435ec7d9e80ae1e2891e1f118e6. I had the update of ax_python_devel anyway lying around locally since some time.

After my update of m4/ax_python_devel.m4 I retested the patch, but now the build fails for python 3.10:

I am sorry, I am not too familiar with this development platform. Where do I see your changes and what went wrong?

https://dev.gnupg.org/source/gpgme/history/master/

Using pip 23.3.2 from /usr/lib/python3.10/site-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
ERROR: Directory '../../../../src/gpgme/lang/python' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

But there *is* a setup.py in lang/python, The .in file is even part of the patch

No, there isn't. There is a setup.py in the build folder, but not in the src folder. I suppose the problem doesn't show on build.opensuse.org because they do in-source builds.

I think this patch needs to keep the old code for older versions of Python and only use the new code for 3.12+.

With this patch applied, the openSUSE package for 1.23.2 builds fine with Python 3.9, 3.10, 3.11: https://build.opensuse.org/package/live_build_log/openSUSE:Factory/gpgme/standard/x86_64
... and with 3.12 in staging: https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:K/gpgme/standard/x86_64 (These logs are not permanently stored)

I'm using openSUSE Tumbleweed locally with Python 2.7, 3.8, 3.9, 3.10 and 3.11 being installed, but only devel packages for 3.8, 3.10 and 3.11. Therefore, AX_PYTHON_DEVEL failed for me for 2.7 and 3.9 with this patch because the "optional" option wasn't set to true. But that's fixed with the above mentioned commit.

So what now? You just updated the m4 files in master yourself and I should remove it here? Way to encourage contributions.

The patch didn't make the necessary change to configure.ac which makes a missing Python a non-fatal warning instead of an error.

I don't understand this. It was fatal before and by default still is. The new serial just adds the optional option.
Edit: Okay the new optional option cleans one of the dropped custom changes to the ax_python_devel.m4, got it.

bnavigator edited the summary of this revision. (Show Details)

Remove the changes for m4/ax_python_devel.m4 serial 36 commit to master in the meantime.

But there *is* a setup.py in lang/python, The .in file is even part of the patch

No, there isn't. There is a setup.py in the build folder, but not in the src folder. I suppose the problem doesn't show on build.opensuse.org because they do in-source builds.

Indeed. The problem was the ${srcdir} parameter in the pip call. Replacing it with . solves it.

However, I noticed that the last approach built each wheel multiple times. So I changed it to a split python -m build --wheel --outdir=wheels, pip install --find-links ./wheels gpg approach.

bnavigator edited the summary of this revision. (Show Details)

Fall back to distutils for old Pythons: setuptools for Python 2.7 does not have setuptools.command.build.build

Sorry for the delay. I just had another look at the patch. Unfortunately, it doesn't work with Python 2.7.

/usr/bin/python2.7: No module named build

I think it's missing https://pypi.org/project/build/, but this package doesn't support Python 2.7 since version 0.6. Maybe installing version 0.5.1 of build for Python 2.7 would help.

This brings us to another problem: The changes require the build package, but configure doesn't check for it. Due to this the build also failed for Python 3.10 with the same error.

I have merged the changes for using setuptools if distutils isn't available. This fixes the immediate problem that the bindings couldn't be built for Python 3.12. I did not merge most of the changes in lang/python/Makefile.am because of the reasons mentioned above. Maybe we can address the open changes in another patch. In any case, thanks a lot for your patch! And sorry that it took so long to get some of it merged.

Is supporting Python 2.7 such a high priority? That version of python is super duper EOL and this might be a good opportunity to drop support for it.

In D545#6468, @sandro wrote:

Is supporting Python 2.7 such a high priority? That version of python is super duper EOL and this might be a good opportunity to drop support for it.

That's a valid question. I will consider dropping support for Python 2.7 after the next release.