Build and upload portable macOS and Linux binaries to python.org for CI flows

[solvingj]

Feature or enhancement

The Python team and automation is currently providing "official portable precompiled binaries" for Windows for the expressed intent and scope of "CI use cases" via the nupkg package referenced below. This request is for the Python team to provide equivalent "official portable precompiled binary" for macOS and Linux, for the same expressed intent and scope of "CI use cases".

Ref. https://docs.python.org/3/using/windows.html#the-nuget-org-packages
Related ref: #103011

Pitch

Historically, it has been impractical and prohibitive to build "official portable binaries" for both macOS and Linux for numerous factors. Thus, these use cases have been foregone, leaving them to be handled by third-party teams addressing smaller scopes, enabling more assumptions to be made, and thus resolving most of the problem factors (apt/yum/brew/pyenv/snaps/conda/etc). There is now a plethora of such third-party projects providing such unofficial distros, many downloaded millions of times per month.

However, each of the most challenging factors has been mitigated or rendered completely irrelevant over the last 5 years. It is now possible to build "official portable binaries" for both macOS and Linux. In the face of an overwhelming number of niche third-party options, an official option would be of tremendous value to the world of CI engineers and users.

Making it Easy:

Of all the unofficial projects which build such "portable pythons", the following would be the most useful to the Python team in considering the scope and potentially attempting this undertaking:

https://github.com/codrsquad/portable-python

This project is the most pure and general-purpose in it's design, focused simply on providing a higher-level interface to the otherwise low-level CPython build system. It codifies in a comprehensible and transferrable way, the nuanced flags, patches, and steps, necessary to build portable-pythons on both macOS and Linux. It is very well polished, and would provide a good reference. All the other options are largely coupled to the related package managers or distribution systems, and so it would be much more difficult to distill out the actual requirements.

A Unified Cross Platform Solution

One final factor: virtually all the other third-party solutions are proprietary to a single platform. If this proposal were accepted and completed, when taken together with the .nupkg solution already provided on Windows, it would give CI engineers a single, official, cross-platform solution for "CI engineering teams" the world over. The practicality and value of that seems obvious and immeasurable.

[arhadthedev]

@Yhg1s (as a release manager)

cc @ronaldoussoren, @ned-deily (as macOS experts)

[ronaldoussoren]

This requires someone to do the work w.r.t. specifying what such a portable binary should and shouldn't be. That, to be honest, isn't me because the status quo is good enough for me (GitHub runners where possible, and for the most part manually managed VMs where I run tests in virtual environments).

For macOS this should also explain why the current installers aren't good enough for this. Those installers can be used to install Python from the command-line, although this does require a user admin privileges. The rights issue might not be a problem in isolated CI environments though (e.g. a fresh VM per job).

Note that adding more distribution artefacts has a cost: the release management team has to build and test those artefacts for every release, and that team consists of volunteers like the entire core team.

[bjia56]

Chiming in with a non-CI use case - I'm a contributor to Scrypted (home video integration), and our platform supports plugins written in Python. To simplify user installation and ensure compatibility with available libraries (e.g. Coral AI libraries only provide wheels for Python 3.9) across different OSes and architectures, we're downloading a prebuilt Python under the user's home directory. Given that the installation path is unpredictable, we've had to ship custom Python builds to support arbitrary install prefixes.

Our approach has been to use the CMake-based Python build system (original here, but forked here) to statically link many of the dependent libraries into libpythonX.Y.so, then change the rpath so the main interpreter executable looks for libpythonX.Y.so in a relative directory. We've also had to patch the sysconfig module to dynamically populate sysconfig.get_config_vars() with the install prefix, which allows binary extension compilation to find the headers shipped with Python.