DOC/TST: add a how-to page and test packages for dynamic versioning

[rgommers]

Closes gh-159

Most of our most prominent users (NumPy, pandas, Matplotlib) figured this out, and other packages copy from there if they want dynamic versioning. It's quite a common question (see incoming links to gh-159), so it's past time to document this.

---

I do feel like the tradeoffs of versioneer/setuptools-scm are quite poor, and this stuff really isn't so hard that it needs separate dependencies (4 in the case of setuptools-scm?!?!) and thousands of lines of code. So there may be space for a follow-up where we either improve Meson's vcs_tag (or an alternative to it) so that works inside project(), or we provide something simple to vendor or a CLI script, a la what NumPy does (that just works in O(100) lines of code). But for now, this should help.

[rgommers]

Cirrus says: Cirrus CI shuts down on June 1st 2026. - guess we'll have to deal with that soon.

[eli-schwartz]

I do feel like the tradeoffs of versioneer/setuptools-scm are quite poor, and this stuff really isn't so hard that it needs separate dependencies (4 in the case of setuptools-scm?!?!)

setuptools-scm has the advantage of being the only one that is actually e.g. redistributor friendly. So ideally I would like to see information about versioneer be buried forever so that nobody ever uses it again. I don't even want its existence acknowledged. As a downstream builder and repackager of software seeing a project use versioneer is a red flag that I'm in for detrimental experiences and suffering, and that I should think carefully about whether I want to use the project at all, if that project has viable competition.

I'd also like to correct you about setuptools-scm's dependencies:

Recent work on setuptools-scm has involved splitting out its core logic into "vcs-versioning", which has only one dependency (assuming recent python) and contains the bits you need to parse versions from git without the integration into the setuptools build backend. The setuptools-scm entrypoint and __main__.py for those versions post split, are just a compatibility shim that imports and executes vcs_versioning._cli.main().

I won't deny that the packaging module isn't strictly necessary for our use cases, but it's still pretty lightweight compared to what you feared (and the packaging module is pretty much ubiquitous as a dependency for any package which is at all pyproject adjacent and doesn't actively have a policy forbidding the use of dependencies).

So there may be space for a follow-up where we either improve Meson's vcs_tag (or an alternative to it) so that works inside project(),

This is compatible with my long term desires. I want meson to directly support extracting versions from git, and Meson can't have dependencies so this would anyways require building it ourselves. I think that setuptools-scm has lots of cool knobs that... few people really need. :D So reimplementing it won't be a bad thing. Setuptools-scm also has to support a lot of legacy burden in its UX, sooo...

Modern setuptools-scm prefers using .git_archival.txt where possible (this required a fix to git itself that allows using git describe in gitattributes export-subst, I'm the one who pushed for that in git upstream; old projects will still have depended on other, legacy workflows in setuptools-scm). My feeling is that meson doesn't need to support anything other than this.

So what I'd probably want to see is

project(..., version: files('.git_archival.txt'))

with special detection of archival information, and then vcs_tag() could "know" when this was used and redetect the version.

People could still use custom scripts if they want to do something more complex than this, as they can today. It for many common use cases things can become a lot simpler.

[rgommers]

setuptools-scm has the advantage of being the only one that is actually e.g. redistributor friendly. So ideally I would like to see information about versioneer be buried forever

I agree setuptools-scm is the best of the bunch, so I'm fine with rewording as "setuptools-scm and other similar tools" or some such thing to not mention versioneer explicitly. We used to use versioneer in NumPy, and I agree it's a bit of a mess behavior and code-wise (although we never had distro packager complaints I believe). It does have the advantage of being designed for vendoring; which at least addresses one of the main downsides of setuptools-scm.

Recent work on setuptools-scm has involved splitting out its core logic into "vcs-versioning", which has only one dependency (assuming recent python) and contains the bits you need to parse versions from git without the integration into the setuptools build backend

I did look at that, and it seems much better in the long term, but as of now it has no docs of its own, and the setuptools-scm docs only say it's for building other packages on top of (https://setuptools-scm.readthedocs.io/en/latest/integrators/) plus they warn explicitly about using setuptools-scm functions like get_version directly. So as of now, I don't think we can be pointing at it.

So what I'd probably want to see is

project(..., version: files('.git_archival.txt'))

with special detection of archival information,

With that last statement, do you mean some special magic to make that work for direct installs from vcs (pip install . & co) so that files('.git_archival.txt') doesn't choke when that file doesn't exist? Might work, but a bit confusing. A new function like vcs_version() that uses .git_archival.txt under the hood might be cleaner?

[eli-schwartz]

The file always exists. The way it works is you commit the file with special tokens into git, and then git archive replaces the tokens.

[rgommers]

Thanks, I missed that. The point remains though: git archive is not run for direct installs. So shouldn't it be something like version: vcs_version(), with some awareness of both .git and .git-archival.txt (or perhaps with .git-archival.txt as an input argument)? That also keeps typing easier, with version: only accepting a string.

[eli-schwartz]

Version already accepts a files() object today -- it is currently expected to have a single line containing the version number.

It wouldn't be that hard to detect when a multi-line file has replacement tokens instead of the processed metadata and run git archive internally to process it.

I will admit that mostly what exists at the moment is shower thoughts in my head, so don't necessarily treat this as immutable requirements. ;)

[rgommers]

Okay, I did a quick check: git archive --format=tar -o scipy.tar.gz HEAD takes about 20 ms on an M5 Macbook, which is much faster than I expected (and similar to invoking something like git -c log.showSignature=false log -1). Generating an 82 MB tarball in a tmpdir isn't complete ideal, but also not a showstopper since it's only done once per build. For very large repos it might become a problem though.

[eli-schwartz]

To be clear, on a 1.6 gb git archive of the Linux kernel, it takes 0.182s -- that's because git archive HEAD COPYING does essentially no work. It takes 6 seconds if I want to write the entire 1.6gb out to a pipe for | wc -c.

20 seconds to write the whole thing to a file is still only 0.189s to do just COPYING. But a single text file wrapped in a tar envelope is trivially small enough to do all operations in a python bytes buffer anyway.

I don't like wasted work and I don't like slow-as-heck tools that do tons of work for "convenience". Trust me, this was on my mind. :D