[HN Gopher] Underappreciated challenges with Python packaging
___________________________________________________________________
Underappreciated challenges with Python packaging
Author : groodt
Score : 42 points
Date : 2023-01-03 19:45 UTC (3 hours ago)
(HTM) web link (pypackaging-native.github.io)
(TXT) w3m dump (pypackaging-native.github.io)
| irskep wrote:
| This is an incredible example of organizing information well and
| making a case to a wide audience. It's difficult enough to shave
| all the yaks necessary to get a high-level view of all issues
| related to a problem, and to express all those problems in good
| writing is an additional tough challenge. These folks have done
| an amazing job at both.
|
| Shoutout to Material for MkDocs enabling the swanky theme and
| Markdown extensions. https://squidfunk.github.io/mkdocs-material/
| 0x008 wrote:
| Python packaging's greatest challenge is 10 competing tools and
| standards.
| wiredfool wrote:
| That's ... completely missing the point of the article. There's
| nothing about competing standards that solves the problem of
| the C-ABI and how to package non-python library dependencies.
| [deleted]
| groodt wrote:
| Python packaging gets a lot of criticism. It's a meme. The
| thing is, it's actually improved dramatically over the years
| and continues to improve.
|
| The problems it solves are very complex if one looks a little
| below the surface. It is solving different problems to the
| ecosystems that it's often compared to: golang, rust, java, js.
| short_sells_poo wrote:
| I wonder to what degree these complex problems are self
| inflicted due to more than a decade of flipflopping between a
| myriad packaging and environment management solution. In such
| an environment, you are bound to have so many different
| approaches between important projects that trying to bring
| them all under one umbrella becomes next to impossible. What
| is a python package is relatively well defined, but how you
| build one is completely up to you.
|
| Edit: I've just read another comment which I think pointed
| out the most pertinent fact - that python has often served as
| mere glue code for stuff written in lower level languages.
| This then results in an explosion of complexity as a python
| package not only has to package the python part, but has to
| be able to build and package the lower level stuff.
| wiredfool wrote:
| I'd have to look at numbers to be sure, but I think that
| the number of popular packages that include compiled
| libraries is dramatically higher than it was 10 years ago,
| and that's about when wheels were taking over. The
| Data/Science stack has really exploded in that time, and
| it's very heavily skewed towards packaging c/fortran/rust
| libraries. Previously there was PIL, some database drivers,
| and numpy was just getting started. I think more of the big
| packages were pure python. The earlier
| egg/easy_install/msi/exe installers have all faded away,
| and now it's really just wheels and sdists.
| atoav wrote:
| This is true, still: Everyday things that should be
| straightforward, or _pythonic_ if you will, are way to
| convoluted and complicated.
|
| As a python dev with experience since 2.6 I agree it has
| gotten better, but it is also rotten at it's core. The
| problems python has to solve there are _hard_ , but this is
| why it should be priority number one to solve them elegantly
| and in a fashion so that 99% of python users never have to
| worry about them.
|
| Right now packaging and dependency managment are my number
| one concern when I think about rexommending the language to
| beginners. The things I needed to learn just to figure out
| what is a good way of developing something with dependencies
| and deploying it on another machine is just way too much.
| When I was going through this there was no official "this is
| how it is done" guide.
|
| Now there is poetry for example. Poetry is good. But there
| are still hard edges when you need to deploy to a system
| without poetry on it for example.
| woodruffw wrote:
| At this point, there _are_ 10 competing tools but no longer so
| many competing standards: _the_ standards for Python packaging
| from 2015 onwards are PEP 517 (build system standardization),
| PEP 518 (using pyproject.toml to configure the build system),
| and PEP 621 (storing project metadata, previously standardized,
| in pyproject.toml). These standards build on top of each other,
| meaning that they don 't offer conflicting advice.
|
| The TL;DR for Python packaging in 2022 is that, unless you're
| building CPython extensions, you can do _everything_ through
| pyproject.toml with PyPA-maintained tooling.
| wiredfool wrote:
| Pillow has most of the issues that are listed in the article.
| (oddly enough for a graphics library, the GPU part is the only
| part that I don't think we've stumbled over at one point or
| another.)
|
| From a quality of life issue -- having the sdist install behind
| an opt-in flag by default for our package would be great. Unless
| you're a developer with a lot of -dev packages for imaging
| libraries already on your system, you're not going to be able to
| build from source. And even if the error that pops up is along
| the lines of The headers or library files could
| not be found for {str(err)}, a required dependency when
| compiling Pillow from source. Please see the install
| instructions at:
| https://pillow.readthedocs.io/en/latest/installation.html
|
| We still get people posting issues about pillow failing to
| install.
|
| Build farms would be nice. We've burned tons of time on it
| between travis and GH actions and @cgohlke single handedly making
| all of the windows builds for the entire scientific python
| community.
|
| Ultimately, something like the debian packaging system is
| probably the best open source model for this. (though the
| splitting of the python standard library so that virtual envs
| aren't in the base install is a pita). Unstable gets a reasonably
| current set of packages, and crucially all of the underlying
| library dependencies are compiled together. It's also not _that_
| hard to rebuild individual packages from source, in an automated
| fashion. (This may be what Conda is doing, but I've never looked
| in detail at their system)
| david2ndaccount wrote:
| Could you release the sdist as a separate package and only
| upload binary wheels for a normal install?
| miohtama wrote:
| Python joke:
|
| Sdist is only one letter away from sadist.
| CJefferson wrote:
| I've been planning on packaging a python package recently, and
| the internet is annoyingly full of guides which are, I think, out
| of date. They at least suggest quite different things.
|
| I just have a single python file, meant to be treated as an
| executable (no package at present). There are a whole bunch of
| tests, but that's obviously separate. Any suggestions on modern
| best practices welcome!
| woodruffw wrote:
| If it's pure Python, the only packaging file you _need_ is
| `pyproject.toml`. You can fill that file with packaging
| metadata per PEP 518 and PEP 621, including using modern build
| tooling like flit[1] for the build backend and build[2] for the
| frontend.
|
| With that, you entire package build (for all distribution
| types) should be reducible to `python -m build`. Here's an
| example of a full project doing everything with just
| `pyproject.toml`[3] (FD: my project).
|
| [1]: https://github.com/pypa/flit
|
| [2]: https://github.com/pypa/build
|
| [3]: https://github.com/pypa/pip-audit
| quietbritishjim wrote:
| > including using modern build tooling like flit[1] for the
| build backend and build[2] for the frontend.
|
| Or you can use setuptools, which is the package that enables
| old setup.py builds, as the backend with pyproject.toml. This
| has the advantage of being mature, unlikely to be abandoned,
| and possibly some familiarity if you've used it before. Even
| then, you can use build as the front end build tool.
| woodruffw wrote:
| Yes, setuptools is also perfectly fine. It had some rough
| edges around PEP 621 support for a while, but those were
| mostly smoothed out in 2021.
|
| (I'll note that maturity is not strong evidence here:
| distutils is _very_ mature, but is in the process of being
| deprecated and removed from Python entirely. I don 't think
| that's likely to happen to setuptools, but the fact that
| behavioral PEPs now exist for all of these tools means that
| the decline/abandonment of any poses much less of an
| ecosystem risk.)
| ensignavenger wrote:
| If you are wanting to release it to pypi as a python package, I
| would personally use Poetry. But your case- a single pure
| Python package, is a simple case that won't have many problems
| like are brought up in the article, whatever tool you use.
|
| If you want a stand alone executable, I haven't found a good,
| single, cross platform tool for that yet... seems like there is
| a separate tool for each platform.
| quietbritishjim wrote:
| Nuitka works on Windows, Linux and Mac
|
| https://nuitka.net/
| mixmastamyk wrote:
| Keep it simple and fashion-proof. Been using setup.py for a
| one-script package for one or two? decades:
| from setuptools import setup setup(
| name = 'foobar', scripts = ['foo'],
| # install a script from current fldr # ...
| )
|
| A few years ago I had to start using twine to register and
| upload it to pypi.
___________________________________________________________________
(page generated 2023-01-03 23:00 UTC)