[HN Gopher] Thoughts on the Python Packaging Ecosystem
___________________________________________________________________
Thoughts on the Python Packaging Ecosystem
Author : BerislavLopac
Score : 45 points
Date : 2023-01-21 17:16 UTC (5 hours ago)
(HTM) web link (pradyunsg.me)
(TXT) w3m dump (pradyunsg.me)
| DougMerritt wrote:
| At the top is a "tl;dr" which makes some points that I think are
| good to keep in mind for any set of tools: Pick
| from N different tools that do N different things is a good
| model. Pick from N ~equivalent choices is a really
| bad user experience. Picking a default doesn't make
| other approaches illegal.
| zdragnar wrote:
| As someone who has had to take over the projects of other
| people, I will say the third point needs a caveat:
| Picking a non-default tool needs a written explanation with
| sound reasoning
|
| When there is a default, new members to the community will
| gravitate towards it and, if there isn't a good reason, will
| avoid learning other options.
|
| In time, this ends up being anywhere from a minor annoyance to
| very frustrating, as not only are you losing time to learning,
| but potentially losing time to not having access to extras as
| the community around the default expands and your now special
| snowflake project goes without.
|
| For very specific tools and libraries, it might be okay if they
| are easily interchangeable (but is apparently bad UX?) But for
| things with larger responsibilities - a full blown ORM, package
| manager, etc- it can end up being a millstone.
| di wrote:
| Previous discussion:
| https://news.ycombinator.com/item?id=34464657
| woodruffw wrote:
| This is a great writeup by a central figure in Python packaging,
| and gets to the core of one of Python packaging's biggest
| strengths (and weaknesses): the PyPA is _primarily_ an "open
| tent," with mostly independent (but somewhat standards-driven)
| development within it.
|
| Pradyun's point about unnecessary competition rings especially
| true to me, and points to (IMO) a hard reality about where the
| ecosystem needs to go: at some point, there need to be some
| _prescriptions_ about the _one good tool_ to use for 99.9% of use
| cases, and with that will probably come some hurt feelings and
| disregarded technical opinions (including possibly mine!). But
| that 's what _needs_ to happen in order to produce a uniform
| tooling environment and UX.
| saghm wrote:
| > Pradyun's point about unnecessary competition rings
| especially true to me, and points to (IMO) a hard reality about
| where the ecosystem needs to go: at some point, there need to
| be some prescriptions about the one good tool to use for 99.9%
| of use cases, and with that will probably come some hurt
| feelings and disregarded technical opinions (including possibly
| mine!). But that's what needs to happen in order to produce a
| uniform tooling environment and UX.
|
| Not to mention that all efforts would be focused on improving a
| single package manager. Even if the one that was standardized
| wasn't the "best" at first (by whichever metric you prefer),
| giving everyone an incentive to improve that one will likely
| make it the best within a few years.
| atomicnumber3 wrote:
| Why does Ruby not have similar problems? (Or if it does, why does
| nobody seem to care?)
|
| Like why does Ruby have gem and bundler, each doing one thing,
| whereas python has fifty bazillion tools that all do nearly the
| same thing if you squint but all have their own weird problems?
|
| I've personally just ended up using poetry and that has mostly
| stopped me from having to care overmuch about the tooling.
|
| I feel like I'm spoiled coming from Java land, where aside from
| too much XML, maven just works and if you need to do weird shit
| (like if you're android, for instance) then you use gradle, which
| still doesn't fuck up the maven repository format and we can all
| just live happily. (And nobody uses Ant anymore).
| woodruffw wrote:
| In some ways, it's actually pretty weird that Ruby hasn't had
| similar problems (I say wearing both my Ruby fanboy and Python
| packaging contributor hats).
|
| In others, it's less weird: Ruby's upswing happened with Rails
| and a handful of other "killer" frameworks, which helped to
| solidify developer workflows (and expectations) around
| packaging. Python, by contrast, has had multiple generations of
| "killer" usecases, each with their own baggage (and many
| predating any real packaging standards).
|
| The history of Python packaging is also much older, and much
| more devolved than anybody in 2023 would consider reasonable
| for a packaging ecosystem: the earliest generation of PyPI, for
| example, was _just_ an index that pointed to other webhosts for
| downloads, rather than a full package host. This helped ossify
| manual workflows that developers at the time were content with,
| and some of that cost is still being paid forwards.
| [deleted]
| Kwpolska wrote:
| Python packaging has too many people inventing their own tools
| and their own little fiefdoms instead of picking one good tool
| and pushing it forward. There's also the PyPA, which is an
| organization (or a group of semi-related volunteers, depending
| which interpretation suits them best) that maintains many of
| the tools (although none of the most feature-complete, namely
| poetry and PDM) and that produces standards that promote
| tooling proliferation.
| wiredfool wrote:
| I've just looked through the docs for poetry and I don't see
| anything there about compiled extensions.
| ram_rar wrote:
| I get a lot of flak for saying this, but I do hope python
| committers and psf members take a hard look at perl and its
| demise. Perl had a very similar problem of having N equivalent
| choices for doing the same thing and python for better or worse
| is heading in the same direction. In the last few yrs of my
| professional career many projects I worked, were on are migrating
| python -> golang cuz frankly many people were fed up of
| python2->3 migration. Not that golang doesnt have it fair share
| of issues, but the the packaging and deploying aspects of a
| single binary makes CI/CD and operations a breeze. I hope python
| makes this is first class citizen in its ecosystem.
| philsnow wrote:
| > A class of users expect a packaging tool that provides a
| cohesive experience (like npm (NodeJS), [...], etc) - a single
| tool that provides a build system, dependency manager,
| publishing, running project-specific tasks/scripts, etc. I've
| referred to this as "workflow tool" in this post.
|
| I don't know enough about node, but aren't there at least two or
| three package managers (npm, yarn, maybe pnpm)? Then there are
| half a dozen different things for the "build" / transpile /
| compile stage of frontend work, and
|
| > Pick from N ~equivalent choices is a really bad user experience
|
| this has caused me to bounce off of getting into frontend work
| several times. It's so aesthetically displeasing that my brain
| doesn't want to learn it.
| JeremyBanks wrote:
| [dead]
| jessaustin wrote:
| _...aren 't there at least two or three package managers (npm,
| yarn, maybe pnpm)?_
|
| I'm not sure about yarn, but pnpm has exactly the same API as
| npm has. It simply has a different disk organization and
| caching strategy. If npm maintainers so chose, pnpm's behavior
| would be an option within npm. Since they haven't chosen that,
| it seems completely reasonable to "compete" in the way that
| pnpm does.
| chaxor wrote:
| Packaging and Nvidia are pretty much the main reasons python has
| degraded in status for the past 7 years. It started with the
| added and unnecessary confusion from conda (and silly things like
| pythonxy), but has really escalated through extra stupidities
| like poetry. Much of the features that may be enticing for these
| extra packages should have pushed the developers to improve the
| standard tools in python (pip). The other problem is due to
| python's lead in ML and the need to work with GPUs. That started
| with tensorflow, which had immense (still does) issues in dealing
| with versioning, mostly due to Nvidia (matching cuda versions and
| such). So effectively, Nvidia and package management killed a
| good language.
| kelsolaar wrote:
| Poetry certainly made our life much easier and was a
| productivity improvement. It the the tool that brought us back
| from Conda because it was able to solve environments than
| nothing in Pip land was able to do.
| totalhack wrote:
| Poetry seems pretty slow due to some of the extra dependency
| logic it builds on top of pip. Whether this trade off is
| worth it probably depends on the scope of the project. For
| most of my projects I stick with pip and almost never run
| into issues.
| jvolkman wrote:
| Was that before or after pip's new resolver was released at
| the end of 2020?
| [deleted]
___________________________________________________________________
(page generated 2023-01-21 23:00 UTC)