[HN Gopher] How to improve Python packaging
___________________________________________________________________
How to improve Python packaging
Author : Kwpolska
Score : 146 points
Date : 2023-01-15 16:27 UTC (6 hours ago)
(HTM) web link (chriswarrick.com)
(TXT) w3m dump (chriswarrick.com)
| Lyngbakr wrote:
| It seems most people agree that Python's packaging isn't great,
| but, conversely, is there a language where most people agree that
| the approach to packaging is awesome? I mean, what's the gold
| standard to aspire to?
| nixpulvis wrote:
| Ruby gems and Cargo are pretty awesome. Also, I find I like
| Arch linux's Pacman quite a bit, though that's a slightly
| different use case where versioning isn't resolved.
| oconnor663 wrote:
| Cargo is a gold standard imho, but there are definitely some
| simplifying decisions it makes that might not fit Python:
|
| - Dependencies and executables can run arbitrary code during
| build. For example, Cargo knows almost nothing about how to
| build C code, and the common workflow is to pull in the
| popular `cc` library for this and call it in your build.rs.
|
| - There's mostly no such thing as "installing a library", and
| each project builds its dependencies from source. This is
| baked in at the Cargo level (the install command just doesn't
| work for library crates) and also at the language level (no
| stable ABI besides "extern C").
|
| - Related to that, the final product of a Cargo workflow is a
| mostly-statically-linked binary, so there isn't really an
| equivalent to virtualenv.
| Kwpolska wrote:
| > - Dependencies and executables can run arbitrary code
| during build. For example, Cargo knows almost nothing about
| how to build C code, and the common workflow is to pull in
| the popular `cc` library for this and call it in your
| build.rs.
|
| Python packaging involves arbitrary code execution, even
| today. While it's less obvious than it was with `setup.py`,
| if you're installing from source, the package can specify
| any build backend it wants, and that build backend can do
| anything. This could be mitigated by having an allowlist of
| build backends, but who would be responsible for vetting
| the tools, and how would it be guaranteed that an allowed
| tool is not taken over by someone hostile, and that an
| allowed tool does not have a system to run arbitrary
| package-provided code?
| earthling8118 wrote:
| I don't think the final build output being statically
| linked has any connection to a virtualenv. I've only ever
| used Python's virtualenvs to avoid having to install
| dependencies globally. When using Cargo you don't need to
| worry about that because their data is saved per-project
| already and Cargo can sort itself out when you invole a
| command
| orhmeh09 wrote:
| R with CRAN. renv for reproducibility
| gen220 wrote:
| I think Go (if you started using it in the `go mod` era) and
| Rust have the best stories around package management.
|
| The interface around `go mod` is kind of confusing, but I have
| actual trust in the dependency graph it generates for me. Cargo
| has, afaict, nailed both the interface and trust in what's
| going on under-the-hood.
|
| In the Python world, Poetry isn't too bad. It's terribly slow
| in comparison to `go mod` or Cargo, but I generally trust and
| understand what's happening on the inside, and it's interface
| is fairly legible to newcomers.
| TheChaplain wrote:
| Don't know if it counts but Java has done pretty well in my
| opinion. A single JAR-file can combine everything you need to
| run an application or be distributed as a library, and as long
| as the java command is installed it just works.
|
| Likewise WAR's make it dead simple to add/remove/update
| applications during runtime.
|
| Of course there are always special cases, but packaging and
| distribution in the Java world have always been painless to me.
| neillyons wrote:
| I'm a big fan of `mix` for Elixir.
| https://hexdocs.pm/mix/Mix.html
| di4na wrote:
| This. Mix is basically a mix (pun unintended) of Leiningen
| and Bundler. It works. Quite well.
|
| Part of this is due to erlang solving a lot of the problems
| upfront (releases) and then the Hex team handling a lot of
| the rest.
|
| But in general, the "Yehuda Katz" lineage (Bundler, Cargo,
| Yarn v1, mix as Jose used to be mentored by Yehuda) have
| pretty good tools worth copying. At least as a base.
| Kwpolska wrote:
| I don't think there's much complaining in the C#/.NET
| ecosystem. Sure, there are alternate tools (FAKE for F#), but I
| believe most people are happy enough with
| dotnet.exe/msbuild/nuget.
| DangitBobby wrote:
| I think this is what success for unification looks like:
|
| Let's say there are two tools, one called pyup and one called
| pygo. pyup: responsible for which versions of
| python are available on your machine, as well as
| which one is the default "system" python. It also keeps itself
| and pygo updated. pygo: responsible for (among
| other things) - allowing a user to specify a
| python version for a project - allowing a user to
| specify a python version for a script - helping
| the user install a compatible python version for the project
| with pyup - helping the user install a compatible
| python version for a script with pyup - selecting
| a compatible python version/env for project -
| selection a compatible python version/env for a script
| - allowing a user to specify project dependencies
| - allowing a user to specify script dependencies within a
| script - determining which dependencies are
| required for a project - determining which
| dependencies are required for a script -
| installing project dependencies - installing
| script dependencies - installing projects or
| scripts as executables
|
| I MUST NOT need a base python install to make any of this work.
| Tools are constantly mucking with system python and which one
| is on my path so I can't trust system python, period. pyup and
| pygo should be their own binaries which invoke python.
| example.py: #!/usr/bin/env pygo #
| version: ~3.11 # requirements: # requests~=2
| import requests requests.get("https://example.com")
|
| When I run ./example.py for the first time: -
| pygo helps me install python 3.11 - pygo installs
| requests in a location that I don't have to worry about
|
| When I run ./example.py for the second time, the script runs
| without error.
|
| If I still need to use something like virtualenv, poetry, or
| conda on top of this, the unification project has failed.
| korijn wrote:
| I get the impression the author has not been exposed to enough
| pain doing package maintenance on a webpack codebase.
| mkoubaa wrote:
| I use sites as composable mini venvs, but there's no tooling for
| that
| Spivak wrote:
| Not being able to exclude system packages from resolution when
| using __pypackages__ is a pretty glaring omission that will force
| people to continue using venvs until it's fixed.
| [deleted]
| tiziano88 wrote:
| Interesting article though it's missing at least a mention of
| Nix, which IMO solves a lot of the pain points described.
| charliermarsh wrote:
| It's interesting to note that PEP 582 (i.e. __pypackages__) mode
| used to be the PDM default -- and was sort of its flagship
| feature -- before being made opt-in with 2.0 release, the
| explanation being that the PEP had stalled, and that editor and
| IDE support for virtualenvs is much better
| (https://www.pythonbynight.com/blog/using-pdm-for-your-
| next-p...).
|
| If you read through the discussion on the Python forum, one of
| the counterarguments to the PEP is that "If existing tools can
| already support this, why do we need a PEP for it?" But
| presumedly a PEP would help push the ecosystem to accommodate
| __pypackages__, and to solve the aforementioned problems (like
| broader editor and IDE support).
|
| For what it's worth (as someone that builds Python tooling full-
| time): I'm generally a fan of moving to a node_modules-like
| model.
| charliermarsh wrote:
| Oh, also, I think I've read the entire Discussions thread on
| this PEP, and from what I can tell, the PEP is missing
| champions who are eager to push it forward.
|
| Part of the problem, I'm guessing, is that the PEP kind of sits
| in a weird space, because it's not up to the PyPA to "approve"
| it, or whatever -- PEPs are approved by the Python Steering
| Council.
|
| Per Brett Cannon (Steering Council Member):
|
| > Either consensus has to be reached (which it hasn't) and/or
| someone needs to send this to the SC to make a decision
| (although it is a weird PEP in that it also impacts packaging,
| so it isn't even clear who would get final say).
|
| https://discuss.python.org/t/pep-582-python-local-packages-d...
| Kwpolska wrote:
| That tooling argument is quite weak. PDM is the only tool I was
| able to find that has support for __pypackages__, the paths it
| uses seem to be slightly different to the PEP wording, and it
| uses a $PYTHONPATH/shell hack (`pdm --pep582` adds a folder
| with `sitecustomize.py` that does the magic). How do you change
| the $PYTHONPATH used by an IDE (and its magic features)?
| charliermarsh wrote:
| I agree!
| smitty1e wrote:
| Time for GvR to saddle up the BDFL pony for a (last?) ride?
| dataflow wrote:
| Regarding system package upgrades breaking virtual environments:
| would hard links address this? If the system package manager
| removes a file, your venv still has a hard link to it so it
| doesn't see a difference.
| Denvercoder9 wrote:
| It would also prevent the Python in your venv from receiving
| any updates when the system-wide Python is installed. At that
| point it's probably easier to just use your own Python
| installation, e.g. with pyenv.
| dataflow wrote:
| Sure but installing a new copy would defeat the space
| savings. The goal of this was to get the desired effect
| without wasting as much space. I wasn't trying to optimize
| purely for ease of use.
|
| Put another way, my suggestion was to migrate from symbolic
| links to hard links.
| arunkant wrote:
| One more thing we need is app distribution for desktop apps
| written in python. I need to be able to package python app as
| single binary. There are tools like pyoxidizer
| https://pyoxidizer.readthedocs.io/en/stable/ . hopefully it
| become standard in python community
| ctoth wrote:
| You probably already know about this, but Nuitka[0] is pretty
| great for building distributable Python apps.
|
| [0]: https://nuitka.net
| egberts1 wrote:
| What is clearly still missing on python.org website is the
| obvious edit-code-run-debug step-by-step HOWTO for a typical
| tight but circular Python development iterations, complete with
| standardized packaging approach.
| rektide wrote:
| Cant resist digging at Node.js even when writing up how
| infinitely better Node.js is at dealing with packages than
| python, haha:
|
| > _Let's try removing is-odd to demonstrate how badly designed
| this package is:_
|
| You literally just deleted _is-even_ 's dependency on _is-odd_
| then have the audacity to be shocked that it broke?
|
| There's a lot of hatred for the _small package_ philosophy of
| node.js, but it 's also a huge win, stands a good chance of being
| why javascript has been such a winner, gotten so far: very
| explicit small things that say what they do on the tin. Rather
| than repeat yourself by making a copy pasted _is-even_ and then
| maintaining both, it makes perfect sense to compose
| functionality, to build off what we have. And it is easier to
| understand the scope of what a package is & what it can do when
| it is explicitly limited in scope.
|
| This is another place where there is lots of loud vociferous
| animosity against what is, but it's there for good reason. And
| with rare rare exception- left-pad rage-quit deliberate breakage,
| for example, it serves well. With the exception that yes,
| inventorying your stuff is hard.
| Kwpolska wrote:
| There's a place for small packages, but is-even/is-odd is a bit
| too small to be a reasonable package. It is far easier to just
| write `x % 2 === 0` inline, which is an obvious idiom, instead
| of installing and importing a separate package for this. The
| use of is-odd by is-even can be confusing for users. For
| example, you may call isEven(0.5) and get the following error:
|
| RangeError: is-odd expects an integer. at isOdd
| (/tmp/mynodeproject/node_modules/is-odd/index.js:17:11) at
| isEven (/tmp/mynodeproject/node_modules/is-even/index.js:13:11)
| at Object.<anonymous> (/tmp/mynodeproject/index.js:3:13)
|
| (But the main point of the demonstration was to showcase
| dependency resolution and where it looks for packages.)
| rektide wrote:
| isEven is in thay stack trace - should not confuse anyone
| with even a basic introductory level fluency at coding.
|
| Is it too small? What if latter the language evolves BigInt?
| Donwe suffer a patchwork of libraries which have & havent
| upgraded, sussing around each time to find out?
|
| I think the key thing to recognize is that this is all
| opinion. Many people dont like the availability of many
| opions, the ease at which dependencies have grown. And that's
| fine, there's some real pain here to having ballooning
| package trees. There's a pevel of conceit though that I feel
| that often arises, where we mock & shiv packages like _is-
| even_ . But to me, it 's not absolute, it's a matter of taste
| & preference. It looks weird to outsiders, but it has been
| enormously powerful & helpful, has been such a key successful
| element of JS that npm arose & made package management easy &
| package publishing easy & that we begat a new behavior of
| capturing all the little helpful things we do & making them
| available.
|
| Maybe there are good reasons for inlining simple things, but
| it's not clear to me what the gains really are, or what's
| wrong with is-even.
| zdragnar wrote:
| It would be nice if major node packages invested time in re-
| inventing the wheel, just a little bit.
|
| Back when I used it, I really appreciated how little bloat Hapi
| added to node modules, compared with webpack for example.
|
| Obviously there's a world of difference in what problems the
| two solve, but still...
| kdazzle wrote:
| I'm biased, but I do like Python's package management much
| better than Node. Even just regular old virtualenvs. Can't tell
| you how many times deleting the node packages dir and
| reinstalling fixes a weird issue, but that's happened very
| rarely for me on the python side.
|
| Also, having to comb through a bunch of node packages of
| dubious quality to find some sort of standard approach happens
| way too often. Like take python requests vs axios.
| phailhaus wrote:
| I'd say that Node's package management has run laps around
| Python's, to the point where it's pretty embarrassing. Since
| it's relatively new, it was able to hit the ground running
| with best practices:
|
| 1. Declarative package manifests. Python's ecosystem is still
| a mess of various approaches, and the fact that you have to
| _run the setup.py script_ to determine dependencies is a
| nightmare. Because of this, running dependency resolution and
| installing is an order of magnitude faster in Node than in
| Python.
|
| 2. Drop-dead simple isolated environments: everything's in
| `node_modules`. You literally can't make a mistake by blindly
| running `npm install` in the project dir. With Python it's on
| you to manage your virtualenv, which boils down to PATH
| manipulation and symlinks that you'll have to remember to
| undo when switching around. There's no default for what to
| call your venv either, so it's on you to settle on a standard
| and gitignore it. Every time you run `pip install`, you have
| to hesitate and make sure you're in the right environment, or
| else risk borking the wrong env (or your global!)
|
| 3. Out-of-the-box comprehensive lockfile support. Debugging
| Python dependency issues is a nightmare. There's literally no
| way to figure out why a dependency was installed without
| using 3rd party tools (like pipdeptree). In Node, simply
| running `npm install` will automatically generate a proper
| lockfile.
|
| I work full stack, and the difference is like night and day.
| I barely think about dependency management in Node.
| gen220 wrote:
| Respectfully, this reads like you're using outdated Python
| tools.
|
| Give Poetry [1] a shot, it has all the things you've listed
| here. Just as Node.js has come a long way in the last 5
| years, Python has, too. Albeit, in a fashion that was much
| less centralized, arguably to Python's detriment.
|
| [1]: https://python-poetry.org/history/#100---2019-12-12,
| released 1.0 in 2019/12.
| phailhaus wrote:
| Oh yeah, I've used Poetry. I'm talking about Python out
| of the box. The dependency management ecosystem is super
| fractured and relies on 3rd party tools developing their
| own standards, and even then, they can't overcome
| fundamental limitations. For example, Poetry is great but
| locking/installation still takes way longer than Node
| because there are no declarative manifests.
| revskill wrote:
| I've tried with venv before, and the console (Iterm) just get
| broken.
| charliermarsh wrote:
| > There are two tools for dealing with packages in the Node
| world, namely npm and Yarn.
|
| Not to mention pnpm. Or Bun. Or even Deno. All of which deal with
| packages, and make different choices.
|
| This isn't to be a pedant, but I think it's a reasonable example
| of why it's less important that there's a single way to do a
| thing, or even a single tool vs. multiple alternatives; instead,
| what I care about is that I don't have to string together tens of
| tools to do that thing, and that there's some degree of
| compatibility and standardization in the API between those
| different alternatives.
| matsemann wrote:
| Because of the insanity of python I run everything in Docker
| (through compose). No more issues with it not working on some
| dev's computer because of a missing wheel, or that they need to
| have X and Y c++ toolchain packages installed locally etc. No
| more trying to fix a broken setup after upgrading python or
| poetry versions, just "docker compose build" and you're up and
| running. No spending days getting a freshly cloned project up and
| running.
|
| Then I point Pycharm to use that as my remote interpreter through
| docker compose.
|
| Not watertight, but a hundred times better dev experience.
| groestl wrote:
| I secretly believe Docker only exists because of how f'up
| Python's distribution story is.
| jessekv wrote:
| I also have often wondered how much Docker owes its success
| to the shortcomings of python.
| DangitBobby wrote:
| I've tried this a couple of times but just generally not liked
| the experience and gone back. When you get a shell into the
| container so you can use your project's python env, the tools
| you have installed on your machine (bashrc, rg, aliases) aren't
| available. I ultimately prefer the hassle of managing
| environments natively over forfeiting the use of my tools or
| juggling shells.
| goodpoint wrote:
| No way. Docker is not designed for security.
| wheelerof4te wrote:
| When the solution to basic problem like this is "use Docker",
| you realise how deeply flawed modern software development (in
| Python, at least) is.
| jessekv wrote:
| Not sure I agree with "deeply flawed" verdict.
|
| The flaws of python's packaging system (as well as the GIL
| and other hangups) emerge from the tradeoffs of inventing a
| language optimised for reusing and connecting as many
| disparate system binaries as possible.
|
| Its not surprising that such a server scripting language is
| tightly coupled with the server environment that it runs in.
| Docker is just one way to ensure a reproducible server
| environment across machines.
|
| What do you think it could have done differently?
| LtWorf wrote:
| .deb packages...
| groestl wrote:
| Do you happen to have a link to a python application that
| ships it's own dependencies and has state-of-the art debian
| packaging?
| sam345 wrote:
| The main reason I have stayed away from Python. packaging is a
| mess!
| c7DJTLrn wrote:
| And one of the reasons I moved away from it! My blog post on
| the subject was posted here and it was very unpopular despite
| having similar points and also making comparisons with Node.
|
| There's nothing special about Python that makes it worth
| enduring this pain. We have other languages with a better
| developer UX.
| noisy_boy wrote:
| conda is a bleeding pain in the ass; installed a ton of stuff
| without even asking if I needed them. Then when I wanted to
| install a different version of Python, kept running conflict
| resolution for hours before I got tired and killed it. Might as
| well just do the setup oldschool via virtualenv using
| requirements.txt.
|
| Dealing with all this is why I chose to use golang for writing a
| CLI utility (even though I'm not a big fan of its error handling
| boilerplate); static typing + produces a single binary that can
| be run without needing any of the environment setup. I am aware
| of various Python tools that can produce a binary too but I think
| they have their own edge cases and it is just nicer when that can
| be done out of the box without any dependencies.
|
| > You can also find deficiencies in the tools for the other
| languages mentioned. Some people think Maven is terrible because
| it uses XML and Gradle is the way to go, and others think
| Gradle's use of a Groovy-based DSL makes things much harder than
| they need to be and prefer Maven instead.
|
| Yeah but I have never had the installer crap out like pip or its
| ilk do when installing a package with maven. At worst, it can't
| find the package if the repositories are not configured properly.
| optimalsolver wrote:
| Python packaging is a solved problem:
|
| https://python-poetry.org/
| revskill wrote:
| Yes, there should be (one way) to do packaging.
| Kwpolska wrote:
| Poetry is one of the better options, but its nonstandard
| pyproject.toml is not ideal. PDM is basically Poetry, with the
| metadata standard in place, and with support for
| __pypackages__.
| c7DJTLrn wrote:
| It's not really solved if you have no choice but to work with
| codebases that don't use Poetry (quite common). There's 14
| tools for people to choose from that aren't going away any time
| soon.
| throwawaaarrgh wrote:
| as a Poetry user myself, ummmmmmm. no. lol
| [deleted]
| josteink wrote:
| Every time someone asks me what I think is wrong with Python
| packaging, I'll show them this link.
|
| Saved. Thanks for sharing!
| woodruffw wrote:
| It's not very well documented, but the PyPA tools _do_ provide a
| unified experience when used correctly.
|
| Here's a PyPA project (FD: one I work on) that uses a single
| pyproject.toml to handle all aspects of packaging (and most non-
| packaging tool configuration, to boot)[1]. With a single file
| like that, the _only_ thing you need to do to start a local
| development environment is: python -m venv env
| && . env/bin/activate python -m pip install .[dev]
|
| (We provide a `make dev` target that does that for you as well.)
|
| Similarly, to build distributions, all you need is `build`:
| python -m build
|
| [1]: https://github.com/pypa/pip-audit
| Denvercoder9 wrote:
| That repository doesn't seem to pin dependency versions. How do
| you integrate that in this workflow?
| woodruffw wrote:
| You could use `pip-compile` if you want full pinning. That's
| what we do on another project -- we use GitHub Actions with
| `pip-compile` to provide a fully frozen copy of the
| dependency tree for users who'd like that[1].
|
| In the context of `pip-audit`, that makes a little less
| sense: most of our dependencies are semantically versioned,
| and we'd rather users receive patches and fixes to our
| subdependencies automatically, rather than having to wait for
| us to release a corresponding fix version. Similarly, we
| expect users to install `pip-audit` into pre-existing virtual
| environments, meaning that excessive pinning will produce
| overly conservative dependency conflict errors.
|
| [1]: https://github.com/sigstore/sigstore-
| python/tree/main/instal...
| Izkata wrote:
| Or if you don't want to install something else and are
| willing to just use version numbers (instead of also hashes
| like pip-compile in that link), "pip freeze" is built in.
| woodruffw wrote:
| The tricky thing with `pip freeze` is that it dumps your
| _environment_ , not your _resolved set_ : your
| environment also contains things like your `pip` and
| `setuptools` versions, any developing tooling you have,
| and potentially your global environmental state (if the
| environment has global access and you forget to pass
| `--local` to `pip freeze`).
|
| In other words, it's generally a superset of the
| resolutions collected by `pip-compile`. This may or may
| not be what you want, or what your users expect!
| Izkata wrote:
| By default (at least in python3 nowadays) it also
| excludes pip, setuptools, distribute, and wheel; you need
| "--all" to include them in the output.
| woodruffw wrote:
| Oh, that's news to me! I stand corrected.
|
| (The point about other development tooling is, I believe,
| still accurate -- if you e.g. have `black` installed,
| `pip freeze` will show it.)
| fbdab103 wrote:
| Which is a huge limitation of many of the other tools. I
| have some beef with poetry, but it did at least get one
| thing correct: differentiating between the code required
| libraries and the development tooling (pytest, black,
| etc). There are hacky workarounds to accomplish this with
| other tools, but codifying this differentiation is
| incredibly valuable.
| lmm wrote:
| That's still about 4 times as much command as I'd consider
| normal/reasonable, compared to say "mvn install".
| qbasic_forever wrote:
| Nowhere in pypa documentation is your simple workflow described
| or mentioned. Instead it's a jumble of links to a myriad of
| tools including hatchling, flit, pdm, etc. and basically just a
| shoulder shrug, 'I don't know, figure it all out yourself'
| message. This article makes a great point that the current pypa
| 'guidance' is too confusing and vague for actual end users
| (i.e. people that don't work directly on pypa or haven't been
| studying python packaging for decades).
| woodruffw wrote:
| I agree that it's confusing. That being said, there is an
| official PyPA tutorial that goes through the exact steps need
| to produce the commands I suggested, so I don't think it's
| accurate to say that it's nowhere to be found[1].
|
| Edit: Unfortunately it's easy to confuse the above tutorial
| with this one[2], which is specifically for setuptools. So I
| can appreciate end user confusion around the documentation,
| particularly in that respect!
|
| [1]: https://packaging.python.org/en/latest/tutorials/packagi
| ng-p...
|
| [2]: https://packaging.python.org/en/latest/guides/distributi
| ng-p...
| qbasic_forever wrote:
| No I'm looking at link 1 and specifically have an issue
| with the huge table of options for how to setup your
| pyproject.toml--it has tabs for hatchling, flit, pdm, etc.
| but _zero_ description of why I would want to use one of
| those tools. It's just more confusing, like you're being
| thrown a bag of parts and no description of what to build
| or how to do it.
|
| To be honest that entire pypa doc should be like two
| paragraphs long instead of 1000+ words. It should be
| basically, "ok to package your python app just run
| <official python tool to init a package>, that's it you're
| done!". Every decision should be made for me, like it is
| with npm, cargo, etc. I shouldn't have to think beyond
| running one command.
|
| That's what python end users need. We don't need a million
| tools and huge docs with seemingly no vision or goal. We
| need to get shit done and the less time we faff about with
| packaging and tooling the better.
| Izkata wrote:
| AFAIK in the python 2 era it was the de-facto standard (just
| swap "python -m venv" with "virtualenv"). All these new tools
| have just made it more complicated and it's not clear to me
| what they gain, seems to me like they've simply succeeded in
| convincing people it's complicated by obscuring what's
| actually going on.
| fbdab103 wrote:
| A few weeks ago, I was attempting to help introduce someone
| to Python. I felt embarrassed at trying to paper-over how
| complicated and confusing the initial configuration is for a
| newbie. No officially endorsed best-practices to which I
| could point. Half a dozen tribes with their own competing
| solutions and guidance.
|
| A definitive statement from Python.org as to The Way would go
| so far.
| samwillis wrote:
| Ultimately it needs to be "Python.org" that endorses the tool,
| not PyPA, no one in the scheme of things know who PyPA are and
| if it's the "one true way".
|
| If you go to Python.org and follow through to the beginners
| guide [0] this is what's suggested:
|
| > _There are several methods to install additional Python
| packages:_
|
| > _Packages can be installed via the standard Python distutils
| mode (python setup.py install)._
|
| > _Many packages can also be installed via the setuptools
| extension or pip wrapper, seehttps://pip.pypa.io/._
|
| That is so out of date and fundamentally confuses people coming
| to Python for the first time. How is pip secondary, and no
| mention of venv?
|
| The PyPA need to get buy in from Python Core to put one tool
| front and centre first. It needs to be like Rust with Cargo,
| literally the first thing you learn to use and core to all
| beginners guides.
|
| That's not to diminish the work of PyPA, you are all amazing I
| just want your work to be more obvious!
|
| 0: https://docs.python.org/3/using/mac.html#installing-
| addition...
| Godel_unicode wrote:
| The future is here, it's just inconsistently distributed.
| There's also the problem of python.org not wanting to pick
| winners, i.e. not promoting activestate vs conda vs pypa.
|
| https://packaging.python.org/en/latest/tutorials/installing-.
| ..
|
| https://packaging.python.org/en/latest/tutorials/packaging-p.
| ..
| agumonkey wrote:
| where is the install .[dev]
|
| format/syntax defined ? I was trying to find what was possible
| and how to make sense of it in setup.cfg tools
|
| I found one mention in the docs but no more.
| baq wrote:
| you may have done pip install
| jupyter[notebook]
|
| this is the same thing, just for . (current directory) and
| dev variant.
| agumonkey wrote:
| yeah but I'm missing data on how variant interacts with
| various requirements defined in setup.cfg (testing
| dependencies for instance)
| woodruffw wrote:
| Yep. The key difference is that the former is specified in
| PEP 508, while `.` and its variants are a convenience
| feature that pip (and maybe some other tools) provide.
| woodruffw wrote:
| It's a pip-ism; as far as I know, it's not defined in any
| PEP. It should be in their documentation, however.
| agumonkey wrote:
| gonna dig deeper there then, thanks
| nprescott wrote:
| PEP-508[0] explains the grammar for "extras":
|
| > Optional components of a distribution may be specified
| using the extras field: identifier_end =
| letterOrDigit | (('-' | '_' | '.' )* letterOrDigit)
| identifier = letterOrDigit identifier_end* name
| = identifier extras_list = identifier (wsp* ',' wsp*
| identifier)* extras = '[' wsp* extras_list? wsp*
| ']'
|
| as well as explaining their behavior, albeit briefly:
|
| > Extras union in the dependencies they define with the
| dependencies of the distribution they are attached to.
|
| The resolution on . is explained by the pip documentation[1]:
|
| > pip looks for packages in a number of places: on PyPI (if
| not disabled via --no-index), in the local filesystem, and in
| any additional repositories specified via --find-links or
| --index-url. There is no ordering in the locations that are
| searched. Rather they are all checked, and the "best" match
| for the requirements (in terms of version number - see PEP
| 440 for details) is selected.
|
| [0]: https://peps.python.org/pep-0508/#grammar
|
| [1]: https://pip.pypa.io/en/stable/cli/pip_install/#finding-
| packa...
| nezirus wrote:
| Reading this is really depressing. I just want Cargo for Python.
| Poetry it is for now, but it has quirks, and it is dog slow...
| DangitBobby wrote:
| I have a single, simple script (not a package!) that has
| dependencies. Actually, I have a few of these, just sitting in
| /usr/local/bin so I can execute them whenever.
|
| How should I be managing environments for these scripts? Do I
| install dependencies in shared system python? Should I create a
| shared venv? Where should I store it? Any tools out there that
| make this decision for you and manage it?
|
| Just the fact that homebrew occasionally updates my installed
| Pythons and breaks everything makes me reluctant to use Python as
| a scripting language at all. It's pretty reliable when I have a
| folder dedicated to a project (poetry is good for this) but I'm
| pretty fed up with how brittle it all is.
| DangitBobby wrote:
| In an amazing coincidence, I just found another front-page post
| exploring this same topic! For Python, it seems to suggest
| using nix-shell.
|
| 1. https://dbohdan.com/scripts-with-dependencies
| DangitBobby wrote:
| My comment with an example of a single file script which can
| specify its own dependencies:
|
| https://news.ycombinator.com/item?id=34393630
| saila wrote:
| If you don't mind adding a pyproject.toml, you could use
| pipx[1] to install these scripts. The directory structure would
| look like this: <name>/
| pyproject.toml <name>.py
|
| The pyproject.toml would look like this (using poetry, but you
| could use a different tool for this):
| [tool.poetry] # NOTE: Set <name> to your project name
| name = "<name>" description = "My Script"
| version = "0.0.0" authors = ["Me <me@example.com>"]
| [tool.poetry.dependencies] python = "^3.11"
| requests = "^2.28.2" [tool.poetry.scripts]
| # NOTE: Change <name> to match your package name in
| [tool.poetry] <name> = "<name>:main"
| [build-system] requires = ["poetry-core"]
| build-backend = "poetry.core.masonry.api"
|
| Then you'd run `pipx install -e .` and the the executable
| script will be installed in ~/.local/bin.
|
| [1] https://pypa.github.io/pipx/
| DangitBobby wrote:
| Thank you for trying to provide a workable solution, it's
| really not bad, but it has some downsides for me. pipx itself
| is installed inside a python environment, so when brew breaks
| my pythons, it breaks pipx as well. Anytime brew breaks my
| pythons, I would need to do the install step again for every
| script (or write a tool myself which does it). Not a total
| deal breaker, but not really much better than my current
| situation which pretty much just assumes any version of
| `requests` or `fire` is acceptable. Because python itself is
| constantly being updated to break the base python environment
| on my machine, a workable solution would need to include the
| fact that the base python environment might need to have
| things installed.
| noisy_boy wrote:
| One option could be to have Python installed in a separate
| location specifically for this purpose and to NOT include
| it in PATH. Then it is "out-of-sight" of brew and such
| packages and sort. You can even make the entire location
| read-only once you are done with installation of Python +
| pipx etc.
| DangitBobby wrote:
| That would definitely work.
| fleekonpoint wrote:
| I use a separate directory and venv for each script. To execute
| the script, I use a shell script to call the venv's python
| interpreter. This is also how I use python scripts with
| cron/systemd. #!/bin/bash #
| myscript.sh venv/bin/python3 myscript.py
|
| You could also skip the shell script and use aliases in your
| .bashrc.
| DangitBobby wrote:
| I do something kind of like this, but all of my scripts break
| when the underlying env is suddenly broken when e.g. brew
| updates python without asking me and breaks all existing
| environments.
|
| I'm sure I could come up with solutions that are very robust
| for my particular machine, but I would like something that
| allows me to share it as a gist and a teammate could just as
| easily use it, or I can use it myself on another machine
| without hassle. In other words, a solution contained within
| the script iteself and maybe one or two binaries outside that
| make it possible.
| fleekonpoint wrote:
| I see, it might be heavy handed but running them inside
| Docker containers might provide you with the isolation
| you're looking for. You could also build and share these
| images with your teammates.
|
| I've actually started using a lot of different CLI tools
| with Docker, especially when the tool isn't available for
| my OS.
| wpietri wrote:
| I don't understand how there still isn't a good answer for
| this. It seems like such an obvious need, and for years I've
| heard go being promoted for solving this problem. I get why
| Python didn't start out with an answer for this, but in an era
| of 20 TB hard drives, I'm more than willing to burn space for
| ease of install.
| tempest_ wrote:
| I've pretty much settled on poetry for most things at this point.
|
| It still has a ton of rough edges like error messages still
| closer to a stack trace than something actionable and it's slow
| as hell resolving but it does the job 90 percent of the time
| without issue.
|
| Also running everything in docker now anyway which side steps
| some other problems.
| npage97 wrote:
| For anyone curious, the reasoning for the slowness is briefly
| described here: https://python-poetry.org/docs/faq/#why-is-the-
| dependency-re...
| odiroot wrote:
| Poetry still doesn't seem to support PEP621, instead requiring
| its custom, vendor-specific shape of pyproject.toml.
| gen220 wrote:
| Have they spoken about why they don't support it? Or plans to
| support it in the future?
|
| It seems like the shortest path to a universal package
| management tool for Python is to add this capability to
| Poetry, rather than to build something new.
| imglorp wrote:
| That's where we are as well, but yes the problems you didn't
| sidestep with poetry+docker are still there: the pypi
| fecosystem out there does not stop quantum fluctuations just
| because you want it to. If you pin too much you're not getting
| security fixes and if you don't pin enough, you get bitten
| every time a tertiary dep changes that you never asked for. Oh
| yeah and there are now trojans upstream almost daily.
| tempest_ wrote:
| Yeah but that trade off is entirely separate.
|
| We trade security for speed(or "velocity" if you want to be
| jargon about it).
|
| I just pin everything and go through my projects every couple
| weeks and bump the deps (unless some really big CVE hits the
| news).
| gen220 wrote:
| Does any FOSS package ecosystem have a good solution to this
| particular set of problems?
|
| I know Go has historically done something interesting with
| selecting the "minimum viable version number" for each
| dependency, but that's the only relevant idea that comes to
| mind.
|
| With PyPi, at least it's relatively trivial to self-host a
| pypi instance and self-manage dependency updates.
|
| In the Python ecosystem (vs e.g. Node), at least the total
| set of first+second-order dependencies for most normal
| projects is quite small (e.g. medium double digits to low
| triple digits). It doesn't feel too painful to manage that
| magnitude of upgrades on a monthly or quarterly basis.
| samwillis wrote:
| This post is prompted by the survey of Python users, their
| feedback, and a current thread [0] on the Python forms discussing
| a way forward. I read the thread the other day, it's long, and
| there are a lot of opinions.
|
| Towards the end of this post there is an interesting observation:
|
| > _Discourse, the platform that the discussion was held on, shows
| the number of times a link was clicked. Granted, this count might
| not be always accurate, but if we assume it is, the link to the
| results summary was clicked only 14 times (as of 2023-01-14 21:20
| UTC). The discussion has 28 participants and 2.2k views. If we
| believe the link click counter, half of the discussion
| participants did not even bother reading what the people think._
|
| Along with that, my concern is that there are too my cooks in the
| kitchen. A packaging system is never going to cater for more than
| 90% of use cases, but there will likely be a higher than average
| representation from people who would sit in the 10%. There is a
| danger of them never being able to agree on a solution that
| solves all problems, and then never launch anything.
|
| It would be far better to get _something_ out, have a stake in
| the ground, and build from there. It also desperately needs to be
| "Core Python", my understanding is that the PyPA (Python Package
| Authority) is somewhat disconnected from the core Python
| development.
|
| Ultimately they need one system that is front and centre, that is
| the "officially endorsed" tool, and very clear messaging on that.
|
| 0: https://discuss.python.org/t/python-packaging-strategy-
| discu...
| noisy_boy wrote:
| > Ultimately they need one system that is front and centre,
| that is the "officially endorsed" tool, and very clear
| messaging on that.
|
| I think this is important; otherwise there is a risk behind
| switching to a new packaging tool and then it being subject to
| neglect/lack of resources - that makes people averse to
| switching to it. That is why virtualenv/pip is the lowest
| common denominator - everyone knows that worst case, those will
| always continue to work. The "official" tool needs to inspire
| that sort of confidence.
| nmstoker wrote:
| Yes, there was some questionable comms around pipenv being
| "officially recommdnded" which upon closer inspection seemed
| to have come from the people who wrote it and not really been
| that official so far as I could see! That seems to have been
| walked back after a while but not before it gained traction
| and randoms bring up that it's official even now.
| carapace wrote:
| I think you're right. At risk of making a "lowbrow dismissal"
| the very length of TFA is a clear symptom of the problem:
| _Design-by-committee_ (The PyPA (Python Package Authority)
| reminds me of the movie _Brazil_. Don 't get me started.)
|
| - - - -
|
| Someone should do for packaging what _pathlib_ did for file I
| /O: make a Pythonic model/API for it. Then deployment becomes
| simple, scriptable, testable, repeatable, etc...
| woodruffw wrote:
| This might be a fun thought to entertain, but the PyPA
| doesn't really form design committees (at least, I haven't
| been on one). You can see exactly how Python packaging
| standards are made: they're done with PEPs[1], exactly the
| same as with every other Python standardization effort.
|
| Indeed, most packaging PEPs start out exactly the way you've
| laid out: a tool or service writes a Pythonic API or model,
| and it gets standardized so that other tools can rely on it.
| TFA's problems (which are real ones!) stem mostly from the
| error before there were serious standardization efforts among
| Python packaging tools.
|
| [1]: https://peps.python.org/topic/packaging/
| carapace wrote:
| > _Don 't get me started._
| marsten wrote:
| > there will likely be a higher than average representation
| from people who would sit in the 10%
|
| And also an under-representation of "average users".
|
| Python the language has always benefitted from its relative
| simplicity, which I attribute to people like GVR saying "no" to
| specialized features that accrue on languages like barnacles on
| a ship (looking at you C++).
|
| With newer languages we see core design teams being much more
| opinionated about tooling for build and dependendency/packaging
| management. This is no doubt in response to the dumpster fires
| of C++ and Python, where "the ecosystem will sort itself out"
| clearly hasn't worked.
| mkoubaa wrote:
| I always like to say that's barring divine inspiration, it's
| impossible to make the right choice. It's far more possible to
| make a choice and then work hard to make it right
| forkerenok wrote:
| Thanks, this is an interesting nugget. Did you intend to append
| a link reference for [0]?
| samwillis wrote:
| Yes, edited. Thanks.
| benreesman wrote:
| Trying to build Python under Nix (i.e. solid, not "close enough)
| is an education in how fucked up the Python packaging ecosystem
| is.
|
| Which is a shame because Python is an amazing scripting language,
| the de facto numerical computing standard, and probably a better
| bash for most bash use cases.
| synergy20 wrote:
| So we should all use PDM then as it's the best so far among all
| others? I read it somewhere once in the past but never used it. I
| just use `venv + pip` all the time so far, worked well for my
| case though.
| jasonhansel wrote:
| Over time, I've grown to appreciate "pip-tools." Since it's a
| dead-simple extension of pip, I wish it could be upstreamed into
| pip itself; that seems like the most straightforward way of
| fixing a number of Python's packaging issues.
| vlovich123 wrote:
| Yup. I like to make it so that each executable has a symlink to
| a shell script.
|
| The shell script checks if there's a virtual environment set up
| with the packages in the requirements.txt installed (it takes a
| snapshot of the file because virtualenv doesn't have a DB to
| query cheaply). Once the environment is set up, it dispatches
| to running from the virtualenv.
|
| That way when you update a requirements.in file it recompiles
| it (if the txt is out of date) and installs any new packages,
| removes packages that shouldn't be there anymore and updates
| ones whose version changed (if there's any changes found). It
| also lets you trivially run tools with disparate
| requirements.in without conflicts because each is siloed behind
| its own virtualenv.
|
| This makes it a trivial experience to use these tools in a
| shared repo because there's no worrying about packages /
| needing to remember to run some command before the right
| environment is set up. You just modify your code and run it
| like a regular command-line tool and packages automatically get
| deployed. It's also amenable to offline snapshotting /
| distribution. In fact, I used this to distribute support
| tooling for factory lines of the Pixel Buds and it worked
| extremely well.
| orf wrote:
| You've described a worse, homegrown version of poetry.
| delijati wrote:
| I work on mutlirepos and i really dislike gits subtree, subrepo
| i use https://pypi.org/project/zc.buildout/. Yes i know i can
| do [1]. But editing on multirepos at the same time i can only
| do with zc.buildout [2] Still not perfect but it does the job.
|
| [1] $ cat req.txt requests
| git+ssh://git@gitlab.com/foo/bar/amber.git@0.4.0
| git+ssh://git@gitlab.com/foo/bar/rebam.git@0.4.0
|
| [2] $ cat buildout.cfg [buildout]
| extends = versions.cfg extensions = mr.developer
| auto-checkout = \* develop = . show-picked-
| versions = true update-versions-file = versions.cfg
| sources-dir = git-sources parts = py
| eggs = amber rebam hammer
| [sources] amber = git
| git@gitlab.com/foo/bar/amber.git@0.4.0 rebma = git
| git@gitlab.com/foo/bar/rebma.git@0.4.0 [py]
| recipe = zc.recipe.egg eggs =
| ${buildout:eggs} interpreter = py-backend
| dependent-scripts = true
| pdw wrote:
| Yes. It usually gets glossed over, but if you're building a
| Python application it's all you need.
| TigerTeamX wrote:
| I must admit I just gave up on all these tools. Instead I just do
| pip install -r requirements.txt -t .lib
|
| and I have import sys import os
| sys.path.append(f"{os.getcwd()}/.lib")
|
| in the top of my script. Some will tell me this is silly, but it
| just works. Rememer to at .lib to your .gitignore. Else you'll
| have lot of fun.
| TigerTeamX wrote:
| This is similar to how node is doing. I just basically copied
| that idea.
| hkgjjgjfjfjfjf wrote:
| [dead]
| reducesuffering wrote:
| https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...
|
| Is quite damning on how much of a better experience Node/JS/TS
| NPM is than Python's clusterfuck. Working with any other JS/TS
| open source is easy to get running. Even if you use Poetry in
| Python, you will run into everyone else using pip-tools, pipenv,
| pdm, conda, etc.
| baggiponte wrote:
| been using pdm since at least one year and have no regrets: it's
| the go tool and we should flock to it!
___________________________________________________________________
(page generated 2023-01-15 23:00 UTC)