[HN Gopher] How Python virtual environments work
___________________________________________________________________
How Python virtual environments work
Author : amardeep
Score : 211 points
Date : 2023-03-13 05:10 UTC (17 hours ago)
(HTM) web link (snarky.ca)
(TXT) w3m dump (snarky.ca)
| cozzyd wrote:
| The "global" vs. "directory" dichotomy seems... off. Haven't
| PYTHONHOME and PYTHONPATH been supported since approximately
| forever?
| josteink wrote:
| All other languages: use whatever packages you like. You'll be
| fine.
|
| Python: we're going to force all packages from all projects and
| repos to be installed in a shared global environment, but since
| nobody _actually_ wants that we will allow you to circumvent that
| by creating "virtual" environments you can maintain and have to
| deal with instead. Also remember to activate it before starting
| your editor or else lulz. And don't use the same editor instance
| for multiple projects. Are you crazy???
|
| Also: Python "just works", unlike all those other silly
| languages.
|
| Somebody IMO needs to get off their high horse. I can't believe
| Python users are defending this nonsense for real. This must be a
| severe case of Stockholm-syndrome.
| frou_dh wrote:
| Python is over 30 years old so it's hardly surprising that it's
| got plenty baggage in at least some areas.
| rad_gruchalski wrote:
| Yeah, how does it go? There's at least one obvious way to do
| something? Python... makes me anxious. I don't mind writing
| Python but setting it up is crazy.
| Max_Limelihood wrote:
| Answer: they don't
|
| (Seriously, I've gotten so fed up with Python package management
| that I just use CondaPkg.jl, which uses Julia's package manager
| to take care of Python packages. It is just so much cleaner and
| easier to use than anything in Python.)
| birdstheword5 wrote:
| It sounds mean to say it, but it's 100% true. I moved away from
| using python wherever I can. I've had colleagues struggle for
| days to install well used packages like pandas and numpy in
| conda.
| prds_lost wrote:
| Your colleagues should consider skipping conda and stick to
| using venv. It will make life much easier. Given pandas/numpy
| is huge in data science, moving away is not much of an option
| unless you are working on a personal project or already have
| a dedicated team comfortable with using a different stack.
| There is also the Docker option which is great but much more
| involved.
| zelphirkalt wrote:
| My advice would be to not use Conda or other such "extra
| tooling". More trouble than benefit. Stick with venv and
| poetry.
| zzzeek wrote:
| poetry is "extra tooling"
| justinsaccount wrote:
| Things like pandas and numpy are not python packages. Yes,
| they are packages FOR python, but they are not python.
|
| https://hpc.guix.info/blog/2021/09/whats-in-a-package/ does a
| good job of explaining why installing packages like that is a
| complete shitshow.
| danielvaughn wrote:
| I just began writing Python a few months ago. For years
| prior, I'd been a JS dev, and while NPM can be frustrating at
| times, I never encountered so many issues as I have in
| Python. It's crazy.
|
| I'm now curious whether there are languages out there that
| _do_ have a really nice packaging system.
| dvlsg wrote:
| Cargo (Rust) is pretty solid. Most of my minor complaints
| (like being unable to add packages from the CLI) have been
| resolved with time, as well.
| Nadya wrote:
| Personally, zero complaints about Cargo (Rust) and very
| minimal complaints about NuGet (C#/.NET). My issues around
| NuGet are probably self-created because I refuse to learn
| the CLI [0] for it and I've had occasional issues with
| Visual Studio's UI for managing things.
|
| https://learn.microsoft.com/en-us/nuget/reference/nuget-
| exe-...
| evntdrvn wrote:
| In a lot of ways, Paket is significantly better than
| NuGet, if you ever want to try something new :) It uses a
| lockfile approach like Cargo, has better dependency
| resolution, etc
| https://fsprojects.github.io/Paket/index.html
| hot_gril wrote:
| I can't think of any package/dep systems I actually like
| other than npm. And they're even starting to screw that up
| with the `import` weirdness instead of the `require` that's
| been so simple and easy.
|
| Rust's system is probably the next best.
|
| ObjC/Swift packaging is a flaming disaster in practice,
| unless it's improved since I jumped that ship. Last time, I
| remember every single project having to rely on Cocoapods.
| throwawaymaths wrote:
| Elixir's packaging system is quite good. We went from empty
| project to working stable diffusion in 2h. 1.75 of those
| hours was installing CUDA.
| cstrahan wrote:
| FWIW, I find Cargo to be one of the biggest reasons I like
| Rust so much -- maybe even more than anything to do with
| Rust itself or safe code.
|
| I'll often look for command line tools written in Rust, but
| not because of Rust fanboyism, but because I know I can
| just git clone the project and immediately start hacking on
| a new feature I need or a quick bug fix. In almost every
| other language I have to jump through one million hoops
| before I can build and run whatever it is, let alone have a
| nice developer experience (autocomplete, go to definition,
| etc).
| adgjlsfhk1 wrote:
| Yeah, one of Julia's best decisions was taking heavy
| inspiration from Rust for the package manager. Rust was
| 100% the first language to get dependency management
| right.
| okeuro49 wrote:
| > Rust was 100% the first language to get dependency
| management right.
|
| In my experience, Java, Go, PHP, NodeJS have all got
| similar package management that works.
| justinsaccount wrote:
| See my above comment, there's a reason why all of your
| examples work:
|
| Java package managers tend to install packages written in
| java
|
| Go installs packages written in go, and maybe C using cgo
|
| Cargo installs packages written in rust
|
| php package managers install packages written in PHP,
| extra extensions are rare
|
| etc
|
| People having trouble with python are NOT having trouble
| with python. They are having trouble because they are
| trying to use packages that are just python bindings to
| absolutely massively complex c++ and Fortran libraries.
|
| Often people using python don't even have a C compiler
| installed (let alone a fortran one for the scientific
| stuff), so pip blows up the first time they try to
| install a package that hasn't been pre-built for their
| system+python version.
| fiddlerwoaroof wrote:
| Yeah, npm was the first good package manager. It gets a
| lot of hate but my experience is that its strategy is the
| optimal solution for the problem it solves. And, I think
| a lot of things people complain about (lots of trivial
| packages, huge dependency trees, etc.) are an effect of
| solving the packaging problem well: if you make it easy
| to add dependencies, people will take advantage of that
| to add lots of dependencies.
| cpach wrote:
| IMHO, Go's packaging system is very pleasant to use.
| ChancyChance wrote:
| Same here, I've been using yarn for years, and when I
| started using venv, I didn't understand why it had to be so
| complex. Even after reading this article, I still don't see
| why it is so complex! Yarn/npm has the right idea:
| dependencies go in the working folder and expect that
| hierarchy/protocol. Problem solved. The only problem I have
| with yarn/npm is the problem any package manager has and
| that is the attrition of dependencies and how to rank their
| security risk.
| dangerlibrary wrote:
| I hate python package management - I really do. But I've never
| actually had a problem with virtual environments, and I think
| it's because I just use virtualenv directly (rather than conda
| or whatever else).
|
| I have these aliases in my .bashrc, and I can't remember the
| last time I had a major issue.
|
| alias venv='rm -rf ./venv && virtualenv venv && source
| ./venv/bin/activate'
|
| alias vact='source ./venv/bin/activate'
|
| alias pinstall='source ./venv/bin/activate && pip install . &&
| pip install -r ./requirements.txt && pip install
| ./test_requirements.txt'
|
| I don't have all the fancy features, like automatically
| activating the virtualenv when I cd into the directory, but
| I've always found those to be a bigger headache than they are
| worth. And if I ever run into some incompatibility or duplicate
| library or something, I blow away the old venv and start fresh.
| It's a good excuse to get up and make a cup of tea.
| spprashant wrote:
| I might steal these aliases, thank you.
|
| Using virtualenv directly has also been my approach, and has
| not failed me yet.
|
| I also used Poetry for one of my personal projects, and I
| liked what I saw.
| rewgs wrote:
| This is similar to what I do, except my "new environment"
| alias executes a function that takes a python version,
| installed/specified via pyenv.
|
| Never had a single problem, venv + pyenv is a great combo. As
| far as I can tell, like so many sources of frustration in
| tech, the issue typically lies with user error/not fully
| understanding the tool you're using. That isn't saying that
| there isn't room for improvement -- most notably, package
| management in Python flies in the face of "there should be
| one -- and preferably only one -- obvious way to do it" --
| but the tools we have work quite well.
| Joker_vD wrote:
| > source ./venv/bin/activate
|
| To this day I'm not quite sure why the venv developers
| decided that sourcing was a good idea; all it does can be
| effectively replaced with #!/bin/sh
| export VIRTUAL_ENV="path to venv" export
| PATH="$VIRTUAL_ENV/bin:$PATH" unset PYTHONHOME
| exec "$SHELL"
|
| Just run this script to get into an "activated" shell. To
| deactivate, just press Ctrl+D. If you're really fancy, you
| can replace the last line with exec
| "${@:-$SHELL}"
|
| to run a command directly in the activated environment (and
| then deactivate it immediately).
| pottertheotter wrote:
| I use pyenv[1] and the pyenv-virtualenv[2] plugin and I've
| not had a problem. It's so easy.
|
| [1] https://github.com/pyenv/pyenv [2]
| https://github.com/pyenv/pyenv-virtualenv
| Karrot_Kream wrote:
| pyenv needs to have its shims in place by running the pyenv
| init. You can run it when your shell starts but I find it
| kind of slow and for a while it used to be wonky in fish.
| But once I run the init, pyenv does work.
|
| That's just for managing your python installation and
| virtualenv though. You still need to manage your packages
| and for that you have options like requirements.txt, pipenv
| (not pyenv lol), Poetry, and others.
| Izkata wrote:
| > virtualenv venv
|
| That would be python2, in 3 it's "python -m venv venv" (first
| venv is package to run, second is directory to put it in)
|
| Otherwise yeah, it's the same and I also use it manually.
| Never had any problems.
| jvolkman wrote:
| `virtualenv` still exists and is still actively developed.
| It's true that Python 3 ships with `venv` but I think
| `virtualenv` offers some additional features.
|
| https://github.com/pypa/virtualenv
| danielvaughn wrote:
| A few weeks ago I spent about a week debugging my Poetry
| environment. Turns out, their latest release (which I believe
| was a patch bump!) brought in some breaking changes. And on
| top of that, a bunch of stuff was forcing python3.11 under
| the hood, whereas I was on python3.10.
|
| It was a nightmare.
| silverwind wrote:
| Poetry seems to break compatibility with every release of
| either itself or Python, double the fun.
| albert_e wrote:
| I have struggled with conda and the huge space it usually
| eats up
|
| I should learn to use venv properly
|
| Thanks
| jimnotgym wrote:
| Agreed. I tried the new package manager combined with venv
| and using venv directly seems best. A lot faster for a start.
| kgodey wrote:
| I use virtualenvwrapper[1] and can't remember any problems
| with virtual environments either. It sets up human readable
| aliases for you like "mkvirtualenv" to create a virtualenv
| and "workon" to activate a virtualenv.
|
| [1] https://github.com/python-
| virtualenvwrapper/virtualenvwrappe...
| swyx wrote:
| also there's like 3 different flavors of virtual env now and me
| being 8 years out of date with my python skillz i have no idea
| what the current SOTA is with python venv tooling :/
|
| i dont need them demystified, i need someone smarter than me to
| just tell me what to do lol
| dwringer wrote:
| > and me being 8 years out of date with my python skillz i
| have no idea what the current SOTA is with python venv
| tooling :/
|
| It doesn't really matter, by the time you sit down and use it
| you'll find whatever that is, has also been deprecated and
| replaced by 2 more.
| billforsternz wrote:
| The problem with software development in 2023 in a
| nutshell, well played sir.
| hot_gril wrote:
| > i dont need them demystified, i need someone smarter than
| me to just tell me what to do lol
|
| Dockerfile ;)
| gosukiwi wrote:
| It also makes it very hard for new devs willing to learn
| Python. Coming from Ruby and JavaScript, you just use bundler
| or npm, but Python is so strange, even the way it runs files
| is different, with the module thing.
| ollien wrote:
| The reality is that if you ask 3 different people you're
| going to get 3 different answers. They're fundamentally the
| same, just a matter of package management. As far as I'm
| aware, the current "SOTA" is Poetry. I liked Pipenv for quite
| some time, but Poetry is just so much faster IME.
| buildbot wrote:
| I personally hate Conda with a firey passion - it does so much
| weird magic and ends up breaking things in non obvious ways.
| Python works best when you keep it really simple. Just a python
| -m venv per project, a requirements.txt, and you will basically
| never have issues.
| insane_dreamer wrote:
| I'm the opposite. We have to maintain a lot of different
| environments for different projects, and with conda things
| "just work" (esp now with better PyCharm integration). Venv is
| much more of a hassle.
| pbronez wrote:
| Until you want to use anything with a c extension..
| wheelerof4te wrote:
| Why would you want to use Python with a C extension?
|
| If you need performance, just use native code.
| peterhil wrote:
| Numpy and Scipy are good reasons. Unfortunately Scipy does
| not even compile on FreeBSD lately, and I have opened three
| issues about it against Scipy and Pythran (and the fix was
| with xsimd).
|
| https://github.com/serge-sans-paille/pythran/issues/2070
| buildbot wrote:
| I use it all the time for things with c extensions?
| bobbylarrybobby wrote:
| If it were really that simple, surely all these other solutions
| wouldn't exist?
| zzzeek wrote:
| it really is that simple
|
| conda exists because it deals with an entirely different
| package registry and is a whole distro on its own (I dont
| know why people need that either, my vague impression is that
| scence-y types want complete pushbutton installation, typing
| a command == fail, I guess, I dont know).
|
| poetry exists because it does some kind of automated version
| management thing (as did pipenv), that I'm sure is nice but
| not something I've ever needed, but the buzz got out and it
| took over, people who have never typed "virtualenv" use
| poetry and they have no idea why.
| squeaky-clean wrote:
| Conda is very nice for people who just want to write some
| python and not need to learn about the ecosystem or weird
| issues like some Visual Studio dependency not being
| installed to get scipy to compile on Windows or whatever.
| Like someone coming from R to Python. But aside from that
| situation I agree with you.
| zzzeek wrote:
| so there's four responses here and 100% of them refer to a
| single Python package, Scipy, as the source of all the
| problems, where they had bad experiences over ten years ago
| with Python 2.
|
| the Python package index now supports binary wheel files
| for all platforms and Scipy is there
| https://pypi.org/project/scipy/ with a few dozen distros.
|
| is the problem solved yet ?
| Filligree wrote:
| You can't install the CUDA base libraries with pip. You
| can with Conda.
|
| So no, it isn't solved yet.
| buildbot wrote:
| Good? I don't want my package manager messing with my
| cuda toolkit setup!
| jayvius wrote:
| > (I dont know why people need that either, my vague
| impression is that scence-y types want complete pushbutton
| installation, typing a command == fail, I guess, I dont
| know).
|
| Essentially, yes, and justifiably so. Try installing
| science-y Python packages on Windows written in C. When
| conda was created in 2012 this meant installing Visual
| Studio 2005 (for Python 2.7) which was hard to find on
| Microsoft's own website even back then.
| teruakohatu wrote:
| > vague impression is that scence-y types
|
| Because often python is only one part of the puzzle, we
| also need a variety of third party libraries and maybe R as
| well (along with R packages). And we would rather not have
| to explain to another researcher how to install it all.
| crabbone wrote:
| > (I dont know why people need that either, my vague
| impression is that scence-y types want complete pushbutton
| installation, typing a command == fail, I guess, I dont
| know).
|
| It started in a way similar to how Active Perl started. An
| attempt to build a reliable, curated and "batteries
| included" solution. (But it went downhill very quickly).
|
| So, what were the problems:
|
| * Most importantly, using Python on Windows is a pain (it
| still is, but less so). Many Python packages rely on native
| libraries which they don't provide. Even if they ship with
| some binary blob, that blob is often just a wrapper to the
| actual library. But, at the time Conda was created, even a
| binary blob was not an option. If you wanted to install
| stuff, you had to compile it. And, compiling on MS Windows
| is a world of pain. Especially, if you compile something
| for Python. Again, credit where it's due, Python sort of
| fixed some of these problems... but just a tiny bit. It
| still sucks a lot. So, conda is more of a MSYS2 really:
| it's a Linux user-space in Windows (firstly), and Python on
| top of that. But, it kind of also doesn't want to admit it,
| so, it still pretends it's Windows... but with a bunch of
| Linux-ey stuff...
|
| * Secondly, pip was a joke until about a year ago. It was
| not really a package manager (it still isn't) and it
| couldn't properly solve dependencies (it sort of does
| today, but poorly because of source-distributions). Conda,
| when it worked, was at least somewhat consistent. (But
| usually, it just hangs). Also, conda is a package manager
| (however awful). I.e. with pip you can run install once,
| and then you can run a different install command again in
| the same environment, and god help you to make sure you
| have consistent dependencies between packages. Conda, in
| principle, should always keep your environment consistent.
| (But it hangs more often than not). Also, there's no such
| thing as installing from source with conda. If you want to
| use source: git clone, install conda-build and suffer
| through non-existent documentation and get help from non-
| existent community of people building conda packages.
|
| * Conda provides a curated set of packages (a.k.a default
| channel). Anyone worth their salt ditches that channel the
| moment they install conda and installs everything from
| conda-forge (cuz its got more stuffs!) So, they tried
| curated. Didn't work.
|
| ---
|
| In a sense, I think that what happened is that conda was
| too ambitious of a project, using wrong technology to
| implement it. It also didn't have good enough planning for
| the kind of ambition they had. So, every time they came to
| solve a real problem, they didn't have time, human
| resources and system resources to solve it well. They've
| accumulated technical debt at close-to-light speed and
| ended up being a huge mess nobody knows how to deal with.
|
| Some of the obvious mistakes would be: creating too many
| releases of conflicting packages. They should've worked on
| some sort of LTS solution, where they release a set of
| packages with very permissive dependencies of very few
| versions that had been proven to work well together.
| Instead it's very common for conda packages to be very
| peculiar (and without real need to do so) about the
| versions of their dependencies.
|
| Conda people didn't build good CI. They often release
| absolutely broken packages, and only discover it
| retrospectively from community input. (Tensorflow would be
| a great example). This creates a lot of problems with
| automatic updates.
|
| Conda gave in to community pressure and decided to build
| integration with pip. It doesn't work well, and it's not
| possible to make it work well, but they added this and a
| lot of people instantly created dependencies on this
| feature.
|
| Conda picked a bone with some of the infra tools outside of
| Python ecosystem. In particular with CMake. This just adds
| an extra aspect of pain trying to build conda packages /
| work with native libraries / wrappers. It might seem like a
| minor thing, but it prevented a lot of people who might
| otherwise release packages both for conda and PyPI from
| doing so. Often package that is released to conda is ported
| by someone who's not the package author. It also means that
| sometimes the names of packages differ between conda and
| PyPI for the same package.
|
| ----
|
| NB. In terms of amount of commands you need to type when
| working with conda vs with PyPI tools is not noticeably
| different. Conda is, perhaps, more organized, but is also
| more buggy due to trying to integrate with various shells
| in special ways, and failing quite a bit.
| pletnes wrote:
| Before conda, you needed a C and a fortran compiler, a BLAS
| and LAPACK installation, and various build tools, to
| install scipy. Scipy is one - 1 - dependency used in
| science and engineering. The pain used to be massive. Now,
| we're complaining about the existence of 2 high quality
| (not perfect) alternatives, and pip (part of the official
| python distro) can install tensorflow etc, GPU libs
| included, with a oneliner.
| schainks wrote:
| If software were that simple, surely all these other
| languages wouldn't exist?
| bobbylarrybobby wrote:
| Right, software is anything but simple
| crabbone wrote:
| Any program with real world application solves some problem.
| That's not the point. The point is in _how_ does it do it.
| And when it comes to conda... it 's like MS Outlook on
| steroids. I cannot really think about a better way to
| describe it. It's like a movie that can be so bad that it's
| actually good.
| hot_gril wrote:
| Idk, I still have no idea why yarn exists when there's npm.
| buildbot wrote:
| Not saying Conda et. al don't solve problems, especially in
| specific cases, but they also add them in my opinion.
| Spivak wrote:
| So there's two notions of simple.
|
| Simple in the sense that it's actually simple, the software
| you need can be installed with pip install with precompiled
| binaries for your platform when necessary it supports Python
| 3.something+, and all it's dependencies are either >= version
| or the version is >= x.y <= x+1.0
|
| Then there's simple as in the software is actually incredibly
| useful but is an absolute nightmare of complicated dependency
| trees where only specific pinned minor versions work
| together, you need multiple incompatible compiler toolchains
| and distro packages, it only works if you have CUDA,
| precompiled binaries exist for some but not all and if you
| use the precompiled binaries then it changes the dependency
| story, if you want jupyter support that's a whole different
| thing AHHHHHHHHHHHHH
|
| In that case some people with more time than sanity said fuck
| it we'll make it work and conda was born. For me it's a
| lifesaver when you want to use a piece of software but I
| wouldn't ever dare deploy production software with it without
| it being strongly isolated from everything else.
| crabbone wrote:
| I remember when Conda just appeared... I was so high no
| hopium... well, the word "hopium" didn't exist yet...
|
| Anyways. Today I have to help scientists to deal with it.
| And... I didn't think it was possible to be worse than pip or
| other Python tools, but they convinced me it is. Conda is the
| worst Python program of note that I had ever dealt with. It's
| so spectacularly bad it's breathtaking. You can almost
| literally take a random piece of its source code and post it to
| user boards that make fun of bad code. When I have a bad day, I
| just open its code in a random file, and like that doctor who
| was happy running around the imaging room exclaiming "I'm so,
| so, so happy! I'm so unimaginably happy that I'm not this guy!
| (pointing at an X-ray in his hand)" I'm happy I'm not the
| author of this program. I would've just died of shame and
| depression if I was.
| jstx1 wrote:
| With plain venv it's hard to maintain multiple different Python
| versions on the same machine; conda makes this much easier.
|
| Also on M1/M2 Macs some libraries (especially for ML) are only
| available through conda-forge.
| dragonwriter wrote:
| > With plain venv it's hard to maintain multiple different
| Python versions on the same machine
|
| Ironically, given the usual software dev experience on
| Windows vs. Unixy systems, this is not a problem with the
| standard install on Windows with the py launcher.
| zzzeek wrote:
| I have like seven pythons on my machine, and I use virtualenv
| all over, there's no issue, what's the issue? I have to type
| the path of a specific interpreter when i make a virtualenv,
| is that the problem? I bet that's the problem
| crabbone wrote:
| > With plain venv it's hard to maintain multiple different
| Python versions on the same machine;
|
| plain venv never advertised itself as a solution to this
| problem... I don't like this tool, but, sorry to say so, you
| are barking on the wrong tree.
|
| Also, it's not hard to maintain different versions of Python
| on the same machine without conda. I can literally do this
| with my eyes closed w/o touching the mouse: it boils down to:
| cd ~/src/cpython git checkout 3.8 git fetch
| --all git reset --hard origin/3.8 git clean
| -Xdf ./configure --with-ensurepip=install --with-
| system-ffi=yes make sudo make altinstall
|
| Sorry. I've done this from memory and I kept one eye open.
| So, I kinda lied. But not really.
| [deleted]
| [deleted]
| buildbot wrote:
| Is it? I just install different version with brew, and choose
| which python3.X to create a venv with. The ML packages issue
| is much more of an issue, but now that the ARM transition is
| well underway more and more packages work via normal pip.
| Already__Taken wrote:
| Been really enjoying trying out pdm in PEP 582 mode. I've just
| found it behaves when used across multiple devs, not necessarily
| that used to working with python.
| gt565k wrote:
| Just setup a django project with pipenv, works just fine.
| aniforprez wrote:
| Pipenv has never once worked just fine personally. The
| dependency resolution is a joke and the slowest of any project
| in this space, they have tons of bugs and the project is
| languishing
|
| I prefer to use a combination of pip-tools and pyenv for my
| projects
| PhysicalNomad wrote:
| I don't bother with venvs anymore and just use podman instead.
| sakex wrote:
| It feels like it is one of the reasons experienced devs are
| ditching Python for production systems. Besides horrendous
| performance and lousy semantics. The cost of setting up,
| maintaining the environment and onboarding people is just not
| worth it.
| ggregoire wrote:
| Experienced Python devs run their projects in docker, which
| solves the 3 issues you listed at the end.
| lucb1e wrote:
| Experienced Python dev checking in here. I use neither
| containers nor virtual environments. Honestly not sure what
| all the fuss is about.
|
| Maybe everyone has come to think we need layers on layers on
| layers because management tools (like venv) are blogged and
| talked about, whereas it's a bit dull to write/talk about
| nothing (i.e. not using/needing such tools)? I genuinely
| wonder
| phonebucket wrote:
| Are experienced devs really ditching Python for production
| systems? I wasn't under this impression.
| lucb1e wrote:
| > The cost of setting up, maintaining the environment and
| onboarding people is just not worth it.
|
| I have yet to come across a situation where I need a virtual
| environment at all. A lot of projects use it, but then lazy me
| just runs git clone && python3 clone/main.py and it works just
| fine, sometimes after an apt install python3-somedependency or
| two.
|
| It always seemed weird to me to depend on such a specific
| version ("package foo can't be older than 7 months or newer
| than 4 months"), how even does one manage to use such obscure
| features that they were removed and so you need an older-than
| version?
|
| And then there's the people in this thread (seemingly a
| majority when judging by top level comment loudness) that have
| trouble with virtual environments and so _add another layer on
| top_ to manage that virtual environment thing. The heck. Please
| explain
| hot_gril wrote:
| Do you deal with the scientific libs? I remember that whole
| MatPlotLib/Scipy/Pandas/Jupyter/whatever stack having weird
| requirements on dependencies, with Py2 vs 3 confusion added
| to the mix.
| lucb1e wrote:
| I've used matplotlib and pandas fairly recently because
| applicants used it in their code, don't remember any
| specific problems with that. Well, with the dependency
| installation that is. The applicant loading all data into
| RAM so that Pandas can operate on it _was_ an issue. (The
| whole point was that the data was too big to reasonably
| load into RAM, and would constantly grow as more sensor
| readings come in, and the questions are such that you can
| easily stream it and keep a few KB of state in a dictionary
| to find the answers needed... alas.)
|
| I do remember python2-only being a problem back in the day,
| but this was solved... hmm, maybe in 2017 somewhen? At
| least for the packages I used then that had py3 issues
| before, like sklearn, numpy, and scapy come to mind. I
| think it more or less coincided with Debian deciding the
| next release was not going to ship Python 2. Somehow that
| made everyone 2to3, fix a few remaining bugs, release,
| done. I'm too young (I'm 30) to really have done much with
| Python 2 so I didn't have this legacy in my own code
| (besides a few early files), I've ~always just tried to
| find and install Python 3 versions of everything.
| analog31 wrote:
| I think the problems have been resolved. For my last few
| installations of Python, I've just used pip install for
| those packages, without any issues. Linux and Windows,
| can't comment about Mac. And the important libraries are
| all in Py3 now.
| lockhouse wrote:
| Excuse my ignorance, but aren't Virtual Environments something
| you setup once per project? Why would that be a dealbreaker?
| How is it any more difficult than putting everything in a
| docker container like all the cool kids are doing these days?
| tmpz22 wrote:
| * Local state is changing such as brew updates or new
| dependencies are added.
|
| * External state is changing such as project contributions
|
| So its not a one-off unless the project and dev environment
| is static. The real problem is different tooling doing
| different amounts of hand holding and automation. Your editor
| may configure some things automatically, brew may configure
| some things automatically, and so a set of instructions for
| setup or problem fixing could be voided without the end user
| knowing. So now you're off on a adventure of unknown duration
| wading through internet forums trying to find the resolution
| that works for you.
|
| Ironically using Docker to isolate the environmental changes
| is a approach some people use to avoid some of this esoteric
| crap.
| groestl wrote:
| > Ironically using Docker
|
| I'd even dare to say that Docker _is_ the answer to the
| Python's packaging problems, and might have never taken off
| without that "killer app" that is the Python packaging
| sh*tshow.
| hot_gril wrote:
| Yep. Any big Python repo is gonna have a Dockerfile. "Fix
| your mess or we'll fix it for you."
| crabbone wrote:
| Docker is not the answer to any packaging problems
| because it's not a packaging tool. I have no idea how
| people don't understand this... oh wait! I'm talking to
| _Python_ programmers!
|
| But... I'm not going to hold you in the dark: the reason
| and the major drive to have a packaging system is that
| you can define dependencies between packages s.t. users
| installing package can coordinate and install the stuff
| they need. Docker simply doesn't do that. You get images.
| They are completely independent. Whether any two images
| will work with each other is anyone's guess.
| lockhouse wrote:
| Isn't it considered best practice not to use brew to
| install Python for exactly this reason?
|
| I've always seen it recommended to use pyenv or just
| download directly from Python.org instead.
| hot_gril wrote:
| I avoid brew whenever possible, partially because it does
| weird stuff just to avoid installing things as root.
| Something like Python with a convenient .pkg download
| directly from python.org makes the choice obvious.
|
| That said, some Python packages rely on native code that
| you might find yourself brew-installing. That can be a
| nightmare.
| PaulHoule wrote:
| Python has always struggled with distribution maintainers
| that just don't get it.
|
| Python barely survived becoming the default scripting
| language on Red Hat and other Linux distros which was a
| major obstruction to the Python 3 transition. If the new
| cohort of pandas and scikit-learn users had not been such
| a force of nature we"d be talking today about Python the
| way we do about Perl.
|
| Not installing venv is a serious crime on the part of
| Debian as beginners don't need any more excuse to trash
| their system with site-local directories and other wrong
| answers for how to manage packages in Python.
| dima55 wrote:
| What's the crime in Debian? The "right answer about how
| to manage packages" is "apt install package". Anything
| else (including venvs) is a hack
| PaulHoule wrote:
| The word package is overloaded. There is a Debian package
| and there is a Python package.
|
| If somebody wanted to package a modern Python application
| as Debian package they'd have to make a Debian package
| that contains several Python packages maybe as a venv or
| executable wheel, it is a solvable problem but a bit like
| comparing tic tac toe to 12d chess with supersymmetry in
| terms of the ambition of Linux distros if not the
| difficulty.
| tmpz22 wrote:
| People just want to get things done. If they can get away
| with two commands instead of six they will, which is why
| I think poetry has gotten more adoption.
| antod wrote:
| _> Isn't it considered best practice not to use brew to
| install Python for exactly this reason?_
|
| The only time I've ever had a big mess with broken Python
| installs was after using brew on Linux - luckily killing
| off brew brought the system Python back fine. I'll
| grudgingly put up with brew on a Mac out of necessity,
| but keep it away from Linux.
| crabbone wrote:
| > but aren't Virtual Environments something you setup once
| per project?
|
| My typical count is 2-3 per project per machine I work on
| (could be anywhere from one to one hundred). Then there's a
| different number of people who need to set up these
| environments on different machines too (and sometimes require
| my support).
|
| So, the answer is: who knows?
|
| > putting everything in a docker container
|
| I think, you meant image, not container, but that's an easy
| mistake to make. And the answer is: both options are
| terrible. It's a choose your poison kind of thing.
|
| > cool kids
|
| My impression from working with the cool kids is that our
| industry selects for particular taste in t-shirts rather than
| knowledge or experience. I'm afraid that the correlation
| might swing into the negative, if we take either knowledge or
| experience vs coolness.
|
| ---
|
| Most importantly: venv is not the problem. It's a bad fix for
| a problem. Bad as in it doesn't fix the problem actually, it
| pretends to. I mean, maybe it covers some 90% of the problem
| -- who knows, I didn't count. So, it kinda works for a lot of
| people. But, honestly, I'd prefer that the problem was fixed
| s.t. it doesn't require venv. It's kind of like discussing
| various dental prosthetics options: it's better to just have
| healthy teeth.
| [deleted]
| _coveredInBees wrote:
| I'm surprised at the number of people here complaining about
| venvs in Python. There are lots of warts when it comes to package
| management in Python, but the built-in venv support has been rock
| solid in Python 3 for a long time now.
|
| Most of the complaints here ironically are from people using a
| bunch of tooling in lieu of, or as a replacement for vanilla
| python venvs and then hitting issues associated with those tools.
|
| We've been using vanilla python venvs across our company for many
| years now, and in all our CI/CD pipelines and have had zero
| issues on the venv side of things. And this is while using
| libraries like numpy, scipy, torch/torchvision, etc.
| black3r wrote:
| > Most of the complaints here ironically are from people using
| a bunch of tooling in lieu of, or as a replacement for vanilla
| python venvs and then hitting issues associated with those
| tools.
|
| That's because the vanilla python venvs feel like a genius idea
| but not thought out thoroughly, they feel as if there's
| something missing..., So there's naturally lots of attempts at
| improvements and people jump at those...
|
| And when you think about it in bigger depth, venvs are
| themselves just another one of the solutions used to fix the
| horrible mess that is python's management of packages and
| sys.path...
|
| The "Zen of Python" says "There should be one-- and preferably
| only one --obvious way to do it.", so I can't understand why
| it's nowhere near as easy when it comes to Python's package
| management...
| JohnFen wrote:
| Honestly, virtual environments are one of the reasons why I
| prefer to avoid Python whenever I can.
| Jackevansevo wrote:
| Likewise, I think people have a negative first experience
| because it doesn't work exactly like node, throw their toys out
| the pram and complain on HN for the rest of time.
|
| Guess in taking this stance we're both part of the problem...
| \s
| hot_gril wrote:
| I've never used anything but vanilla Python venvs, and no they
| don't work reliably. What does is a Docker container. I keep
| hearing excuses for it, but the prevalence of Dockerfiles in
| GitHub Python projects says it all. This is somehow way less of
| an issue in NodeJS, maybe because local environments were
| always the default way to install things.
| neuronexmachina wrote:
| > This is somehow way less of an issue in NodeJS, maybe
| because local environments were always the default way to
| install things.
|
| There's also NodeJS's ability for dependencies to
| simultaneously use conflicting sub-dependencies.
| hot_gril wrote:
| Yeah, you can't have two deps use different versions of the
| same sub-dep, cause they flatten everything instead of
| having a tree. In practice I rarely have issues with this
| except in React-Native, where it's a common problem, but
| then again RN is doing some crazy stuff to begin with.
| TheRealPomax wrote:
| Except when you try to move it, or copy it to a different
| location. This _almost_ made sense back when it was its own
| script, but it hasn't made sense for years, and the obstinacy
| to just sit down and fix this has been bafflingly remarkable.
|
| ("why not make everyone install their own venv and run pip
| install?" because, and here's the part that's going to blow
| your mind: _because they shouldn 't have to_. The vast majority
| of packages don't need compilation, you just put the files in
| the right libs dir, and done. Your import works. Checking this
| kind of thing into version control, or moving it across disks,
| etc. etc. _should be fine and expected_. Python yelling about
| dependencies that _do_ need to be (re)compile for your os
| /python combination should be the exception, not the baseline)
| 9dev wrote:
| Oh, yeah? It's working great? Like figuring out which packages
| your application actually uses? Or having separate development
| and production dependencies? Upgrading outdated libraries?
|
| Having taken a deep-dive into refactoring a large python app, I
| can confidently say that package management in python is a pain
| compared to other interpreted languages.
| peterhil wrote:
| I strongly agree with this, and I have been actively using
| Python since 2009.
|
| Trying top keep a Pygame/Numpy/Scipy project working has been
| a real struggle. I started it with Python 2 and ported to
| Python 3 some years ago. The whole Python 3 transition is a
| huge mess with every Python 3 point release breaking some
| things. No other interpreted language's packaging system is
| so fucked up.
|
| On a positive note: Lately I've liked using pdm instead of
| pip, and things seem to work quite a lot better. I evaluated
| Poetry, Flit and something else also.
|
| I just commented about this on Twitter, when someone asked
| "Which programming language do you consider beginner's
| friendly?"
| https://twitter.com/peterhil/status/1633793218411126789
| rowanseymour wrote:
| Virtual environments aren't package management. For example
| we use Poetry for package management - it supports separate
| dev and prod dependencies, upgrading etc. It _generates_ a
| virtual environment.
| 9dev wrote:
| The distinction feels entirely academic to me. Managing
| packages means having a sane way to define the dependencies
| of software projects, so they can be checked into version
| control and be installed reproducibly later and/or
| elsewhere.
|
| I don't know which problem python intended to solve by
| separating the two, but it doesn't occur often in
| contemporary software engineering work.
|
| Having said that, the point you make is valid and Poetry is
| a good option, but it feels so maddening having to learn
| about like seven different tools which all do more or less
| the same but not quite, and everyone and their mother
| having an opinion on which is the best. Doesn't help that
| there's an arbitrary yet blurry line where package managers
| end and environment managers begin.
| winrid wrote:
| Because even with --copy it creates all kinds of symlinks, and
| if you're using pyenv, hard coded paths to the python binary
| which can break from CI to installation.
|
| If you're using docker then it's a lot easier I guess.
| mikepurvis wrote:
| It also quietly reuses the stdlib of whatever python you
| start from. Which _mostly_ doesn't matter in real world
| usage, but can be quite surprising if you ever get into your
| head the idea that that venv is portable.
| whalesalad wrote:
| I've been using Python since like 2006, so maybe I just have
| that generational knowledge and battlefront experience... but
| whenever I come into threads like this I really feel like an
| imposter or a fish out of water. Like, am I using the same
| Python that everyone else is using? I echo your stance - the
| less overhead and additional tooling the better. A simple
| requirements.txt file and pip is all I need.
| coldtea wrote:
| Is it "generational knowledge and battlefront experience" or
| just "getting used to the (shitty) way things have always
| been" and Stockhold Syndrome?
| anongraddebt wrote:
| Twice bricking my laptop's ability to do python development
| because of venv + symlink bs was the catalyst I needed to go
| all-in on remote dev environments.
|
| I don't drive python daily, but my other projects thank
| Python for that.
| crabbone wrote:
| Lol. You put "simple" and "requirements.txt" unironically
| next to each other...
|
| I mean, I think you genuinely believe that what you suggest
| is simple... so, I won't pretend to not understand how you
| might think that. I'll explain:
|
| There's simplicity in performing and simplicity of
| understanding the process. It's simple to make more humans,
| it's very hard to understand how humans work. When you think
| about using pip with requirements.txt you are doing the
| simple to perform part, but you have no idea what stands
| behind that.
|
| Unfortunately for you, what stands behind that is ugly and
| not at all simple. Well, you may say that sometimes it's
| necessary... but, in this case it's not. It's a product of
| multiple subsequent failures of people working on this
| system. Series of mistakes, misunderstandings, bad designs
| which set in motions processes that in retrospect became
| impossible to revert.
|
| There aren't good ways to use Python, but even with what we
| have today, pip + requirements.txt is not anywhere near the
| best you can do, if you want simplicity. Do you want to know
| what's actually simple? Here:
|
| Store links to Wheels of your dependencies in a file. You can
| even call it requirements.txt if you so want. Use curl or
| equivalent to download those wheels and extract them into
| what Python calls "platlib" (finding it is left as an
| exercise for the reader) removing everything in scripts and
| data catalogues. If you feel adventurous, you can put scripts
| into the same directory where Python binary is installed, but
| I wouldn't do that if I were you.
|
| Years of being in infra roles taught me that this is the most
| reliable way to have nightly builds running quietly and
| avoiding various "infra failures" due to how poorly Python
| infra tools behave.
| aflag wrote:
| It's incredibly lacking in features. PyPI doesn't even properly
| index packages, making pip go into this dependency resolution
| he'll trying to find a set of versions that will work for you.
| It works for simple cases with few dependencies/not a lot of
| pinning. But if your needs are a bit more complex it certainly
| shows its rough edges.
|
| I actually find it amazing that they python community puts up
| with that. But I suppose fixing it is not that pressing now the
| language is widely adopted. It's not going to be anyone's
| priority to mess with that. It's high risk low rewards sort of
| project.
| Groxx wrote:
| What does that have to do with venvs?
|
| I agree the _packaging and distribution_ setup in python is
| an absolute mess, but that 's entirely unrelated to venvs.
| It's like bringing up how python uses whitespace instead of
| curly-braces.
| oneepic wrote:
| I think the GP comment might have caused some confusion
| since it mentioned both package management and venvs very
| close together.
| [deleted]
| whalesalad wrote:
| I've been writing Python for a looong time. I have pushed out
| thousands and thousands of deployments across probably 40+
| distinct Python codebases and only once or twice have I ever
| encountered a showstopper dependency resolution issue. At the
| end of the day you should _want_ to have fine grained control
| over your deps and frankly there are many times where a
| decision cannot be automatically made by a package manager.
| Pip gets beat on _hard_ but it puts in work all day every day
| and rarely skips a beat. It 's entirely free and developed
| with open source contributions.
|
| Areas where I _have_ felt a lot of pain is with legacy Ruby
| projects /bundler. Don't get me started on golang.
|
| Can pip be made better? Sure. Should we have an attitude of
| disgust towards it? Heck no!
| rxhernandez wrote:
| > only once or twice have I ever encountered a showstopper
| dependency resolution issue
|
| I've encountered them with other languages and they're the
| sort of thing where one time is more than enough to make me
| feel like it could get me fired; they're Never (with a
| capital N) okay imo
| crabbone wrote:
| > once or twice have I ever encountered a showstopper
| dependency resolution issue.
|
| Hahaha... (rolls on the floor) Do you want to know why that
| is? No seriously? I'm not laughing at you as much as I'm
| laughing at Python now, but hey, well, anyways, do you want
| to know why that happened to you? I know you don't. But
| I'll tell you anyways!
|
| Until quite recently, pip didn't give a rat's ass if the
| dependencies it installed were consistent. It would blink a
| message in the long stream of vomit it spills on the screen
| saying something like "you have package X installed of
| version Y, but package Z wants X of version Q, which will
| not be installed". And happily streamed more garbage to
| your screen.
|
| It was an issue that was filed against pip for something
| like 12 years until it got resolved about a year or so ago.
| Even after it got resolved a lot of people tried to
| upgrade, saw that that would "break" their deployment, and
| rolled back to the latest broken version.
|
| Things are sort of improving gradually since then, but we
| are light years away from the system working properly, and
| I know you don't want to know why, but I'll tell you
| anyways!
|
| So, when for whatever reason pip doesn't find a dependency
| it thinks you need, a lot of packages, when they roll out
| their "releases", they upload also what Python calls
| "source release". Which should have never been treated as
| an installation option, but it is, and is treated like that
| by default. So, what will happen once pip finally gives up
| on finding a match, right, you guessed it! -- It's going to
| try to build it! Installing build dependencies along the
| way. What you get in the end is anyone's guess, but most
| likely, it's something broken because the developers who
| made this release didn't make a release specifically for
| your version.
|
| Don't despair. There's a flag you can use with pip install
| that should prevent it from trying to build stuff. But two
| bad things will happen to you if you use it: in any non-
| trivial project your dependencies will irreparably break.
| And, who knows if that flag is implemented correctly...
| nobody in the real world is using that. So, who knows,
| maybe it'll format your hard drive along the way.
| muhokutan wrote:
| It behaves like a kid you send to the store with a 100
| dollars
| kapilvt wrote:
| I moved to poetry (ergonomics) and publishing wheels with
| frozen requirements, at least for apps, here's the plugin
| I used https://github.com/cloud-custodian/poetry-plugin-
| freeze .. readme has details, tldr freeze the graph for
| repeatability regardless of tool, Ala pip install works
| years later.
| crabbone wrote:
| I hate PyPI probably even more than you do, but venv doesn't
| do that. All it does is write a handful of files and make a
| bunch of symlinks. It doesn't deal with installation of
| packages.
| crabbone wrote:
| The most important part about venv is that you shouldn't need
| it. The very fact that it exists is a problem. It is a wrong
| fix to a problem that was left unfixed because of it.
|
| The problem is fundamental in Python in that its runtime
| doesn't have a concept of a program or a library or a module
| (not to be confused with Python's modules, which is a special
| built-in type) etc. The only thing that exists in Python is a
| "Python system", i.e. an installation of Python with some
| packages.
|
| Python systems aren't built to be shared between programs
| (especially so because it's undefined what a program is in
| Python), but, by any plausible definition of a program, venv
| doesn't help to solve the problem. This is also amplified by a
| bunch of tools that simply ignore venvs existence.
|
| Here are some obvious problems venv doesn't even pretend to
| solve:
|
| * A Python native module linking with shared objects outside of
| Python's lib subtree. Most comically, you can accidentally link
| a python module in one installation of Python with Python from
| a "wrong" location (and also a wrong version). And then wonder
| how it works on your computer in your virtual environment, but
| not on the server.
|
| * venvs provides no compile-time isolation. If you are building
| native Python modules, you are going to use system-wide
| installed headers, and pray that your system headers are
| compatible with the version of Python that's going to load your
| native modules.
|
| * venv doesn't address PYTHONPATH or any "tricks" various
| popular libraries (s.a. pytest and setuptools) like to play
| with the path where Python searches for loadable code. So much
| so that people using these tools often use them contrary to how
| they should be used (probably in most cases that's what
| happens). Ironically, often even the authors of the tools don't
| understand the adverse effects of how the majority is using
| their tools in combination with venv.
|
| * It's become a fashion to use venv when distributing Python
| programs (eg. there are tools that help you build DEB or RPM
| packages that rely on venv) and of course, a lot of bad things
| happen because of that. But, really, like I said before: it's
| not because of venv, it's because venv is the wrong fix for the
| actual problem. The problem nobody in Python community is bold
| enough to address.
| cmcconomy wrote:
| My personal approach is:
|
| - use miniconda ONLY to create a folder structure to store
| packages and to specify a version of python (3.10 for example)
|
| - use jazzband/pip-tools' "pip-compile" to create a frozen/pinned
| manifest for all my dependencies
|
| - use pip install to actually install libraries (keeping things
| stock standard here)
|
| - wrap all the above in a Makefile so I am spared remembering all
| the esoteric commands I need to pull this all together
|
| in practice, this means once I have a project together I am:
|
| - activating a conda environment
|
| - occasionally using 'make update' from to invoke pip-compile
| (adding new libraries or upgrading), and
|
| - otherwise using 'make install' to install a known working
| dependency list.
| nose-wuzzy-pad wrote:
| This seems simplistic and low drag. Do you have an example you
| can share?
|
| Thanks!
| its_over_ wrote:
| I use poetry or docker or nixpkgs
|
| I've given up.
|
| EDIT: also just finding myself reaching for go in most cases
| 89vision wrote:
| I haven't used these since docker
| foooobaba wrote:
| With docker, do you use debugging in pycharm/vscode, or just
| for compiling/shipping?
| 89vision wrote:
| Both. Setting up the editor took a little doing, but it works
| well.
| https://code.visualstudio.com/docs/containers/quickstart-
| pyt...
| ginko wrote:
| That's just giving up.
| epgui wrote:
| But it's giving up correctly*! :)
| rad_gruchalski wrote:
| That's being pragmatic.
| chao- wrote:
| Yes it is giving up, but not only. It is giving up _and_
| being able to get back to the actual work you want to be
| doing.
| PaulHoule wrote:
| I worked at a place where people tried using Docker to
| manage Python and wound up with a bunch of twisty little
| images that were all misconfigured in a different way.
| (E.g. default charsets that I'm not sure anybody uses.)
| cpach wrote:
| Learning to make sound container images is tricky, but
| once you got it figured out it is, IMO, a very nice way
| to distribute software.
| frgtpsswrdlame wrote:
| Sometimes that's the best option.
| Havoc wrote:
| These days I'm just throwing each project into a fresh LXC on a
| server.
|
| All these different languages have their own approach and each
| then also user/global/multiple versions...it's just not worth
| figuring out
| bagels wrote:
| Same, but with Docker. I don't like conda, was fine with
| virtualenv, but using docker there's only one python and you
| can just pip install and not worry about multiple environments.
| spyremeown wrote:
| Question: what makes you choose LXC over Docker?
| wyufro wrote:
| IMO, while their use cases do overlap, LXC is more geared
| towards a user installing things while Docker is more geared
| towards a developer creating a ready to use package.
|
| LXC creates environments, while Docker creates apps, is
| another way to say it.
| Havoc wrote:
| Much of a sameness really but I prefer the more persistent
| disk style of lxc plus ssh plus vscode ssh remote extension.
|
| Depends on task though I've got dockers and VMs in use too
| Supermancho wrote:
| This writeup needs work.
|
| > So while you could install everything into the same directory
| as your own code (which you did, and thus didn't use src
| directory layouts for simplicity), there wasn't a way to install
| different wheels for each Python interpreter you had on your
| machine so you could have multiple environments per project (I'm
| glossing over the fact that back in my the day you also didn't
| have wheels or editable installs).
|
| This is a single run-on sentence. Someone reading this, probably
| doesn't know what "wheels" means. If you are going to discount it
| anyway, why bring it up?
|
| > Enter virtual environments. Suddenly you had a way to install
| projects as a group that was tied to a specific Python
| interpreter
|
| I thought we were talking about dependencies? So is it just the
| interpreter or both or is there a typo?
|
| > conda environments
|
| I have no idea what those are. Do I care? Since the author is
| making a subtle distinction, reading about them might get me
| confused, so I've encountered another thing to skip over.
|
| > As a running example, I'm going to assume you ran the command
| py -m venv --without-pip .venv in some directory on a Unix-based
| OS (you can substitute py with whatever Python interpreter you
| want
|
| Wat? I don't know what venvs are. Can you maybe expand without
| throwing multi-arg commands at me? Maybe add this as a reference
| note, rather than inlining it into the information. Another thing
| to skip over.
|
| > For simplicity I'm going to focus on the Unix case and not
| cover Windows in depth.
|
| Don't cover Windows at all. Make a promise to maintain a separate
| doc in the future and get this one right first.
|
| > (i.e. within .venv):
|
| This is where you start. A virtual environment is a directory,
| with a purpose, which is baked into the ecosystem. Layout the
| purpose. Map the structure to those purposes. Dive into
| exceptional cases. Talk about how to create it and use it in a
| project. Talk about integrations and how these help speed up
| development.
|
| I also skipped the plug for the mircoenv project, at the end with
| a reference to VSCode.
| ianbutler wrote:
| I expect most everyday python users know what these things are.
| I also expect this was targeted at python users who use these
| things but haven't thought deeply about them.
|
| Charitably, I will assume you are a non python user, and that's
| why this is a miss for you.
| hkgjjgjfjfjfjf wrote:
| [dead]
| warner25 wrote:
| > One point I would like to make is how virtual environments are
| designed to be disposable and not relocatable.
|
| Is the author saying that relocating them will actually break
| things, or that it's just as easy to recreate them in a different
| location? Because I've moved my venv directories and everything
| still seemed to work OK. Did I just get lucky?
| Helmut10001 wrote:
| I have my whole conda env folder symlinked to my second drive.
| Impossible to store 120GB of environments otherwise.
| Falell wrote:
| Relocating them will actually break things in many cases,
| especially when native code is involved.
| korijn wrote:
| Depends if any of your packages use absolute paths (generated
| at install time for example).
| JamesonNetworks wrote:
| I've had problems with symlinked python bins not existing in
| the same place requiring relinking as one example of a problem
| noisenotsignal wrote:
| There's also relocating across machines. For example, maybe
| your build environment has access to internal registries but
| your release environment does not. I naively thought you could
| build your venv and just copy to the new machine (both
| environments were Ubuntu) but ran into errors (due to links
| breaking). We also used pex for a bit, which is kind of like
| building a binary of a venv, and _that_ eventually stopped
| working too when the C ABI was no longer the same between
| environments. There didn't seem to be an easy way to pick the
| ABI version to target when creating the pex file, so I gave up
| and just downloaded the wheels for internal packages in the
| build.
| Izkata wrote:
| By default when you activate a virtualenv, it uses hardcoded
| absolute paths (determined at the time the environ was
| created), so moving the directory will break it.
| acomjean wrote:
| I've tried to move things and broken everything. (Conda
| environments). I tried replacing the paths in the files and it
| didn't work. We run a bunch of different tools with various
| python requirements and would like to be able to duplicate them
| for the next tool.
|
| We ended up making a new environments for each. Honestly it's a
| bit of a mess.
| senko wrote:
| > relocating them will actually break things
|
| Yes, absolute paths are hardcoded in several places.
|
| I actually have a use case for copying/relocating them (for
| https://apibakery.com), and simple search/replace of paths
| across the entire venv works, but I wouldn't recommend that as
| a best practice approach :-)
| jszymborski wrote:
| It's a gamble to move venvs.
|
| The real way to move venvs is to freeze the venv (i.e. make a
| requirements.txt) and then pip -r requirements.txt to recreate
| the venv.
|
| This process is really the only thing about venvs that ever
| causes me trouble.
| asicsp wrote:
| See also: Virtual Environments Demystified
| (https://meribold.org/python/2018/02/13/virtual-environments-...)
|
| Discussion from 2021:
| https://news.ycombinator.com/item?id=25611307
| rekahrv wrote:
| That's insightful.
|
| It seems that a virtual environment created by Poetry looks very
| similar, except that it doesn't contain an `include` directory.
| It contains:
|
| * `bin` directory
|
| * `lib/<python-version>/site-packages/` directory
|
| * `pyvenv.cfg`
| tomalaci wrote:
| I would highly recommend Poetry for python package management. It
| basically wraps around pip and venvs offering a lot of
| convenience features (managing packages, do dist builds, etc.).
| It also works pretty nicely with Tox.
|
| I would recommend using virualenvs.in-project setting so Poetry
| generates venv in the project folder and not in some temporary
| user folder.
| nerdponx wrote:
| I prefer Hatch over Poetry. I don't have any strong reason for
| that preference, I've just use both and I feel more comfortable
| with Hatch. It feels a little more seamlessly integrated with
| other Python tools, and I appreciate the developers'
| conservative approach to adding features.
| davidktr wrote:
| 100% this. I've always struggled with creating packages, but
| now simply do poetry init and I am done. Magic.
| winrid wrote:
| Thanks. I recently spent a whole afternoon learning how to
| package a new python project. Was really surprised at the
| difficulty even with venv, compared to node and java.
| peterhil wrote:
| I just compared and evaluated Hatch, Flit, Poetry and Pdm and
| found Pdm to be most robust and slimmest. Hatch was a good
| second option, and Poetry and Hatch are easy to use, but have
| too much bloat and magic.
___________________________________________________________________
(page generated 2023-03-13 23:01 UTC)