[HN Gopher] Super Colliding Nix Stores: Nix Flakes for Millions ...
___________________________________________________________________
Super Colliding Nix Stores: Nix Flakes for Millions of Developers
Author : osener
Score : 50 points
Date : 2023-05-25 18:00 UTC (5 hours ago)
(HTM) web link (blog.replit.com)
(TXT) w3m dump (blog.replit.com)
| peter_l_downs wrote:
| I think the best part about Nix is that it makes it super easy to
| get software into your development environment. This layered
| store is a pretty cool setup to make that efficient for Replit.
|
| Not sure how if Nix/Replit has the program you're looking for? Or
| what package name to use to actually get the program you want?
| Check out my cli search tool:
|
| https://github.com/peterldowns/nix-search-cli
|
| Edit: I can't help myself from ranting, it's insane that I had to
| write this tool. Speaks to the lack of user focus in the Nix
| community. Even the blog post here doesn't mention how to
| actually find packages to install!
|
| What good is a package manager that makes finding packages
| difficult? Who should be expected to tolerate such a thing?
| otabdeveloper4 wrote:
| > Speaks to the lack of user focus in the Nix community.
|
| Not really. Nix is not a tool, it's a toolkit for making your
| own tools. Very much in the vein of Unix, shell, etc.
|
| Many people don't use Nix for "installing" stuff. I personally
| use it for deploying software to servers without the
| (congnitive and performance) overhead of containers.
|
| I find search.nixos.org immensely useful, but don't really need
| it from the CLI.
| sterlind wrote:
| _> Not really. Nix is not a tool, it 's a toolkit for making
| your own tools. Very much in the vein of Unix, shell, etc_
|
| even tools for tools should be ergonomic to use. the Nix
| language server should have autocomplete, searches and
| interacting with the language should be easy, etc. and nearly
| everyone needs to hunt for packages, even if they're not
| "installing" them - they might be needed for the toolchain.
|
| Nix is like early Git. all the raw power is there, but little
| of the porcelain. it's so much better than anything before it
| that we accept the pain of dealing with exposed plumbing, but
| we shouldn't get used to it.
| ghuntley wrote:
| The last two years have seen an increased focus on
| usability, education and polish now that after circa 15
| years of establishing solid foundations are in place. All
| good things in time.
|
| There's no venture backed cash here paying for the polish
| that folks typically see in devtools launched in 2023. That
| polish is expensive to do and isn't "fun or engaging" for
| the majority of developers thus classic open source problem
| of over indexing on serving the needs/existing people who
| do not need the polish.
|
| There's some notable call-outs here of folks putting in
| serious effort such as https://nix.dev ala
| https://cachix.org out of love for nix to succeed.
|
| Yeah nix wants to grow, and it is, largest bazaar GitHub
| community/repo out there right now.
|
| starting to see venture backed companies forming who are
| building the polish as porcelain over the top and giving
| back. see https://floxdev.com/
|
| at this stage nix is a damn safe bet because of what it can
| do, the problems it solves and the size of community.
| pxc wrote:
| > at this stage nix is a damn safe bet because of what it
| can do, the problems it solves and the size of community.
|
| In the last few years, Nix has gone from a place where I
| was sometimes uncomfortable recommending it to people to
| one where I feel like it would be unstrategic not to use
| it where it fits. If you know Nix today, you should be
| pushing it at your company wherever you can see a good
| use case.
| adarsh-krishna wrote:
| Ended up building a tool which indexes every version of every
| package into a searchable tool. Was hacked together in a
| weekend, so the UI needs some polishing, but it's functional :)
|
| https://nixdex.dev
| ghuntley wrote:
| or https://search.nixos.org
|
| if folks want a easy (and good way) to get up and running in
| the nix ecosystem w/services ala replace docker compose then
| https://devenv.sh is the shizz
| sterlind wrote:
| why does it query search.nixos.org, rather than looking through
| your local nixpkgs? just because of how tricky it'd be to
| handle the nix language or hacking on the nix toolchain proper?
|
| also I agree that it's ridiculous nix doesn't have a search
| command. it baffles me.
| ghuntley wrote:
| search.nixos.org also includes flakes that people PR into the
| index via GitHub at https://github.com/NixOS/nixos-
| search/issues
|
| ps: $ nix search exists via experimental flags
| https://nixos.org/manual/nix/stable/command-ref/new-
| cli/nix3...
| lilyball wrote:
| It does though. `nix search`.
| aidenn0 wrote:
| Nix has several search commands, it's just none of them do
| quite what people want (e.g. "nix search" with flakes and
| "nix-query" without flakes search by name and description,
| "nix-locate" can search by contents but the interface is a
| bit obtuse)
|
| For those who want a local search for executables, and don't
| mind running nix-index: #!/bin/sh
| results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")"
| for item in $results; do nix search
| "nixpkgs#${item%.*}" done
|
| will search for a specific executable and give you
| (hopefully) reasonable output. If you don't have flakes
| enabled, you can just print the result of the command-
| substitution directly, but you won't get the short
| description.
| peter_l_downs wrote:
| This exists and searches `search.nixos.org` for a few
| reasons:
|
| - you may want to search without downloading and indexing
| nixpkgs and the hydra results yourself
|
| - you may want to search by the name of the binaries/programs
| that will be installed, which is impossible to do by looking
| through a local package store alone
|
| - you may want the search to be fast
|
| - you may want to search from the command line
|
| `nix search` is a cruel joke which doesn't allow searching by
| the name/program that would be installed, only package name,
| and requires a flake name every time. Absolutely terrible
| interface.
|
| My tool is a single-install binary that performs fast and
| accurate search to help you find the right package name to
| install a given binary. I don't understand how after years of
| using other package managers anyone could want a search tool
| that does anything other than this by default.
|
| For more details on why this exists, check out
| https://github.com/peterldowns/nix-search-cli#motivation
| jljljl wrote:
| Finding a specific version of a package is also really
| difficult. It requires you to find the commit hash in the
| Nixpkgs repo that last contained it, and there isn't a central
| search engine that lets you map version to commit.
|
| We ended up building package version search into the Devbox CLI
| for this reason (https://www.jetpack.io/devbox/docs/guides/pinn
| ing_packages/#...). There's also a 3rd party site that lets you
| search for Nix package versions (https://lazamar.co.uk/nix-
| versions/)
| peter_l_downs wrote:
| I'll have to check out devbox, thanks for the link. I don't
| understand why Nix doesn't have some way for derivations to
| claim that they install certain artifacts, which would make
| searching so much easier.
|
| The version thing is also really annoying, thanks for the
| helpful link, I hadn't seen that. Not sure how this will work
| in a Flakes world. Seems weird to me that sometimes packages
| are important enough to have multiple derivations each
| installing a different version (like the pythonFull packages)
| but there is no standard/good/default solution for it.
| jljljl wrote:
| Versioning is a bit easier with Flakes because they're
| generally published in their own Git repository, meaning
| you can use tags or branch refs to publish/install specific
| versions of a package. You can usually do something like
| `github:org/repo/ref#package` to get what you need.
| peter_l_downs wrote:
| Yes, but how will this work for the `nixpkgs` repository?
| Still very unclear!
|
| I'm 100% onboard with flakes in general, and use flakes
| to describe devshells + publish binaries when applicable
| -- like with the tool I linked in my earlier post.
| NathanOsullivan wrote:
| Normally I would eye-roll at these kind of self promotions
| but it looks like you are trying to help with a real pain
| point of Nix.
|
| I put a good amount of time getting to grips with "raw" nix
| with the I imagine common yo-yo-ing between "I don't get it"
| and "oh I see, this is great" but when I realised how the
| intersection of nixpkgs and package versioning actually
| worked.. I was done.
|
| For a tool that from the outside seems is so heavily focused
| on immutability to just continually throw away old versions
| in nixpkgs head is a head scratcher, and as a monorepo isn't
| a great fit for utilising different revisions for different
| packages either.
|
| I can only guess that due to single repo containing every
| package it wasn't seen as practical to just continually
| append versions to, but when the diff log is just full of
| 'delete version X URL and it's hash, add X+1 and new hash'
| from the outside at least, it felt like a real missed
| opportunity.
| viraptor wrote:
| I would suggest one more step of "oh, this is great": I
| don't think people need to care as much about finding the
| right version in the repo as they do. (Unless we're talking
| about something really significant like finding old MySQL
| 5.6) For smaller apps you can either override or copy the
| current nix file and update the version and hash - you get
| the version you want. It will use new version of the
| dependencies, but normally that's just fine.
|
| I've got 3 packages which are pinned to a specific version
| that way in home-manager and I'm happy. It's not an
| approach for the first time user of course.
| jljljl wrote:
| I thought this too, but we got a lot of requests for
| better version search + pinning. Adding overrides and
| looking up the right commit hash is a pretty cumbersome
| for a first time user.
|
| For new projects though, I agree that using `latest` is
| generally the way to go.
| jljljl wrote:
| I do think Flakes help with the "throwing away old
| versions" issue. They also help with the monorepo issue by
| encouraging developers publish the Nix derivation in their
| own repo.
|
| But they currently have a bit of a discoverability issue,
| and they need adoption by legacy packages to completely fix
| the problem.
| woleium wrote:
| But oh my, the security implications. Nix is great, but there's
| no chain of trust on where these packages come from. Did we learn
| nothing from 'left_pad'?
| tomberek wrote:
| Other than the very explicit tracking of the packages, their
| dependencies, the full build instructions, the public history,
| cryptographic signatures for the pre-built binaries, and the
| trivial ability for anyone to re-build from source and audit
| the entire chain all the way from bootstrap if they wished?
|
| I'm not sure what golden standard we are comparing this to. It
| is not perfect, but I'd say this is a far more solid bedrock
| upon which to build software than anything else I've
| encountered.
| smasher164 wrote:
| I love Nix as a package manager, and flakes are awesome for
| setting up environments. I wish it had more affordances for being
| a proper build system. If only there were a Buck2 and Nix hybrid.
| pxc wrote:
| bob.build looks kinda nice as a Nix-based, make-like tool, but
| IDK how it compares to the real, fancy, big tech monorepo build
| tools because I've never used them.
|
| https://bob.build/
| chills wrote:
| > If only there were a Buck2 and Nix hybrid
|
| You'll be interested in
| https://github.com/thoughtpolice/buck2-nix
| ghuntley wrote:
| would love it if the compilation closure was more incremental
| rather than at a derivation level. specifically caching sub
| modules. many folks working on improving this space.
|
| last week benchmarks from tvnix (a rust rewrite of the nix cli)
| was published which saw a THIRTEEN second evaluation speed up
| on a basic derivation evaluation.
|
| https://twitter.com/matthewcroughan/status/16606138356553482...
| pxc wrote:
| I saw that tweet and thought it was pretty exciting!
|
| How much of Nixpkgs can Tvix build these days?
| Stammon wrote:
| So do they share their implementation of overlaying stores
| somewhere? It would be a huge benefit for the community to have
| it open source and available for being used in countless other
| use cases.
| bb010g wrote:
| > In the coming weeks, we will be developing the Layered Store
| features, releasing a Nix Community RFC, and working to
| upstream it into Nix.
|
| Doesn't look like specs or implementations are public yet.
___________________________________________________________________
(page generated 2023-05-25 23:00 UTC)