[HN Gopher] Nixhub: Search Historical Versions of Nix Packages
___________________________________________________________________
Nixhub: Search Historical Versions of Nix Packages
Author : clessg
Score : 36 points
Date : 2023-07-20 19:44 UTC (3 hours ago)
(HTM) web link (www.jetpack.io)
(TXT) w3m dump (www.jetpack.io)
| moojd wrote:
| Oh man I've needed this for a while to the point where I was
| about to build it my self so thanks! It would be great though if
| it also listed the tagged nixpkgs version in addition to the hash
| if applicable.
| noahmasur wrote:
| Roughly equivalent tool that I've used in the past:
| https://lazamar.co.uk/nix-versions/
| jljljl wrote:
| Hey! Author of this post here, happy to answer any questions
| about Nixhub!
| nathan_phoenix wrote:
| First off, amazing tool and thanks for doing the nix community
| a favor!
|
| If you could expand more on the architecture of the tool I'd be
| very curious! Like what goes all on from parsing Hydra builds
| to getting a useful package versions DB and what were the
| challenges while building it? Also, how are you merging all the
| nodejs packages (nodejs_20, nodejs_18, ...) into one "nodejs"?
|
| I'm also in the process of building something very similar
| (didn't know the idea was that popular haha), but I'm not
| parsing Hydra builds but the output from `nix-env -qa --json`
| to get all the package versions and it's proving more involved
| than I anticipated (:
| jljljl wrote:
| Thank you!
|
| We're planning a longer post that explains the architecture
| and process of building the index. An extremely basic
| description of what we do can be found here:
| https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-
| ve... First, we processed build outputs from
| Hydra to map package names + version numbers to commits in
| the Nixpkgs repo. We then created a service that stores this
| index as a prefix tree, making it easy to search for a
| specific package and version in Nix.
|
| There's some extra cleanup required to make the index really
| usable, which we'll share in the follow up post
|
| For the merged packages -- we have a hardcoded "canonical
| names", which map the various attribute paths like
| `nodejs_20` and `nodejs_18` to a single, easier to search
| name (like `nodejs`). We originally designed this so that
| Devbox users (who may not be familiar with Nix naming
| conventions) can just type `devbox add nodejs@20` and get the
| right package version.
| nathan_phoenix wrote:
| Thanks for the answer! Looking forward to the followup post
| :D
|
| And yeah, nix naming conventions are sometimes a mess. Your
| approach does seem much nicer.
|
| Last question if you have a bit more time. How come that
| you don't have every version of a package available? Is it
| because you only parse the Hydra stable 23 builds (which I
| guess aren't updated on every version) or because you parse
| them only periodically like Lazamar's Nix Package Versions?
| jljljl wrote:
| We're parsing periodically (though our index is more up
| to date than Nix Package Versions at the moment). We've
| also truncated pre-Flakes commits, though we are planning
| to include those in the future
| nathan_phoenix wrote:
| Looking forward to that for completeness sake!
|
| Btw, regarding a bit older pre-Flakes commits, at least
| some packages will error out while nix-installing on M1
| (or newer) macbooks because at that point nix packages
| didn't anticipate macbooks ever ditching x86 for aarch...
| Hope this info helps you out, at least took me some time
| to realize that the random obscure error was caused by
| this. Can be almost always fixed by appending "--system
| darwin-x86_64" to the nix command to use Apple's Rosetta
| binary translation, cheers.
| stabbles wrote:
| It's interesting how Nix's selling point is that its non-standard
| file system structure allows multiple versions/flavors of the
| same package to be installed at the same time, yet if I
| understand correctly, it often has one version per package per
| Nix version? It's more like install multiple versions of the same
| package at a _different_ time?
|
| Compare this to the model we have in Spack (which is inspired by
| nix), where there's _one_ recipe per package for _many_ versions,
| and you can define different build flavors, conditional
| dependencies (and conditional pretty much everything).
|
| The difference is that Spack has a dependency solver and Nix
| apparently does not?
|
| For example the curl package [1] has tons of versions, and its
| `tls` package variant has conditional values depending on the
| version: variant("tls", values=("gnutls",
| conditional("mbedtls", when="@7.46:"), ...), multi=True)
|
| And dependencies are specified as
| depends_on("gnutls", when="tls=gnutls") with
| when("tls=mbedtls"): depends_on("mbedtls@2:",
| when="@7.79:") depends_on("mbedtls@:2", when="@:7.78")
|
| This allows you both to install all sorts of curls, like `spack
| install curl tls=mbedtls` (gives you something >= 7.46), or
| `spack install curl@7.77 tls=mbedtls` gives you a curl with an
| old mbedtls, etc.
|
| [1]
| https://github.com/spack/spack/blob/develop/var/spack/repos/...
| cayley_graph wrote:
| Probably because it's pretty easy to just use an older package
| from a different version of Nixpkgs (just use an override that
| defines the attribute for a package in your current Nixpkgs as
| the same package in an older Nixpkgs). It'll transparently pull
| the older version from the binary cache, too, if available.
|
| The only difficulty is finding the right Nixpkgs commit hash,
| which this solves. I think I find the Nix way to be a bit
| cleaner, though I'm unaware of the full set of pros/cons.
|
| Worth noting that the Nix package definition for curl at [1] is
| a little easier to read for me since it doesn't have the
| duplicated code for each version.
|
| [1]
| https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/netw...
| nathan_phoenix wrote:
| > yet if I understand correctly, it often has one version per
| package per Nix version?
|
| Both yeah and no. Taking the example of nodejs, you have all
| the supported major releases like nodejs_16, nodejs_18, etc.
| for each Nix revision[1], but within each revision you only
| have a specific version of the package. E.g. nodejs_16 is
| 16.3.1 for revision at date X. There are benefits and drawbacks
| of managing a package repo like that, with revisions which
| implicitely pin all packages and it's dependencies at once, but
| it's too long of a topic to write about here.
|
| > For example the curl package [1] has tons of versions [...]
|
| So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3,
| etc)[2]. In that regard nix and Spack are similar.
|
| [1]: A revision is like a shapshot of all the packages at a
| given point in time.
|
| [2]:
| https://search.nixos.org/packages?channel=23.05&from=0&size=...
| stabbles wrote:
| The main difference is that Spack has a single definition of
| a package, and in Spack language `curlMinimal`,
| `curlWithGnuTls`, and `curlHTTP3` are all "concretizations"
| of curl. That has the advantage that dependents can say
| `depends_on("curl")` if they don't care about the flavor --
| only if they somehow _require_ curl with GNU tls, they say
| `depends_on( "curl tls=gnutls")`.
| nathan_phoenix wrote:
| Oh, didn't know, thanks for the explanation! Does seem like
| an interesting concept, will check out Speck more in-depth.
| aseipp wrote:
| Nix also has a single definition of the curl package. The
| curl package itself can take tons of flags as inputs,
| indicating how it should be built, and you can then have
| curlMinimal, curlWithGnuTLS, and curlHTTP3 all as different
| 'names' for curl with those arguments applied. A package
| just says it has a 'curl' input and any user can "call"
| that package, providing any flavor of 'curl' they want as
| input. You can override those inputs at a global level and
| local level in various ways.
|
| I suspect you mean some other semantic difference, but if
| that's all you mean, it's not different from Nix in
| practice, from what it sounds like?
|
| https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/net
| w...
| HL33tibCe7 wrote:
| Not sure if anyone has mentioned this yet, but "spack" is an
| ableist slur in Britain
| Cloudef wrote:
| This is what nix lets you do basically
| https://github.com/Cloudef/nixos-flake/blob/master/flake.nix...
| as well as this https://github.com/Cloudef/nixos-
| flake/blob/master/modules/d...
|
| Also here I'm referring to specific version of zig for this
| particular LSP's config file, but it never goes to my PATH
| https://github.com/Cloudef/nixos-flake/blob/master/modules/n...
| aseipp wrote:
| > It's interesting how Nix's selling point is that its non-
| standard file system structure allows multiple versions/flavors
| of the same package to be installed at the same time, yet if I
| understand correctly, it often has one version per package per
| Nix version?
|
| You're talking about Nixpkgs, and it's a cultural choice. In
| the past there used to be more of this, where e.g. the Haskell
| community had lots of variants of the same library all at once,
| because users and other libraries wanted them. But it became
| untenable to maintain and you in practice have to do tons of
| effort to vendor things. And once many of the subcommunities
| began moving to auto-generated packages (e.g. generating Nix
| packages from Haskell packages), it becomes much easier to have
| a single globally consistent repository of versions that all
| are known to work together, generated by the host-language
| dependency resolver, with minor fixes applied on demand.
|
| There are _some things_ that do have multiple versions in the
| main repository, e.g. LLVM. But in practice having like 10
| versions of everything is a huge chore and only done on demand
| in practice if users want it, and it has real overhead
| cognitively and in build time and in maintainer time.
|
| You can define your own Nix flake with every version of every
| Python library in it if you want. But in practice people sort
| of pick versions of things that make QA a real chore.
|
| EDIT: Also, for like 99% of cases, what users think is cool but
| hardly do is to run both pieces of software at the exact same
| time concurrently. They typically just want atomic upgrades and
| rollbacks. For example, you change your git revision at work
| and get whole new tools, your old tools still work if you go
| back to the old git revision at no loss. So the need to only
| have one version "globally" under one name isn't a big deal in
| practice, because people often _are_ only dealing with one
| global name. They just don 't want their packages to get
| corrupted during an upgrade or switchover or something
| otherwise breaks.
| hamandcheese wrote:
| For most people, depending on historic versions of nixpkgs is an
| anti-pattern. The main reason I say this is security. You may
| want an old version of, say, ruby, but you probably don't want
| the old, insecure libraries it was built against (i.e. glibc,
| openssl).
|
| It is often just as easy to use overrides to build a different
| version of a piece of software, and in the rare case that doesn't
| work you can fall back to a custom derivation (drawing
| inspiration from the old version of nixpkgs, perhaps).
|
| I worry about tools like this which make doing the wrong thing
| too easy.
___________________________________________________________________
(page generated 2023-07-20 23:01 UTC)