[HN Gopher] The Magic Nix Cache, a GitHub Action for speeding up...
___________________________________________________________________
The Magic Nix Cache, a GitHub Action for speeding up your Nix
workflows
Author : biggestlou
Score : 63 points
Date : 2023-06-26 19:32 UTC (3 hours ago)
(HTM) web link (determinate.systems)
(TXT) w3m dump (determinate.systems)
| [deleted]
| bketelsen wrote:
| this looks awesome, can't wait to try it out.
| grhmc wrote:
| One project cut their CI time from 18m to 3m:
| https://github.com/awkward-squad/hasql-interpolate/actions. I
| wonder who will see the biggest cut!
|
| Note that when PRs merge to the default branch, their cache
| doesn't come with them. This is how GitHub Action's cache works,
| as a security measure. However: subsequent rebuilds will, and PRs
| off the default branch will too.
| iso8859-1 wrote:
| How does the Magic Nix Cache know which parts of my build are
| deterministic and which aren't?
|
| I suppose maybe it will only work if I split my build up into
| multiple steps such that Nix will know to skip those first steps.
| If Nix knows that, I suppose the Magic Nix Cache also knows?
| biggestlou wrote:
| Could you possibly clarify what you mean by "skip those first
| steps?" Which steps are you referring to?
| iso8859-1 wrote:
| Presumably, the Magic Nix Cache speeds up your build by
| caching the output of the "first steps" (really leaves in a
| tree of inputs). Otherwise, how would it work? So by steps, I
| am referring to the steps that your Nix derivation consists
| of.
| biggestlou wrote:
| When Nix "realises" a derivation, it realises the entire
| dependency tree (here, "realises" means either building or
| fetching, depending on whether a dependency is already in
| the Nix store). For every single derivation (dependency) in
| the tree, Nix first calculates what the store path for that
| dependency _would_ be and then uses that to determine if it
| 's already stored in the Nix store. So it would look at,
| say, a glibc dependency in a derivation and determine that
| the Nix store path would be
| /nix/store/7kn2mkg0g49lfflkdip7i39q3zsck4pc-glibc. If
| that's already in the Nix store then it doesn't need to
| build that. And it applies this logic throughout the entire
| dependency tree. In some cases, the entire dependency tree
| has already been built and written to the Nix store, in
| which Nix knows that it doesn't have to build _anything_.
| So Nix 's caching logic doesn't apply only to the "first
| steps" of a Nix build (or realisation, to be more
| specific); it applies to _all_ steps.
| grhmc wrote:
| Check out https://zero-to-nix.com/concepts/closures and
| https://zero-to-nix.com/concepts/realisation, which I
| think add more color to this (and also is largely
| authored by biggestlou here.)
| grhmc wrote:
| Nix's design gives good guarantees about reproducibility. It
| may not be bit-perfect, but in general it is _very_ good. Maybe
| the Zero to Nix article on Caching, and its linked pages will
| help? https://zero-to-nix.com/concepts/caching
|
| The long and short of it is a merkle tree of hashed inputs :).
| grhmc wrote:
| Graham Christensen here, cofounder of DetSys. Happy to answer any
| questions! The Magic Nix Cache has been a huge boon to us
| internally, and we're really excited to share it with the world
| today.
| zetalyrae wrote:
| Does it work with nix-shell? I don't know how to use flakes
| yet.
| grhmc wrote:
| Great question. Yes. Anything that Nix builds during your
| workflow will get cached. Give it a try and let me know how
| it goes?
| biggestlou wrote:
| Yes! The Action isn't directly aware of what Nix itself is
| doing or which commands are being run; it's only aware of the
| Nix store. So whether you're using flakes or channels it
| works the same.
| miduil wrote:
| Nice, really hope I'll find some sweet way to cache similarish
| with GitLab-CI. Also kinda been thinking about how cool it'd be
| to run Kubernetes with Nix natively (so instead of a docker layer
| registry you have nix paths mounted together to overlayfs)
| grhmc wrote:
| I think it should be pretty straightforward to make the Magic
| Nix Cache work on GitLab, too. They have a similar caching API.
| We'll take a look!
| randomblast wrote:
| Is the same true for Azure Pipelines? Given Actions forked
| from Pipelines I'd imagine it would be straightforward.
| grhmc wrote:
| I'm not sure... want to open a ticket? :)
| whateveracct wrote:
| I just spun up a gitlab-runner on NixOS (super easy due to how
| NixOS works)
| grhmc wrote:
| Nice! Yeah, we're obviously big fans of NixOS over here :).
| In cases where build infrastructure is highly ephemeral, this
| sort of cache would make a lot of sense. We'd love to help
| get it working there!
| miduil wrote:
| Yeah, you are just trading security with those runners
| though. I want the cache to be secure and not being able to
| take over other repos/branches/tags ci-jobs.
___________________________________________________________________
(page generated 2023-06-26 23:00 UTC)