[HN Gopher] Release engineering is exhausting so here's cargo-dist
___________________________________________________________________
Release engineering is exhausting so here's cargo-dist
Author : ag_dubs
Score : 129 points
Date : 2023-02-01 17:50 UTC (5 hours ago)
(HTM) web link (blog.axo.dev)
(TXT) w3m dump (blog.axo.dev)
| ZeroCool2u wrote:
| "Alright so I've given you The Pitch of how this should work, but
| how does it actually work today? Well, yes, I wouldn't announce
| this if not -- but there's a lot of future work to do! Actually
| no I take that back, it's definitely perfect. Those familiar with
| my broader oeuvre know this will absolutely be a perfectly
| crafted demo that avoids all the sharp corners of cargo-dist!"
|
| One of the most honest paragraphs ever written.
|
| Seriously though, great tool and great write up. I hope something
| like this lands as an official cargo feature. Coming from mostly
| Python land at work with crazy dependencies for TF and PyTorch
| GPU support (On Windows sometimes!) makes me super jealous.
| chatmasta wrote:
| I'm really not a fan of the "download the prebuilt binary from
| github releases" workflow that's been proliferating along with
| the popularity of Rust. It seems like a step backward in terms of
| secure package management, and it's surprising to me that Rust
| doesn't offer a more out-of-box experience for this, instead
| encouraging users to build from source. I understand the
| arguments for this, and I even find some of them convincing -
| namely, that the inverse problem of opaque intermediate binaries
| and object files would be much worse as it would cause a cascade
| of brittle builds and barely any packages would work.
|
| But the fact remains that end users want to download a binary,
| and the common approach to this is to simply publish them to
| GitHub actions. Surely Cargo could offer a better option than
| this, while retaining the ability to build all packages from
| source (maybe you can only publish binaries to Cargo _in
| addition_ to the source files... or maybe Cargo.toml could
| include a link to GitHub releases, and Cargo could include a
| command for downloading directly from there.)
|
| In the meantime, I've been considering publishing Rust binaries
| to npm (in addition to GitHub releases). I got the idea from
| esbuild, which is written in Go but distributed via npm. What do
| people think of this approach? Here's a recent blog post [0]
| describing the setup.
|
| [0] https://blog.orhun.dev/packaging-rust-for-npm/
| ag_dubs wrote:
| we actually agree and are working on this! github releases are
| just an easy initial target, and makes our tool a drop-in
| replacement for the kinds of things people are already doing.
| longer-term we'd like to see something more robust, and cargo-
| dist is the first cog in that machine.
|
| i have personally packaged and published many rust devtools on
| npm (cloudflare's wrangler, apollo's rover, wasm-pack) but that
| was largely because they were targeted at a javascript
| developer audience.
|
| as a former npm registry engineer i'm curious what you find to
| be the particular value of publishing to npm? installing node
| is actually very unpleasant and then getting global installs to
| work is also... very unpleasant. i think it works well for
| people already in that ecosystem but i think we can build
| something better for a more agnostic audience that can give a
| similar out-of-box exp without requiring a centralized
| registry. would love to learn more about your perspective!
| simonw wrote:
| I've noticed people publishing binaries to PyPI too - you can
| run "pip install ziglang" to get Zig, for example.
|
| I wrote a bit about that pattern here:
| https://simonwillison.net/2022/May/23/bundling-binary-tools-...
| jacoblambda wrote:
| Worth considering using nix with cargo. Of course it still
| involves a lot of "download from github" or "download from nix
| cache" but reproducibility + tight source hash pinning helps
| guarantee provenance.
| Shared404 wrote:
| > (Did you know the warning you get on Windows isn't about code
| signing, but is actually just a special flag Windows' builtin
| unzipping tool sets on all executables it extracts?)
|
| My jaw hit the floor here.
| Gankra wrote:
| Correction on this someone else sent me:
|
| The check of interest is for a Mark Of The Web[0] flag that
| Windows includes in file system metadata. The builtin unzipping
| utility just faithfully propagates this flag to the files it
| unpacks. Other utilities like 7zip are unlikely to do this
| propagation (effectively clearing it).
|
| But yeah either way it has nothing to do with code signing!
|
| [0]: https://nolongerset.com/mark-of-the-web-details/
| jakub_g wrote:
| If someone's interested with more details about MoTW, EricLaw
| (long time MSFT engineer at IE and Edge teams) got you
| covered:
|
| https://textslashplain.com/2016/04/04/downloads-and-the-
| mark...
|
| https://textslashplain.com/2022/12/02/mark-of-the-web-
| additi...
| chatmasta wrote:
| macOS has a similar feature with Gatekeeper, which bit me
| when preparing a Pyinstaller binary for Mac. The flag doesn't
| get added when you download a file with curl, but it does
| when you download it through a web browser, which can cause
| difficult to debug issues with binaries downloaded from
| GitHub releases.
|
| You can remove this flag with the xattr command:
| xattr -d com.apple.quarantine the_quarantined_binary
|
| I wrote up the details of this in a PR [0] where I last dealt
| with it.
|
| [0] https://github.com/splitgraph/sgr/pull/656
| maldev wrote:
| This is actually pretty similar. The OS has an alternative
| data stream(An idea they stole from Mac), and they list
| what site a exe was downloaded on, or if it came from
| somewhere else. Others incorrectly called it a flag, when
| it works by having two different file data streams for a
| single file, one is the default one.
|
| So for example, a single file can actually contain two
| different "files"(File data).
|
| So, foo.exe, actually will effectively open the file
| foo.exe:DEFAULT. You could also add a piece of malware to
| the foo file in place of a datastream. So foo.exe is legit,
| but if you open foo.exe:MALWARE , it will open up the
| malware datastream.
|
| So tldr, how Windows does this, it when you get a file from
| a third party source(Internet, USB Drive, etc), it adds a
| new datastream in the form of a textfile. And the textfile
| contains info about the source. Namely, a number for
| location it came from(3? for web), and then some more info.
| jph wrote:
| Awesome thank you. I'm adding it to the cargo favorites list:
| https://github.com/sixarm/cargo-install-favorites
|
| Tiny feedback:
|
| - Can you consider changing "git add ." to be explicit e.g. "git
| add Cargo.toml .github/workflows/release.yml"?
|
| - How about modifying Cargo.toml to add cargo-dist as a dev-
| dependency? I know it's not strictly necessary; it's just very
| helpful for typical collaboration.
| Gankra wrote:
| Do you mean this kind of dep? https://rust-
| lang.github.io/rfcs/3028-cargo-binary-dependenc...
|
| That's an interesting thought, I'm not sure I've ever seen
| someone employ that as a pattern. Actually no wait, I thought
| cargo bin-deps specifically gave the developer no way to
| manually invoke it (i.e. there's no equivalent functionality to
| npm's npx)? Without that, what use would the dependency be?
| jph wrote:
| Yes, where that link describes "[dev-dependencies]" and
| "[build-dependencies]".
|
| I use the dev-dependencies section often, and the build-
| dependencies rarely. And anyone here, please correct my
| understanding if there's a better way to do what I'm
| describing.
|
| For me, these sections are an easy way to be explicit with
| collaborators that the project needs the deps installed in
| order to work on the project, and does not want/need the deps
| to be compiled into the release.
|
| My #1 use case is to have a collaborator start with a git
| clone, then run "cargo build", and have cargo download and
| cache everything that's needed to work on the project and
| release it. My #2 use case is to deal with semver, so a
| project can be explicit that it needs a greater cargo-dist
| version in the future.
|
| To your point about cargo bin deps akin to npx, yes, your
| understanding matches mine i.e. not available yet. I do
| advocate for that feature to be added because it's helpful
| for local environments. Cargo does offer "default-run" which
| shows there an awareness of a developer preferring specific
| local exectuables-- maybe Cargo can/will add more like that
| for deps?
| IshKebab wrote:
| > Congrats kid you're A Release Engineer now and your life is
| hell. Enjoy debugging basic typos on a remote machine with 20
| minute latency because you sure can't run those Github CI bash-
| scripts-in-yaml files locally!
|
| Yes! Why is this accepted??
|
| Gitlab has a way of running CI locally (for Docker based builds
| anyway; who knows about Windows or Mac) but a) it doesn't support
| the same features at the "proper" one (even basic ones like the
| `default` key) and b) they deprecated it!
|
| Ok in fairness they've stated in a random comment that they won't
| remove it before providing an alternative.... But still, how is
| this not a core feature of all CI systems?
| sedatk wrote:
| You can run GitHub workflows locally with act:
| https://github.com/nektos/act
| frodowtf wrote:
| Act is okay, but the runner image behaves quite differently
| than a GitHub runner would. The original image would just be
| too big for reasonable local workflows.
|
| Also artifacts don't seem to be supported.
| ollien wrote:
| This is far from perfect, IME. The big problem I have (and
| maybe there's a solution I don't know about) is that there's
| no easy way to test getting event data, unless you wanna
| rebuild the events yourself (which is hardly reliable if the
| problem is something like verifying a conditional that
| enables/disables a stage).
| adrienthebo wrote:
| Seconding this; act has a lot of potential but misses a
| number of features such as support for deployment
| environment variables (eg `${{ var.DEPLOY_SPECIFIC_ENV_VAR
| }}`) and only recently added support for reusable workflows
| (https://github.com/nektos/act/issues/826). It looks like
| fine software and the maintainers deserve praise for their
| work but it's not yet a drop-in replacement for GitHub
| Actions.
| wizzwizz4 wrote:
| > _But still, how is this not a core feature of all CI
| systems?_
|
| Vendor lock-in, presumably.
| WorldMaker wrote:
| In multiple directions, too. It's easier to build the CI
| system itself if you are only targeting one class of
| servers/one means of hosting servers/one specific "cloud".
| capableweb wrote:
| CircleCI solved this so many years ago, allow users to SSH into
| the environment the build happens.
|
| Workflow is something like:
|
| - Guess together a random CI or CD workflow, this is just to
| kick off the process. You can also start with a empty config.
|
| - Something fails, get SSH host+port to connect to
|
| - Enter environment and manually do everything you want to be
| able to automatically do
|
| - Execute `history` and copy output, trim it into something
| nicer and put in your config
| angio wrote:
| If you're using nix to build your repo, it's worth adding
| scripts for releases etc and run them locally and as part of
| the ci https://determinate.systems/posts/nix-github-actions
| brundolf wrote:
| What we need is a standard CI language/config format across
| vendors (stuffing a custom DSL inside YAML doesn't count)
|
| That would allow for a tooling ecosystem of static checkers,
| offline runners, IDE integrations, etc etc, and would also cut
| down on the learning barrier every time you switch to a new
| company that uses a different vendor
| samsquire wrote:
| There's so much work to do to release software. Kind of explains
| why everything is a website.
| oconnor663 wrote:
| Bend the curve!
| MuffinFlavored wrote:
| I think this would benefit from an example repo that shows just
| Cargo.toml for a simple src/main.rs with `fn main() {
| println!("Hello, world!"); }` project with the simplest needed
| .github/workflows/foo.yaml possible to actually use this.
|
| If it was in the article and I missed it I apologize.
| Gankra wrote:
| The "way-too-quickstart" is the minimal example:
| https://github.com/axodotdev/cargo-dist#way-too-quick-start
|
| A key feature of cargo-dist is that
|
| cargo dist init --ci=github
|
| should simply set everything up for an arbitrary* Rust
| workspace and you just check in the results.
|
| * Not sufficiently tested for all the wild workspaces you can
| build, but a "boring" one with a bunch of libraries supporting
| one binary should work -- that's what cargo-dist's own
| workspace is, and it's self-hosting without any configuration.
| Any time I want to update the bootstrap dist I just install
| that version of cargo-dist on my machine and run `cargo dist
| generate-ci github --installer=...` to completely overwrite the
| ci with the latest impl.
| Gankra wrote:
| Just to elaborate on this a bit: as discussed in the Concepts
| section of the docs[0] the core of cargo-dist absolutely
| supports workspaces with multiple binaries, and will chunk
| them out into their own distinct logical applications and
| provide builds/metadata for all them.
|
| However this isn't fully supported by the actual github CI
| integration yet[1], as I haven't implemented proper support
| for detecting that you're only trying to publish a new
| version of _only one_ of the applications (or none of them!),
| and it doesn 't properly merge the release notes if you're
| trying to publish multiple ones at once.
|
| I never build workspaces like that so I'm waiting for someone
| who _does_ to chime in with the behaviour they want (since
| there 's lots of defensible choices and I only have so many
| waking hours to implement stuff).
|
| [0]: https://github.com/axodotdev/cargo-dist/#concepts [1]:
| https://github.com/axodotdev/cargo-dist/issues/69
| MuffinFlavored wrote:
| Cool, thanks.
|
| It creates a ~100 line release.yml file
|
| 1. create GitHub release
|
| 2. upload artifacts
|
| 3. upload manifest
|
| 4. publish GitHub release
|
| I like it, it's opinionated, but I don't know how much it
| will catch on. If somebody needs to maintain a 100 line
| GitHub Actions release YAML file, in my experience you
| typically want to understand everything in it in case you
| need to adjust it for future needs/as they grow.
|
| It's well done though. Curious to see how much adoption it
| picks up.
| Chmouel wrote:
| i really enjoy goreleaser
| https://github.com/goreleaser/goreleaser/ and use it with rust
| https://github.com/chmouel/snazy/blob/main/.goreleaser.yaml#...
| combined with build matrix
| https://github.com/chmouel/snazy/blob/main/.github/workflows...
| ag_dubs wrote:
| definitely inspired by goreleaser! it's a great project
___________________________________________________________________
(page generated 2023-02-01 23:00 UTC)