[HN Gopher] Show HN: Depot - fast, remote Docker container builds
___________________________________________________________________
Show HN: Depot - fast, remote Docker container builds
Author : jacobwg
Score : 81 points
Date : 2022-09-28 18:01 UTC (4 hours ago)
(HTM) web link (depot.dev)
(TXT) w3m dump (depot.dev)
| dougdonohoe wrote:
| This sounds cool. At my current job we do multi-arch builds in
| GCP Cloud Build using `docker buildx`. The native `amd64` builds
| take like 15 minutes and the `arm64` component takes well over an
| hour with occasional time outs. We use the `--cache-to/--cache-
| from` to speed up repeated builds. If we can't wait the hour+ for
| the emulation build to run, we have a `seed-mac-arm64.sh` script
| that a developer (i.e., me) can run on their M1 mac to do the
| `arm64` half of the build, which is then picked up by GCP Cloud
| Build. It's a small pain, but helps us in a pinch.
|
| This solution sounds far easier!
| jacobwg wrote:
| Nice, yeah we had one customer using Cloud Build, and in their
| case their native amd64 builds dropped from around 12 minutes
| to 1-2 minutes on Depot builders. Surprisingly to us, it was
| even faster to push from Depot to Artifact Registry than from
| Cloud Build directly, definitely an odd networking quirk.
|
| arm64 builds should even be a more dramatic change, they should
| drop to at least as fast as your current amd64 builds.
| ranger_danger wrote:
| tluyben2 wrote:
| I am curious if it will help our builds; they are pretty slow,
| even to the point we have all kinds of tricks to speed them up.
| Thanks for trying to fix this upfront anyway.
| verdverm wrote:
| In theory, any of the BuildKit frontends like Depot or Dagger,
| should be able to pass through cross-build caching from
| BuildKit.
|
| This is how they all scale builds, and hopefully things become
| more standardized with a reduction in vendor lock in. It is
| surprising to me how much of the world relies on moby/buildkit,
| and scary if you've ever looked at the code. Primarily a single
| developer, not straightforward to understand, comments & docs
| are basically non-existent. Looks like that might be starting
| to change with so many companies now building and SaaS'n on top
| of BuildKit.
| kylegalbraith wrote:
| Thank you for the comment! We have dealt with this problem
| throughout our own careers at multiple companies. We should
| support all of the tricks you may be doing to speed up your
| builds, but we hope you have to do fewer of them using Depot.
| nicoburns wrote:
| Have you tried self-hosting the runners so that you can use a
| beefier machine to run them on? I switched our iOS builds from
| Azure Pipeline's hosted runner to a rented M1 Mac Mini (from
| Scaleway), and our iOS build time went from 20-60 minutes
| (depending on how Azure was feeling that day) to a consistent
| 3.5 minutes.
| GordonS wrote:
| I did the same thing, but with Linux and Windows build agents
| - it's so nice having a fast feedback cycle again!
| jacobwg wrote:
| Hey HN! We're Kyle and Jacob and we are excited to show you Depot
| (https://depot.dev) and get your feedback! Depot is a hosted
| Docker container build service, providing fully managed remote
| builds from CI and from your terminal. We support both Intel and
| Arm builds natively.
|
| As application and platform engineers, we have experienced the
| challenge of keeping Docker container build times fast. From
| optimizing and reoptimizing Dockerfiles, to implementing layer
| caching in CI, to running & maintaining custom runners for multi-
| platform images.
|
| Still today, there are limitations with the available tools. CI
| runners are ephemeral, and saving and loading cache tarballs is
| slow. CI providers are resource constrained, with limited CPUs
| and disk space to dedicate to fast builds. And with the
| increasing popularity of Arm devices like M1, Graviton, etc,
| building multi-platform images requires slow emulation or self-
| hosted infrastructure.
|
| We created Depot to directly address those limitations. Depot
| provides managed VMs running BuildKit, the backing build engine
| for Docker. Each VM includes 4 CPUs, 8GB of memory, and a
| persistent 50GB SSD cache disk. We launch both native Intel and
| native Arm machines, on Fly.io for Intel builds and AWS for Arm.
|
| We have built a depot CLI that embeds the Docker buildx build
| library, implementing the same CLI flags, so developers can send
| their builds to Depot VMs just by replacing `docker buildx build`
| with `depot build`. We also have a depot/build-push-action GitHub
| Action that can be swapped for docker/build-push-action in CI.
|
| The combination of native CPUs, fast networks, and persistent
| disks significantly lowers build time -- we've seen speedups of
| 2-3x on optimized projects, and as much as a 12x speedup with
| some of our customers.
|
| We believe that today we are the only hosted CI or build service
| offering the ability to natively build multi-platform Docker
| images (--platform linux/amd64,linux/arm64) without emulation.
|
| We are still early though, and would love your feedback.
|
| You can sign up without a credit card at https://depot.dev/sign-
| up to access a free project with thirty days of unlimited build
| minutes to try it out.
| swyx wrote:
| congrats on launching Kyle and Jacob (hi Kyle!)
|
| wondering what kinds of projects are/are not suitable for this.
| the only context i have is from working at open source devtool
| companies that provide docker builds for people to pull down.
| might speed up the release process slightly. i suspect my
| company https://github.com/airbytehq/airbyte/ could benefit.
| but is it also useful for internal usage?
| kylegalbraith wrote:
| Our biggest strength is building images, internal or not, in
| generic CI providers like GitHub Actions, CircleCI, etc.
| Depot excels at building images in these environments because
| it provides solutions to limitations like persistent SSD
| caching, multi-architecture support, and faster compute
| resources.
|
| Things where Depot may not be a good fit are when you only
| ever need Docker images for local development. In those
| situations, the network becomes the bottleneck because you
| have to pull down the built image each time. That said, if
| you need to build multi-architecture images locally, Depot
| becomes useful again as it can build native images for each
| architecture, no emulation needed.
|
| If you're entirely on-prem, we don't currently support that
| so Depot isn't a fit there.
| verdverm wrote:
| > We believe that today we are the only hosted CI or build
| service offering the ability to natively build multi-platform
| Docker images
|
| Dagger.io, by the originators of docker, can do this too,
| though it is not well documented. They have not yet enable
| public signups either by the looks of it.
|
| Pretty sure I've seen some other BuildKit frontends that do as
| well, the space blew up this year with several startups
| jacobwg wrote:
| I haven't had a chance to experiment with it yet, but it
| should be possible to run Dagger pipelines on Depot builders.
| Depot is effectively a BuildKit-API-as-a-service currently,
| and Dagger can connect to an external BuildKit host
| (https://docs.dagger.io/1223/custom-buildkit/).
|
| It would be cool to add an integration between Depot's CLI
| and Dagger's CLI such that Dagger could communicate with the
| remote builder machine. You could use Dagger with remote
| compute without needing to manage that compute, effectively.
| Depot even supports launching builders within your own AWS
| account, so you'd have that same ability with Dagger as well.
| mdaniel wrote:
| Dagger is Apache 2, no signup required:
| https://github.com/dagger/dagger#readme
|
| I struggle to think of what I would need a SaaS service to
| run BuildKit for me unless they're trying to take on GitHub
| Actions's compute side or whatever
| jacobwg wrote:
| I think we are trying to take on GitHub Actions' compute
| side yeah -- you'd want to either self-host your own
| BuildKit or use something like Depot for two major reasons:
|
| 1. Several of BuildKit's caching features either require or
| work best with a local disk. For instance things like `RUN
| --mount=type=cache` need a persistent disk, which Actions
| doesn't provide. You can also hit bottlenecks with saving
| and loading tarball cache from Actions (e.g. cache-from /
| cache-to) where it takes longer to download and unpack the
| tarballs compared to omitting the cache. Having a
| persistent disk makes all this "instant".
|
| 2. If you need to build images for ARM CPUs, QEMU in GitHub
| Actions is _really slow_. And if you want to build multi-
| platform images natively (e.g. --platform linux
| /amd64,linux/arm64), BuildKit needs a simultaneous
| connection to two native builders. Depot automates all that
| (it will launch two VMs at once and connect them to the
| build).
|
| GitHub Actions today also only offers 2 CPUs for the build
| - they're launching a beta of larger machine sizes
| currently.
| verdverm wrote:
| Dagger is helping companies scale BuildKit in a self-hosted
| setup (that I am aware of). Not sure about the Dagger + GHA
| self-hosted runner story, one would assume it would be a
| natural extension of their general self-hosted BuildKit,
| just point Dagger at whatever BuildKit domain/ip you are
| using.
|
| Dagger is open-source, but also has paid things if you talk
| to them. They are very focused on a few key users in their
| paid plans, rather than opening the floodgates, as I
| understand it.
| paulgb wrote:
| Congrats on the launch. Apple Silicon really threw a wrench in
| Docker local development workflows, and this feels like a
| natural solution.
|
| Does docker compose work with this?
| alphabettsy wrote:
| Apple Silicon can still build amd64 though right? Using
| BuildX and arch flags?
|
| I used it, didn't have any issues but i realize there's lots
| of reasons that might not work.
| paulgb wrote:
| You're right, you can, it's just one more thing to think
| about.
|
| I also had in mind Docker Desktop's licensing (paid-only
| for certain companies), but I now realize I was conflating
| that with Apple Silicon when it applies to Intel Macs too
| IIRC.
| jacobwg wrote:
| Thanks! We have a few users using docker-compose with a
| Makefile, where they run `depot build -t name:tag --load` to
| tag and pull the image locally, and then they reference the
| tag in their docker-compose.yml.
|
| That works, but we've considered adding a `depot bake`
| command that does the same thing as `docker buildx bake`, to
| make the integration nicer. Ideally docker-compose would have
| the ability to communicate with buildx drivers (that's what
| Depot registers as), but to my knowledge it doesn't currently
| have that option.
| dboreham wrote:
| imho having docker compose _build_ container images is an
| anti-pattern. Always do what the parent says -- build/pull
| separately (not using compose), then arrange to fix up the
| tag/version specified in the compose file to match.
| maxmcd wrote:
| When people use a service like this do they bake test runs into
| the build? Typically in CI I might build and then run tests, but
| with this service my resulting docker image is in the depot
| environment.
|
| Do I build and then download the image to run it? Or do I have
| this a bit backwards and this is for build-only use-cases?
| jacobwg wrote:
| Yep, so Depot runs BuildKit, so the same workflows work there,
| just with a remote builder. You can do one of three things:
|
| 1. Just build the image -- in CI that would test that the image
| built successfully, and the build cache would be ready for
| future builds. So for instance if you ran a build on a branch,
| but only pushed the image on the main branch workflow, the
| second run could just re-use the cache.
|
| 2. Build and push the image to a registry -- from there you
| could do anything with the image in the registry (pull it from
| any workflow, deploy it, etc)
|
| 3. Build and pull the image to the local Docker daemon. In CI,
| that might be ideal for integration testing the container, like
| you mention.
|
| You can also use option (2) for integration testing the
| container, which is especially useful with multi-platform
| images. Docker doesn't (yet) support storing multi-platform
| images locally, but it will pull the correct platform from the
| registry.
|
| tl;dr -- Depot supports the same options as `docker buildx
| build`, where you can push, pull, or just build the image
| depending on your needs
| charliermarsh wrote:
| Another unusual alternative is to model your test running
| _as_ a Docker build. It's what I ended up doing in Trellis,
| which supports Depot as a runner
| (https://github.com/charliermarsh/trellis): running the
| linter check is modeled as "build the image, then add a
| `cargo clippy` RUN command as the last layer".
| moltar wrote:
| How's is this different than running a Docker daemon on remote
| with build kit support?
| jacobwg wrote:
| It's basically the same, just without you having to do any of
| the work to set up or maintain the remote machines.
|
| When you start a container build with Depot, it will launch an
| Intel or an Arm VM running BuildKit (or both for a multi-
| platform image). It creates persistent SSD cache volumes for
| each project and attaches those to the build machines at
| launch. And it automates connecting your CLI to those machines
| over TLS. Then when the build is done, it shuts down those
| machines to save cost.
|
| Depot does all that either in our cloud account, or you also
| have the option of connecting your own AWS account and Depot
| will act just as the orchestration layer.
|
| So basically you get the remote build machines, but without
| needing to set them up, manage costs, configure access control,
| etc. But at its core, it's running the open-source version of
| BuildKit (the Depot CLI embeds the same Go library as `docker
| buildx build`).
| kmcquade wrote:
| Love using Depot. The speed improvements are insane
| trevoristall wrote:
| How does this compare to nixpacks? I've heard good things but
| admittedly haven't dug in too far yet.
|
| https://nixpacks.com/docs/how-it-works
| jacobwg wrote:
| Yeah Nixpacks is super cool! It uses nix and BuildKit to
| produce images for various different languages, and since it
| uses BuildKit under the hood, it might be able to use Depot's
| BuildKit to perform the actual build. I'd love to see what that
| integration would look like, Depot would provide the compute
| and persistent caching, and nixpacks would simplify the build
| process itself.
| intelVISA wrote:
| Nice looking site, grats on the launch!
___________________________________________________________________
(page generated 2022-09-28 23:00 UTC)