[HN Gopher] Show HN: Earthly - Better Builds
___________________________________________________________________
Show HN: Earthly - Better Builds
Author : agbell
Score : 157 points
Date : 2021-07-09 16:21 UTC (6 hours ago)
(HTM) web link (earthly.dev)
(TXT) w3m dump (earthly.dev)
| mdaniel wrote:
| I would value hearing more about the business risk that going BSL
| was designed to address:
| https://github.com/earthly/earthly/blob/main/LICENSE
|
| the associated PR just says "doing it" without saying why
| https://github.com/earthly/earthly/pull/730
|
| I have a very hard time imagining "AWS Infinibuild(R)" that eats
| the "makefile + dockerfile that builds locally" use case, which
| is why I was curious to learn more
|
| ---
|
| Separately, I would value GitHub's license detector gizmo using
| some of that sweet machine-learning sauce to identify the BSL,
| since it seems to be the path forward for these situations
| [deleted]
| captn3m0 wrote:
| GitHub licensee (the detection gizmo) uses ChooseALicense as
| the source. They recommend filing a PR there, but BSL doesn't
| fit the requirements at
| https://github.com/github/choosealicense.com/blob/gh-pages/C...
|
| https://github.com/licensee/licensee/blob/master/docs/CONTRI...
| theptip wrote:
| This looks interesting, thanks for sharing. I've been looking for
| a solution to some of the problems that this targets; in general
| our build steps are containerized, but it's annoying to have to
| layer a `make <step>` command on top of a dependency tree of
| docker/other commands inside a given CI step, and it's worse to
| have multi-step commands in the CI/CD config file, making the
| jobs harder to reproduce.
|
| One more general idea I'd be interested in others' thoughts on --
| there seems to be an artificial split between build systems (e.g.
| Bazel) and CI/CD systems (e.g. Gitlab). As these tools become
| more generic, they tend to overlap almost-completely; ultimately
| the hard part is DAG management and latency-optimized distributed
| computation. When you look at "how to use Bazel in Gitlab"[1] or
| "how to use Earthly in Github Actions"[2], it's basically having
| a single CI step that calls `buildtool <run my build DAG>`. But
| there could be other CI/CD steps like "run the e2es", and in
| general I want multiple steps in my Gitlab CI/CD pipeline so I
| can break open the individual steps, retry flaky runs, see which
| step failed, and otherwise get all the nice tight integration
| that Gitlab affords. Unfortunately I have a DAG in the CI server,
| and then another DAG in the build job; it really seems like the
| CI/CD system should somehow unwrap the build-tool's DAG into the
| CI/CD DAG, then I could rerun individual failed steps etc.
| (though I think you might perhaps want some folding, because
| Bazel DAGs are probably in general too fine-grained for the way
| CI/CD servers tend to think about the visualization problem).
|
| Maybe this is just a missing feature in GitLab? It seems like the
| Github workflow is less well integrated with the CI/CD pipeline
| (always a click or two to get to the pipeline result details) so
| maybe users there don't see any missing features.
|
| CI/CD systems like Concourse have a really nice model of
| "artifact revisions as inputs/outputs" which would play really
| well with this model; different parts of the pipeline accept a
| hash and will trigger downstream jobs only when those change.
|
| I wonder if the Earthly team has any thoughts on integrating with
| the CI/CD DAG in this way?
|
| [1]: https://about.gitlab.com/blog/2020/09/01/using-bazel-to-
| spee... [2]: https://docs.earthly.dev/examples/ci-examples/gh-
| actions-int...
| agbell wrote:
| I agree that build systems and CI/CD systems should converge.
| One way this could happen is considering one side, say the
| Bazel build or the earthfile canonical, and generating the
| config for the other side from it, including various sub-steps
| and so on. Or Bazel or an Earthfile could be the basis for a CI
| system, but I think things can go further than that. Building
| locally and building in CI should converge, and building on a
| single machine vs. building across a cluster of machines should
| also converge.
|
| I'd love to hear other's thoughts or ideas on this. We are
| thinking about this as well. There is a lot of innovation to
| come in this space.
| wikibob wrote:
| 99.9% of what is called "CI/CD" systems are _NOT_ doing
| Continuous Integration.
|
| Great article here on "CBOI: Continuous Build, Occasional
| Integration": https://www.aspect.dev/blog/cboi-continuous-
| build-occasional...
| escot wrote:
| This seems like it could be better than Docker when you need to
| manage dependent images. Like if I want to build a simple base
| image for my backend that is then used to build images for both a
| prod deployment, a testing image, etc.
| verdverm wrote:
| It's built on Docker's BuildKit which you can also enable with
| some config twerking (lol, not fixing this spell check because
| it's probably more accurate). That's how you get caching and
| DAG solving. BuildKit doesn't have a great wrapper or interface
| a.t.m.
|
| Earthly is one of several companies building on BuildKit and
| providing a better DX
| thdxr wrote:
| Earthly is great! Finally makes it so CI runs the same
| everywhere. I try to use it wherever I can
| agbell wrote:
| Thanks for using it!
|
| Yeah, I was sick of having to write build scripts in vendor
| specific formats and then not being able to run those builds
| locally without jumping through a lot of hoops.
|
| If earthly didn't exist I think I would use a combination of
| make inside docker files to define my build and then use the
| bare minimum of CI specific configs to just call into those.
| That way when I need to switch from Travis to Github or Circle
| or whatever, I don't need to rewrite the steps.
| petethepig wrote:
| I'm working on our CI/CD right now and this looks like it would
| made my life easier.
|
| Killer feature for me here would be if you could run the same
| Earthfile on a mac or a windows machine.
| agbell wrote:
| Thanks for checking it out. You can totally do that. We are
| using moby/buildkit and docker to provide this platform
| independence.
| corpdronejuly wrote:
| Even better would be support for Windows instead of WSL.
|
| I've really come to be frustrated with the a pattern where
| developers treat "it builds on Ubuntu LTS WSL2" as the same
| thing as "it works on windows"
|
| Could I do something like run one build on a windows CI host
| and also test that things behave the same way on linux and
| mac.
| vladaionescu wrote:
| Earthly team member here. Yes, we're working on native
| windows support as we speak. This has been highly
| requested.
| corpdronejuly wrote:
| Glad to hear it!
| [deleted]
| mamcx wrote:
| Yeah, I use azure so I can get build support for linux,
| windows, osx. I wish I could simplify all that, but with
| windows I need call native C++ toolchain (using rust).
| emptysongglass wrote:
| I don't get it: why wouldn't I just use a Dockerfile? The
| Earthfiles look the same. Dockerfiles run the same way every time
| and Docker supports caching. I just see more complexity on top
| without value add.
|
| If I want to move beyond local builds then I'd use something like
| Garden or Skaffold. Skaffold builds the same way every time and
| supports remote builders.
| agbell wrote:
| You can take a look at how the phoenix framework uses earthly
| to simplify their builds. They are testing across OTP versions
| and across different databases (mysql, SQL Server, postgres)
| all in an earthfile[1]. Ecto_sql does something similar[2].
|
| [1] https://phoenixframework.org/blog/improving-testing-and-
| cont...
|
| [2] https://github.com/elixir-
| ecto/ecto_sql/blob/master/Earthfil...
| ithkuil wrote:
| The Lansing page says: earthfile = makefile + dockerfile
|
| There is also an easy to understand example with several
| targets with dependencies between them.
|
| You can more or less achieve the same with multiple stages in a
| docker file, with named targets etc, but it get complicated
| fast.
| [deleted]
| geekdave wrote:
| Does this work with GitHub Actions?
| quickthrower2 wrote:
| Same question for Travis CI, is there an example setup for
| this?
| agbell wrote:
| So you just have to pull in earthly and then call it. Here is
| an example. Note that the travis file has almost nothing,
| because you can move all that to an Earthfile so that you can
| easily port it and run it locally.
|
| https://github.com/zedtux/database_cleaner-
| nobrainer/blob/ma...
|
| https://github.com/zedtux/database_cleaner-
| nobrainer/blob/ma...
| agbell wrote:
| It does. You just put all your actually build logic in an
| earthfile and call it from your github actions. A benefit of
| this is its easy to move CIs and you can test your build
| locally.
|
| Here is an example githubaction script[1].
|
| https://github.com/phoenixframework/phoenix/blob/master/.git...
| no_circuit wrote:
| I wonder what the scope of what would be a competitive service
| now that Earthly's license changed to BSL from MPL on 2021-02-01
| [0]? Seems like Earthly Technologies Inc will launch a SaaS [1].
| However, I'd assume it won't matter for most developers except
| for those selling a CI/CD SaaS. In any case, I'm looking forward
| to seeing what they end up offering!
|
| [0] https://github.com/earthly/earthly/pull/730 [1]
| https://earthly.dev/tos
| gingerlime wrote:
| Great to see earthly alive and kicking. Was in touch with Vlad
| nearly a year ago, and my main blocker to using it was longevity
| of the project in quite a saturated space. Perhaps it's time to
| take a closer look :) It looks great!
| wikibob wrote:
| Can anyone who's familiar with Earthly explain why somebody would
| want to use it instead of Bazel?
| netvarun wrote:
| Same question here too!
| oftenwrong wrote:
| I had the same thought, and found this:
|
| https://github.com/earthly/earthly#how-is-earthly-different-...
| mssundaram wrote:
| > Made with <3 on Planet Earth
|
| Does anyone else gag with when companies do this (made with
| love)? It feels too emotional, but maybe it's really true for
| them?
|
| Edit: Replace the heart character with <3 as HN doesn't seem to
| render it
| agbell wrote:
| Hello HN. Earthly is a tool for build automation.
|
| I like to think of our solution as a modern version of a
| makefile. Many people are familiar with makefiles, and back
| before every CI server was configured using YAML and groovy, this
| was a common way to build things. We are trying to improve the
| simplicity of make by adding on the repeatability of docker
| containers. By mixing those two concepts together, a lot of cool
| properties fall out.
|
| Because each target is containerized, there is no works-on-my-
| machine issues, or tests failing because something is not
| installed on one Jenkins runner or so on.
|
| Second, that isolation means that dependencies between steps need
| to be explicitly declared, and with this knowledge, earthly can
| infer parallelism.
|
| Third, containerization gives caching abilities. We can tell if
| nothing's changed since a previous run, and earthly can cache
| steps across build runs or even across machines. You get to use a
| docker registry as a build cache.
|
| We shared this on here about a year ago[1], and since that, lots
| has changed. First of all, a year ago it was just Vlad working on
| this, and now there is a number of us, including me. But for end-
| users, we have shipped many great features such as multi-arch
| builds, docker-compose support (which is handy for running
| integration tests ), inline and explicit caching, and an
| interactive debug mode. Interactive debug mode is neat but
| straightforward: a failing build step pops you into a shell at
| the point of failure.
|
| Let me know what you think of what we have built.
|
| [1] https://news.ycombinator.com/item?id=22890612
| quickthrower2 wrote:
| I love the idea.
|
| Yesterday I gave up building something locally and just got CI to
| do it because to run locally you have to kind of make a poor mans
| .travis.yml file out of bash locally then play spot the
| difference (I never figured out what was different that was
| breaking it locally, resulting in a k8s deployment with a missing
| mount/volume).
|
| This might sound weird but when you inherit some legacy code
| you'll know what I mean.
|
| Something where you can run the CI locally is such a boon.
|
| Also CI providers do have downtime. If you can run locally, you
| can't deploy your app!
| AlessandroA wrote:
| This looks very interesting. I couldn't see TeamCity mentioned in
| the CI integration section. Any plan to support it?
| agbell wrote:
| Thanks for checking it out.
|
| I've not tried this and I'm not very familiar with TeamCity but
| it should just work. If you are running the teamcity builds
| inside of containers, you may need to switch that off, as
| Earthly brings its own containerization.
| einpoklum wrote:
| I looked at their main page and got the impression this tool is
| not very / not at all useful - except perhaps for some niche I'm
| not in.
|
| > Earthly is a syntax for defining your build.
|
| I already have syntax for defining my build.
|
| > It works with your existing build system.
|
| If I have a build system, or a build system generator, why do I
| need to redefine my build another way?
|
| > With Earthly, all builds are containerized
|
| Huh? What? Why? ... That's already a deal breaker. I don't want
| to be limited to building within containers.
|
| > With Earthly, all builds are ... repeatable
|
| Isn't that true for any decent build system?
|
| > With Earthly, all builds are ... language agnostic
|
| An ambiguous statement. Especially since a build process does
| depend, to some extent, on the language of the source files.
|
| > Get repeatable and understandable builds today.
|
| (shrug) more marketing talk.
|
| > It's like Makefile and Dockerfile had a baby.
|
| I don't want Dockerfile in the family, and as for Makefile - it's
| too low-level. Makefiles get generated these days (and have been
| for over 20 years for most software packages, IIANM.)
|
| > No more git commit -m "try again" .
|
| This has never been a problem for me.
|
| (Well, ok, if there's network trouble, then downloading things
| during a build will fail; but Earthly can't magically reconnect a
| broken network.)
|
| > "Earthly is a pragmatic and incremental solution to a thorny
| problem that all organizations face"
|
| Which thorny problem is that? The "non-repeatable builds"? I'm
| confused.
| fishywang wrote:
| My thoughts exactly.
|
| I guess there's a niche for Mac/Windows developers to be able
| to run the build system on linux, because that's what their CI
| system runs (so it might fail for some issues that does not
| fail on their host machines), but most CI systems cannot run
| nested containers (because docker-in-docker requires some
| privileges), and as a result their CI system usually can not
| run this thing, which totally defeats the purpose.
| verdverm wrote:
| Earthly is essentially a make/dockerfile syntax for a BuildKit
| frontend. If you understand and agree that BuildKit has value,
| than that's the jist of it.
|
| Agree with all your points for sure, but people need words on
| their landing page
| picardo wrote:
| I've been wishing for something like this. Can I generate AWS
| CodeBuild buildspec.yml files from an Earthfile?
| agbell wrote:
| Not right now, but we are happy to take a PR :)
|
| You can probably just manually write a buildspec that has a
| single earthly call: command: -
| earthly +build
|
| I've not used CodeBuild though, so I'm not totally sure.
|
| Long term, the idea of generating the config for N possible
| build systems from Earthly is something we want to explore but
| we aren't there yet.
| picardo wrote:
| After a cursory look, it seems like there is no support for
| parallel and dependent builds in the syntax right now. Is
| this correct? Are there any plans to support that? That's the
| advantage of CodeBuild spec files -- they let me specify
| build dependency and order, but I have to write a file for
| each job. I'd rather write a single file with a simple syntax
| like the Earthfile.
| agbell wrote:
| Hi, just on my phone but Dependencies and parrellism are
| supported. It's inferred from the structure of targets.
| choeger wrote:
| Maybe this is a stupid question, but what is the "make" part? I
| see a lot of docker syntax and a few targets without
| dependencies. But make is all about dependencies. What kind of
| make dependencies do you support? What kind of variables, macros,
| etc?
| duped wrote:
| How tightly coupled is this to docker?
|
| My major gripe is that I legitimately need cross platform build
| and test (windows, Linux, and macos). I will pay hundreds to
| thousands of dollars a month to someone that solves these
| problems (I already do, and release engineers aren't hurting!).
| detaro wrote:
| Cross-platform support would be my question too, yes.
| n3t wrote:
| How would you compare Earthly to creating Docker images using
| Nix[0]? I love Nix declarative nature with strong emphasis on
| reproducibility.
|
| [0]: https://nixos.org/guides/building-and-running-docker-
| images....
| agbell wrote:
| I like nix. If you dig into nixpkgs far enough back you can
| find commits from me bumping version hashes on postgres.
|
| The caching and our future plans for distributed builds have
| some similarities with Nix Hydra and Bazel but creating images
| in earthly is closer to docker multistage builds than it is to
| nix.
|
| We are not just for building docker images, though. In the real
| world a 'build' is more than just building an image. It will
| have linting, and tests and integration tests and maybe some
| services need to be stood up and tore down in between to make
| sure an api contract wasn't broken and you might need different
| binaries generated for different architectures and so on.
|
| I could be wrong but I think nix is focused more on just the
| producing the artifact part. I do love how they approach every
| build as a pure function though.
| n3t wrote:
| It makes sense! Thank you for your response.
___________________________________________________________________
(page generated 2021-07-09 23:00 UTC)