[HN Gopher] Dagger: a new way to build CI/CD pipelines
___________________________________________________________________
Dagger: a new way to build CI/CD pipelines
Author : shykes
Score : 230 points
Date : 2022-03-30 16:00 UTC (7 hours ago)
(HTM) web link (dagger.io)
(TXT) w3m dump (dagger.io)
| nickstinemates wrote:
| Congrats on the launch - best of luck!
| fpopa wrote:
| Congrats on the launch.
|
| Hopefully this will be the last pipelining language I'll learn.
| EddieRingle wrote:
| I support the effort to build a platform-agnostic CI/CD pipeline
| solution, but I don't want it in the form of yet another
| platform. Rather it needs to be a protocol that any platform can
| tie in to. I'm especially wary since this is another VC-backed
| effort that will eventually need to be monetized in some shape or
| form.
|
| Additionally, as someone else here has already mentioned, my mind
| first went to Dagger, the dependency injection tool
| (https://dagger.dev). That tool in particular was named as a play
| on DAG (directed acyclic graphs), whereas in this case I don't
| think it would apply since there may be instances where you'd
| want cycles in a pipeline.
|
| On a whim, I clicked on "Trademark Guidelines"
| (https://dagger.io/trademark) and from that page alone I would
| recommend avoiding this based on the aggressive language used to
| try and claim ownership of generic words. According to their own
| language, it seems I'm violating their guidelines by writing this
| comment.
|
| > Our Marks consist of the following registered, unregistered
| and/or pending trademarks, service marks and logos which are
| subject to change without notice: Dagger; Blocklayer; and other
| designs, logos or marks which may be referred to in your specific
| license agreement or otherwise.
|
| > Blocklayer does not permit using any of our Marks ... to
| identify non-Blocklayer products, services or technology
|
| Which would include Dagger, the dependency injection tool.
|
| Other sections of note:
|
| > Do Not Use As Nouns
|
| (This one just reads amusingly to me, for some reason.)
|
| > Do Not Create Composite Marks
|
| This section seems to suggest that you can't use "dagger" in any
| shape or form, even as a smaller part of some other word or body
| of text.
|
| > Websites And Domain Name Uses
|
| >
|
| > ... Any principal or secondary level domain name should not be
| identical to or virtually identical to any of our Marks.
|
| >
|
| > The website owner should not register any domain name
| containing our Marks and should not claim any trademark or
| similar proprietary rights in the domain name. For example,
| "daggertech.com", "dagger-group.com", "Meetdagger.com" are not
| allowed. Any violation of this policy may result in legal action.
|
| >
|
| > The website should otherwise comply with domain name registry
| policies and applicable laws regarding trademark infringement and
| dilution.
|
| This would technically include dagger.dev, which again refers to
| the dependency injection tool.
|
| ---
|
| Full disclaimer that I'm not a lawyer and there could be totally
| reasonable explanations for these provisions, but they certainly
| look scary to a layperson such as myself. All in all, the
| founders seem to be taking a pretty arrogant approach here, but
| it unfortunately seems to be a common one. I'm choosing not to
| support it, however.
|
| ---
|
| EDIT: _formatting_
| epage wrote:
| > Dev/CI drift: instead of writing the same automation twice -
| once in a CI-specific configuration, and again in a Docker
| compose file, shell script or Makefile - simply write a Dagger
| plan once, then run it in all environments.
|
| Working on developer tooling, a lot of times I would hear from
| people that they wanted CI and dev to be 100% the same and wanted
| a simple "run all CI locally" command to pre-check before
| posting.
|
| Unsure how Dagger is handling this but my concerns with the
| scenario I described
|
| - CI normally divides things up into multiple jobs for speed
| which breaks the shared "do everything" command
|
| - Commands need to scale down to fast iteration as people fix
| things
|
| - Generally people get the best integration by using the
| underlying wrapped tools directly due to pre-commit or IDE
| integeration
| candiddevmike wrote:
| I don't understand why folks can't write everything in a
| makefile and then call make targets in CI.
| shykes wrote:
| That is a common pattern. Dagger is essentially a more modern
| iteration on that pattern.
| vips7L wrote:
| Not to be confused with https://dagger.dev/
| clumsysmurf wrote:
| When I read the headline, my first thought was dagger
| dependency injection got a CI/CD feature ... which is
| ridiculous of course. I think this naming will cause confusion,
| especially since dagger DI is a large project from google.
| j2kun wrote:
| +1 what a strange choice to deliberately conflict with an
| existing project
| ComradePhil wrote:
| Also not to be confused with https://dagster.io/
| cebert wrote:
| Our team has had great success with GitHub Actions and
| Environments for CI/CD. One nice thing going that route is the
| build related code is contained within repositories. A large
| number of developers are already familiar wit GitHub, which makes
| onboarding new team members easier. I don't see anything too
| compelling with dagger.io that is missing with GitHub. You can
| even use ACT to test workflow changes to builds locally.
| mhitza wrote:
| How well does act work nowadays in practice? I was automating
| multiple PHP, Ansible, and Nodejs related projects last year
| and act failed (can't remember the exact errors now) for each
| project at some step.
| mdaniel wrote:
| My experience trying to get vscodium to build using act was
| similarly "oh no," which I think is a cat-and-mouse pitfall
| that's found in every emulator
|
| The patch I made to act was bigger than I thought the act
| project would accept, so I just worked around it with some
| well placed docker volumes and running GH actions "by hand"
| phphphphp wrote:
| I'm in the same boat, but I do think there's a prospect of
| Dagger being a superior option in the long term, if...
|
| 1. They invest in building out their catalog of actions to
| compete with GitHub's. I maintain a few GitHub Actions and
| despite the GitHub catalog's depth, it's still lacking in many
| ways and GitHub don't appear to invest in it too much: a
| "maintainer fund" and creative poaching from Dagger could
| rapidly bring them up to par. A few million of their raise,
| well deployed, could crush GitHub's catalog.
|
| 2. They invest in tight integrations with platforms. GitHub
| Actions is great because of composability, yes, but also the
| deep integration with GitHub itself. Being able to run Dagger
| on GitHub Actions is one thing, but being able to leverage
| deployment environments cross-platform would be another.
|
| 3. GitHub Actions is great, I am a fan of it, I'll speak highly
| of it often, but the codebase... it is bad. If Dagger can build
| out a platform that competes with GitHub Actions on
| functionality, and it has a pleasant codebase, they'll make
| huge gains from community participation. Contributing to GitHub
| Actions is painful.
|
| So, I agree with you today, but a year from now, I could see a
| very different situation and I am optimistic.
| phphphphp wrote:
| Also I'm sure he's on their radar given he works for Docker,
| but they should spend whatever it takes to hire
| github.com/crazy-max
| shykes wrote:
| Fun fact, Crazy Max is the author of the Github Action for
| Dagger :) https://github.com/dagger/dagger-for-github
| barbs wrote:
| Not to be confused with Dagger, the dependency injection
| framework used in Android apps [0]. That's DI, not CI!
|
| [0] https://developer.android.com/training/dependency-
| injection/...
| zinclozenge wrote:
| harness.io was by far the best CD experience I ever had, and it
| allowed me to do exactly what I wanted. I can't stand the gitops
| way of defining every change through a PR and having an
| operator/controller sync the changes. It's a lot more intuitive
| for me to say "everything in master ends up in prod" and define a
| pipeline that moves artifacts between environments. The only
| other software that seems like it does the same thing is
| Spinnaker, which i have no desire to try self-hosting.
| andreineculau wrote:
| I'm happy to see dagger, because it is something that I have been
| working with since 2016, and ended up creating
|
| https://github.com/ysoftwareab/yplatform
|
| It uses well-tested software (GNU Make, Bash, Homebrew, Docker)
| and there's not much to learn really. * Incubated
| at Tobii (eyetracking, Sweden), * it has been used for
| several projects of different sizes, * tested on 12+ CIs !
| * tested on macOS, Linux and Windows (WSL) ! * tested on 8+
| Linux distros ! * comes with prebuilt Docker images !
|
| Early integration with VSCode. And much more. Just read the
| README :)
|
| Happy to help anyone integrate it.
| vhanda wrote:
| How would this handle CI/CD jobs which need to be run on a mac?
|
| From what I understand, everything needs to be a container, and
| that doesn't work for xcode.
| evancordell wrote:
| I played with a similar idea a while ago:
| https://github.com/ecordell/cuezel/ (cuezel as in: "Bazel but
| with CUE"), but I was never sure that what I was doing was in the
| spirit of CUE.
|
| CUE pushes nondeterminism into "_tool.cue"[0] files that are
| allowed to do things like IO and run external processes. Tool
| files scratch a similar itch to Makefiles, but they lack an
| integrated plugin system like Bazel (hence why I played with the
| idea of CUE + Bazel).
|
| With Dagger you seem to be restricted to the set of things that
| the dagger tool can interpret, just like with my Cuezel tool you
| are limited to what I happened to implement.
|
| In CUE `_tool` files you are also limited to the set of things
| that the tool builtins provide, but the difference is that you
| know that the rest of the CUE program is deterministic/pure
| (everything not in a _tool file).
|
| There's clearly value in tooling that reads CUE definitions, and
| dagger is the first commercial interest in CUE that I've seen,
| which is exciting.
|
| But I'm most interested in some CUE-interpreter meta-tool that
| would allow you to import cue definitions + their interpreters
| and version them together, but for use in `_tool` files to keep
| the delineation clear. Maybe this is where dagger is heading? (if
| so it wasn't clear from the docs)
|
| [0]: https://pkg.go.dev/cuelang.org/go@v0.4.2/pkg/tool
| shykes wrote:
| I know exactly what you mean. Earlier versions of Dagger
| actually took this "embedding" approach: any CUE configuration
| is a valid Dagger configuration, with an extra layer of
| annotation that describes how to make that configuration
| runnable. What we learned is that this model is thoroughly
| confusing to almost everyone except hardcore CUE enthusiasts.
|
| Now we use CUE in a more straightforward way: as a superior
| replacement for YAML. There's a (simple) schema that all Dagger
| plans must follow; beyond that, you can import any CUE package
| and apply any definition into your plan. But you don't need to
| go upstream and annotate these packages with additional Dagger
| metadata.
|
| I'm not sure if my explanation is clear - even I get confused
| by this embedding business sometimes.
| evancordell wrote:
| In the examples it looked as though there is still a CUE
| unification loop happening during dagger processing:
| deploy: netlify.#Deploy & { contents:
| build.contents.output }
|
| It looks like dagger is using cue as a bit more than a YAML
| replacement; it hydrates cue values as it runs - which is
| cool! - but that's the part that seemed at odds with CUE's
| philosophy of pushing nondeterminism into clearly marked
| files.
| shykes wrote:
| Yes, we use CUE as more than just a YAML replacement. In
| particular we use CUE references to construct the DAG and
| "compile" it to buildkit.
|
| And, yes, Dagger will gradually fill the missing values in
| the CUE tree during runtime. Essentially resolving the DAG
| on the fly. It is pretty cool :)
|
| We have discussed this topic at length with the CUE
| developers. Our conclusion is that CUE's deterministic core
| is what matters, and the `_tool.cue` pattern is more
| peripheral: more of a reference for how other tools might
| use cue for non-deterministic operations. It's not
| realistic for CUE to be both a ubiquitous deterministic
| language, and a successful non-deterministic tool. Its
| priority is clearly the former, and we're focusing on the
| latter.
| time0ut wrote:
| Congratulations! This looks amazing. I think I am the target
| audience and I cannot wait to try this.
|
| One very important thing for my use case is being able to run
| steps in parallel distributed across multiple agents. Is it
| capable of this?
| shykes wrote:
| Yes :) We rely entirely on buildkit
| (https://github.com/moby/buildkit) to run the pipelines
| (`dagger` itself acts as a "compiler" and buildkit client). So
| anything you can do with buildkit, works out of the box with
| Dagger. That includes clustering.
|
| There is a lot of ongoing work in that area in the buildkit
| ecosystem. For example Netflix has a distributed buildkit farm,
| which I believe relies on containerd storage backends to share
| cache data. There are also integrations to run buildkit nodes
| on a kubernetes cluster.
|
| Dagger itself is obviously quite new, but buildkit itself is
| very mature, thanks to its integration with 'docker build'.
| mountainriver wrote:
| We should definitely escape from YAML, but this language isn't it
| speed_spread wrote:
| Full YAML is crazy, but the StrictYAML subset I can definitely
| live with.
|
| I regularly go over the main alternatives (TOML, JSON, XML),
| and they all have serious warts for config IMO.
|
| Better formats exists (such as HOCON) but are much less
| popular. For a public tool I'd still stick with a YAML subset
| that everyone knows and can work with a minimum of fuss.
| mountainriver wrote:
| I agree everyone knows YAML but its only really great as a
| config language, not as a general purpose turing-complete
| programming language. The answer here is more likely to be
| something like what Pulumi did
| shykes wrote:
| Pulumi is great and we think of it as complementary to
| Dagger: one is focused on managing infrastructure, the
| other on deploying applications on top of the
| infrastructure.
|
| They are more similar in their choice of language than you
| might think:
|
| * Dagger lets you write actions in any language: Go,
| Python, Shell, it doesn't matter. Then it runs it in
| containers, wrapped in a declarative configuration. This
| allows cross-language interoperability.
|
| * Pulumi also supports existing languages, although it is
| more deeply integrated, with language-native SDKs (the
| tradeoff being less supported languages). Eventually they
| realized that they needed an additional layer to bridge
| language siloes, otherwise the Pulumi ecosystem is really
| an archipelago of incompatible components: Pulumi/Go,
| Pulumi/TS, etc. To solve this Pulumi introduced... a
| declarative configuration wrapper.
|
| In summary: the best design for this problem requires two
| layers. One layer for leveraging the existing programming
| languages with maximum compatibility. And a second to add a
| declarative configuration model and reduce ecosystem
| fragmentation. Pulumi and Dagger are both converging on
| this model, each for their own domain of application.
|
| I personally believe we will see many Dagger + Pulumi
| integrations in the future :)
| seeekr wrote:
| This seems similar to what https://earthly.dev is doing?
| shykes wrote:
| Yes, I believe that is a fair comparison. Earhly is more
| focused on builds, whereas Dagger has a wider scope: build,
| test, deployment, any part of a CI/CD pipeline really. But the
| overall philosophy is the same: run everything in containers.
| The choice of buildkit as a runtime is also a key similarity.
| One big difference is that we use CUE as a configuration
| language, and Earthly uses YAML.
|
| We have a lot of respect (and common friends!) with the Earthly
| developers, I am confident we can help each other build even
| better tools, and grow the buildkit ecosystem in the process.
| seeekr wrote:
| Thanks for the info, that is wonderful to hear! Best of luck
| to both you and Earthly, much needed work you're doing!
| mdaniel wrote:
| Another *monster* difference is that Dagger is (at least
| currently) Apache 2:
| https://github.com/dagger/dagger/blob/v0.2.4/LICENSE but
| Earthly went with BSL:
| https://github.com/earthly/earthly/blob/v0.6.12/LICENSE
|
| That means I'm more likely to submit bugs and patches to
| Dagger, and I won't touch Earthly
| shykes wrote:
| I can't speak to Earthly's choice of license (and am not
| familiar with BSL). But can confirm that we have no
| intention of changing licenses, and if we did, it would be
| for another OSI-approved license.
|
| Our monetization model follows the same fundamentals as Red
| Hat before us: open code, strictly enforced trademark
| rules. "You can modify, use and redistribute the code at
| will. If you distribute a modified version, please call it
| something else."
| cprecioso wrote:
| Earthly doesn't use YAML but a syntax which is similar to
| Dockerfile with dashes of Makefile. I'm finding it extremely
| pleasant to write.
| shykes wrote:
| I stand corrected! Earthly developers if you read this, I'm
| sorry for misremembering. I would correct my original post
| if I could.
| oh_sigh wrote:
| Come on guys...there is an _extremely_ well known and widely used
| Java library called Dagger.
|
| Call yours Cinquedea, Jambiya, Anelace, or Rondel if you really
| want a knifey name.
| xrd wrote:
| This seems interesting.
|
| But, I wish there was some code to show me what makes it so
| radically different. It seems like this is targeting developers
| (or is it devops team?) and I'm excited about the new language
| here, but I don't see any examples of code. Code engages both my
| head and heart.
|
| I am reusing a lot of code in my CI jobs. I have an upload script
| that I reuse. I have a DEB package script that I reuse across
| many projects. So, that assertion rings false to me, and seems to
| indicate there is an unhealthy wall between devops and the
| developers that prevents shared code. Maybe I misunderstand.
|
| The thing that always bites me is that I have trouble debugging a
| job that depends on the artifacts from a prior job. My e2e job is
| flakey, and I'm loathe to fix it, because I have to re-run the
| entire pipeline, test -> build -> installer, etc to get the final
| artifact used in the e2e job. I've not figured out a way with
| "gitlab-runner exec" to run a job later in the pipeline and
| somehow pass in my own artifacts locally. This would be something
| (albeit very specific to gitlab) that would make me very excited.
| dijksterhuis wrote:
| > But, I wish there was some code to show me what makes it so
| radically different.
|
| Sorry to be _that guy_ but maybe try the docs page...?
|
| https://docs.dagger.io/
|
| https://docs.dagger.io/1205/container-images
| xrd wrote:
| Thank you.
|
| It isn't hard to find, but my point was that if you say
| "using an intuitive declarative language" then a developer
| will get excited by a code snippet that shows that intuitive
| developer language. It wasn't there, and I think their post
| could be improved by having less fluffy language and more
| code, if they are targeting me, that is.
|
| This link does show the code, but it took a few clicks to get
| there from your links:
|
| https://docs.dagger.io/1202/plan
|
| At first glance I'm not in love with the language. When I
| look at the first example, there is a lot I have questions
| about.
| verdverm wrote:
| The language is CUE, which I think will see mass adoption
| in config / DevOps in the coming years. So regardless what
| you think of the language today, it is likely to become
| important and part of your like in the not too distant
| future.
|
| https://cuelang.org | https://cuetorials.com
|
| Dagger builds on top of CUE and the (DAG) flow engine
| therein
| dijksterhuis wrote:
| Oh wow, the validating yaml example on the front page
| looks super useful. Might try and play with this over the
| next few weeks!
| verdverm wrote:
| Once CUE clicked for me, there was no going back
| xrd wrote:
| That's great context. That should be added to that doc
| link right at the top, it would have me feel much more
| safe about investing the time to learn it!
| verdverm wrote:
| CUE has good pedigree, the creator Marcel wrote the
| prototype for Borg (k8s), worked on both Google config
| languages, and worked on the Go team. CUE is how he
| thinks those Google config languages should have been
| designed.
| KingEllis wrote:
| Another starting point:
| https://github.com/dagger/dagger/tree/main/ci
| topher200 wrote:
| There's some code examples in their docs:
| https://docs.dagger.io/1202/plan
| BenoitP wrote:
| What an unfortunate name collision:
|
| https://github.com/google/dagger
|
| Is it some weird SEO niche-squatting technique? IMHO we see it
| happen way too often
| shykes wrote:
| Hi everyone, I'm one of the co-founders of Dagger (and before
| that founder of Docker). This is a big day for us, it's our first
| time sharing something new since, you know... Docker.
|
| If you have any questions, I'll be happy to answer them here!
| candiddevmike wrote:
| What's the monetization strategy going to be?
| shykes wrote:
| There will be an optional cloud service (not yet available).
| Its features will be based on requests from the community.
| Some problems just can't be solved with a command-line
| client. For example: visualization of your pipelines; a
| centralized audit log of all jobs run across all machine;
| centrally managed access control and policies; etc.
|
| We will _not_ rely on unusual licences to restrict
| competitors from running Dagger as a service. We don 't need
| to, since `dagger` is a client tool. We do encourage cloud
| providers to run buildkit as a service, though :)
|
| Generally, we take inspiration from Red Hat for their
| balancing of open-source community and business. They are
| very open with their code, and tightly control how their
| trademark is used. You can clone their IP and use it to
| compete with them - but you have to build your own brand, and
| you can't confuse and fragment the Dagger developer community
| itself. We think that is a fair model, and we are using it as
| a general guideline.
| EddieRingle wrote:
| > We will not rely on unusual licences to restrict
| competitors from running Dagger as a service.
|
| Your "Trademark Guidelines" appear to contradict you:
|
| > Third-party products may not use the Marks to suggest
| compatibility or interoperability with our platform. For
| example, the claims "xxx is compatible with Dagger", "xxx
| can run your Dagger configurations", are not allowed.
|
| > but you have to build your own brand, and you can't
| confuse and fragment the Dagger developer community itself
|
| If I do an incognito Google search for "dagger", the first
| result is the Wikipedia page for the knife, and the second
| result is for Dagger, the dependency injection tool. By
| naming this "Dagger" you're confusing not just your own
| developer community but the pre-existing one as well.
| shykes wrote:
| > > We will not rely on unusual licences to restrict
| competitors from running Dagger as a service.
|
| > _Your "Trademark Guidelines" appear to contradict you:_
|
| They do not. Software licenses and trademark guidelines
| are two different things. Some commercial open-source
| vendors have changed their licenses to restrict use of
| the software in various ways - typically to limit
| competition from large cloud providers. We don't do that,
| and have no intention to. Our license is OSI-approved and
| we intend to keep it that way. That is what I am
| referring to.
|
| > _but you have to build your own brand, and you can 't
| confuse and fragment the Dagger developer community
| itself_
|
| This is the intent behind the language in the trademark
| guideline which you quoted: you can redistribute and
| modify our code. But if you distribute a modified copy,
| call it something else.
|
| > > _Third-party products may not use the Marks to
| suggest compatibility or interoperability with our
| platform. For example, the claims "xxx is compatible with
| Dagger", "xxx can run your Dagger configurations", are
| not allowed._
|
| > _By naming this "Dagger" you're confusing not just your
| own developer community but the pre-existing one as
| well._
|
| I disagree. Dagger has existed in private beta for over a
| year, thousands of engineers have been given access, and
| I can't remember a single instance of any of them being
| confused by the name. We have registered the trademark,
| and nobody has raised an issue.
| EddieRingle wrote:
| > > > We will not rely on unusual licences to restrict
| competitors from running Dagger as a service.
|
| > > Your "Trademark Guidelines" appear to contradict you:
|
| > They do not. Software licenses and trademark guidelines
| are two different things. Some commercial open-source
| vendors have changed their licenses to restrict use of
| the software in various ways - typically to limit
| competition from large cloud providers. We don't do that,
| and have no intention to. Our license is OSI-approved and
| we intend to keep it that way. That is what I am
| referring to.
|
| I'm glad the product is open source, but that provision
| isn't in the context of source code, it is a top-level
| item listed on that page. That's why I interpreted
| "unusual licences" to generally mean a sort of "legal
| acrobatics".
|
| When you're threatening people with legal action you need
| to be clear, and right now the text on that page is not,
| according to what you're saying here. I doubt many people
| are going to be searching Hacker News comments for the
| true intent behind these guidelines.
|
| > I disagree. Dagger has existed in private beta for over
| a year, thousands of engineers have been given access,
| and I can't remember a single instance of any of them
| being confused by the name. We have registered the
| trademark, and nobody has raised an issue.
|
| I don't think that really addresses the point. Dagger (as
| started under Square) is nearly ten years old, and
| Google's 2.0 fork is from 2016. It's used by thousands of
| published Maven artifacts, countless applications, and
| tens of thousands of developers (at least). This is the
| first time I've heard of your project, but that's bound
| to happen in tech. Whether you registered it or not
| without complaint doesn't much matter either, the issue
| is being raised here, now that you've publicly launched.
| shykes wrote:
| > _When you 're threatening people with legal action you
| need to be clear, and right now the text on that page is
| not, according to what you're saying here._
|
| That's good feedback, thank you. We can try and make it
| clearer as long as it remains legally correct and
| enforceable. Do you have specific feedback on which parts
| you found unclear, and why?
| zepn wrote:
| Thanks for answering Qs. Does this compete directly with Tekton
| ( https://tekton.dev/ ), or do you imagine a way the two could
| interoperate? Why choose Dagger over Tekton to power pipelines?
| ghawk1ns wrote:
| Dagger is already a popular dependency injection framework, so
| why choose a name that will be confusing to people who will
| likely use both of these frameworks in their projects?
| guelo wrote:
| And the reason they're both called Dagger is as a play on
| using a DAG, directed acyclic graph, to model dependencies.
| Stealing that wittiness and pretending it's their own is
| pathetic.
| hetspookjee wrote:
| I believe the required creativity for that is not that high
| as it doesn't seem too far fetched. I'd rather call it a
| piece of convergent evolution.
| TimWillebrands wrote:
| These comments are extremely uninteresting. If you care so
| much, please just create an issue on their bugtracker or
| something.
| [deleted]
| bovermyer wrote:
| So basically, if I want to not write Jenkinsfiles but still use
| my company's existing Jenkins installation, I can use Dagger?
| shykes wrote:
| Yes :) You can write one last Jenkinsfile that runs Dagger,
| then do everything inside Dagger. Then you can run the exact
| same Dagger configuration on another CI system, or on your
| laptop. All you need is a Docker-compatible runtime (we run
| everything on buildkit under the hood).
| chologrande wrote:
| > You can write one last Jenkinsfile that runs Dagger
|
| I'm very confused by this sentiment. This approach loses
| the best of existing CI tooling does it not?
|
| Jenkins users lose understanding of what's actually being
| executed, what stage they're in, and how long it took. It
| might provide convenience, and the a (great) benefit of
| running locally the same as in your CI environment, but it
| seems to me this would make it difficult for devs to easily
| understand where/why their build failed, since it just has
| one megastep.
| ______-_-______ wrote:
| I assume if the jenkins-step failed, you'd click a link
| to the dagger UI to see which dagger-step failed.
| Alternatively, never open jenkins at all and instead keep
| a tab open with the dagger UI.
| bovermyer wrote:
| Man, I am SO excited for that! Kudos!!
| verdverm wrote:
| You should be able to drastically simplify you Jenkinsfile(s)
| and have them just invoke Dagger. The issue you may run into
| is when you have different Jenkins nodes for different types
| of work. You could always invoke Dagger on each of these,
| depending on your setup and needs. Where there is a will,
| there is a way, with Jenkins :]
| topher200 wrote:
| Congrats on launching!
|
| How mature is this? We have a 20 person team and we're
| prototyping different pipelines for our next CI/CD pipeline
| (currently Heroku). Is this ready for production workloads?
| shykes wrote:
| We consider Dagger to be beta-quality. It definitely has
| bugs, and APIs are still moving, though we make a big effort
| to limit breaking changes, and help developers migrate when
| breaking changes do occur. We aim to be able to guarantee
| complete peace of mind and continuity for production
| pipelines, but can't make that guarantee yet.
|
| That said, one nice aspect of Dagger is that you don't need
| to rip and replace your entire system: you can use it in a
| localized way to solve a specific problem. Then expand over
| time. It's similar to writing a quick shell script, except
| it's easier to reuse, refactor and compose over time.
| shredder25 wrote:
| Congratulations! I know exactly how this tool will benefit us
| DevOps Engineers as I knew when you did a demo of Docker at
| PyCon 2013, wishing you and your team the best!
| mhitza wrote:
| Hi! I've browsed the docs quickly, and I have a few questions.
|
| Seems to assume that all CI/CD workflows work in a single
| container at a time pattern. How about testing when I need to
| spin up an associated database container for my e2e tests. Is
| it possible, and just omitted from the documentation?
|
| Not familiar with cue, but can I import/define a common action
| that is used across multiple jobs? For example on GitHub I get
| to duplicate the dependency installation/caching/build across
| various jobs. (yes, I'm aware that now you can makeshift on
| GitHub a composite action to reuse)
|
| Can you do conditional execution of actions based on passed in
| input value/env variable?
|
| Any public roadmap of upcoming features?
| shykes wrote:
| > Seems to assume that all CI/CD workflows work in a single
| container at a time pattern.
|
| Dagger runs your workflows as a DAG, where each node is an
| action running in its own container. The dependency graph is
| detected automatically, and all containers that can be
| parallelized (based on their dependencies) will be
| parallelized. If you specify 10 actions to run, and they
| don't depend on each other, they will all run in parallel.
|
| > How about testing when I need to spin up an associated
| database container for my e2e tests. Is it possible, and just
| omitted from the documentation?
|
| It is possible, but not yet convenient (you need to connect
| to an external docker engine, via a docker CLI wrapped in a
| container) We are working on a more pleasant API that will
| support long-running containers (like your test DB) and more
| advanced synchronization primitives (wait for an action;
| terminate; etc.)
|
| This is discussed in the following issues:
|
| - https://github.com/dagger/dagger/issues/1337
|
| - https://github.com/dagger/dagger/issues/1249
|
| - https://github.com/dagger/dagger/issues/1248
|
| > Not familiar with cue, but can I import/define a common
| action that is used across multiple jobs?
|
| Yes! That is one of the most important features. CUE has a
| complete packaging system, and we support it natively.
|
| For example here is our "standard library" of CUE packages:
| https://github.com/dagger/dagger/tree/main/pkg
|
| > For example on GitHub I get to duplicate the dependency
| installation/caching/build across various jobs. (yes, I'm
| aware that now you can makeshift on GitHub a composite action
| to reuse)
|
| Yes code reuse across projects is where Dagger really shines,
| thanks to CUE + the portable nature of the buildkit API.
|
| Note: you won't need to configure caching though, because
| Dagger automatically caches all actions out of the box :)
|
| > Can you do conditional execution of actions based on passed
| in input value/env variable?
|
| Yes, that is supported.
|
| > Any public roadmap of upcoming features?
|
| For now we rely on raw Github issues, with some labels for
| crude prioritization. But we started using the new Github
| projects beta (which is a layer over issues), and plan to
| open that to the community as well.
|
| Generally, we develop Dagger in the open. Even as a team, we
| use public Discord channels (text and voice) by default,
| unless there is a specific reason not to (confidential
| information, etc.)
| mhitza wrote:
| Thank you for the detailed response. I appreciate you
| taking the time. One last question/note.
|
| > Note: you won't need to configure caching though, because
| Dagger automatically caches all actions out of the box :)
|
| Is this strictly because it's using Docker underneath and
| layers can be reused? If so, unless those intermediary
| layers are somehow pushed/pulled by the dagger github
| action (or any associated CI/CD tool equivalent),
| experience on hosting server is going to be slow.
|
| Sidenote, around 2013 I've worked on a hacky custom
| container automation workflow within Jenkins for ~100
| projects, and spent considerable effort in setting up
| policies to prune intermediary images.
|
| Thus on certain types of workflows without any prunning a
| local development machine can be polluted with hundreds of
| images, unless the user is specifically made aware of stale
| images. Does/will dagger keep track of the images it
| builds? I think a command like git gc could make sense.
| shykes wrote:
| > > Note: you won't need to configure caching though,
| because Dagger automatically caches all actions out of
| the box :)
|
| > Is this strictly because it's using Docker underneath
| and layers can be reused?
|
| Not exactly: we use Buildkit under the hood, not Docker.
| When you run a Dagger action, it is compiled to a DAG,
| and run by buildkit. Each node in the DAG has content-
| addressed inputs. If the same node has been executed with
| the same inputs, buildkit will cache it. This is the same
| mechanism that powers caching in "docker build", _but
| generalized to any operation_.
|
| The buildkit cache does need to be persisted between runs
| for this to work. It supports a variety of storage
| backends, including posix filesystem, a docker registry,
| or even proprietary key-value services like the Github
| storage API. If buildkit supports it, Dagger supports it.
|
| Don't let the "docker registry" option confuse you:
| buildkit cache data isn't the same as docker images, so
| it doesn't carry the same garbage collection and tag
| pruning problems.
| chippiewill wrote:
| > Don't let the "docker registry" option confuse you:
| buildkit cache data isn't the same as docker images, so
| it doesn't carry the same garbage collection and tag
| pruning problems.
|
| IIRC doesn't buildkit store its cache data as fake layer
| blobs + manifest?
|
| I don't see how it can avoid the garbage collection and
| tag pruning problems since those are limitations of the
| registry implementation itself.
| shykes wrote:
| You still need to manage the size of your cache, since in
| theory it can grow infinitely. But it's a different
| problem than managing regular Docker images, because
| there are no named references to worry about: just blobs
| that may or may not be reused in the future. The penalty
| for removing the "wrong" blob is a possible cache miss,
| not a broken image.
|
| Dagger currently doesn't help you remove blobs from your
| cache, but if/when it does, it will work the same way
| regardless of where the blobs are stored (except for the
| blob storage driver).
| asim wrote:
| Congrats on getting this far with the new venture and good
| luck. Wishing you all the success in the world!
| [deleted]
| Aeolun wrote:
| After reading this entire post, I'm still left wondering what
| problem this solves for me, beyond fluffy promises of 'everything
| is going to be better'.
|
| At the very least I'd want to see a comparison with what we have
| now, to show me _how_ this is better.
|
| I get that I can try to explore more, but if I don't get a
| compelling reason to do so after reading the introductory post,
| I'm not very motivated to do so
|
| Any motivation I do have completely hinges on the words 'from the
| creators of docker', not on the merits of this particular product
| itself.
| shykes wrote:
| That's fair, it can be hard to find the right balance of high-
| level explanation and technical detail. We tried to solve this
| by tailoring different parts to different audiences:
|
| * The blog post is more high-level. It describes the very real
| problem of devops engineers being overwhelmed with complexity,
| and the promise of a more modular system, but does not provide
| lots of details.
|
| * The dagger.io website does provide more technical detail. For
| example it talks about the 3 most common problems we solve:
| drift between dev and CI environments; CI lock-in; and local
| testing and debugging of pipelines. It also features animated
| code samples.
|
| * The documentation at https://docs.dagger.io go in even more
| details, and walk you through a concrete example.
|
| We do feel that we can do a better job explaining the "meat" of
| the tool. But we decided to launch and continue improving it
| incrementally, in the open. If you have any specific
| suggestions for improvements, please keep them coming!
| nunez wrote:
| It mostly solves this problem:
|
| - write code
|
| - run tests
|
| - commit code
|
| - update CI
|
| - commit
|
| - CI broken
|
| - update CI
|
| - commit
|
| - CI broken
|
| - update CI
|
| - ...
|
| The workarounds for this are generally awful.
|
| For Jenkins, you stage your own instance locally and configure
| your webhooks to use that. It's exactly as terrible as it
| sounds, and I never recommend this approach.
|
| For Travis and Concourse (I think), you can use their CLI to
| spin up a runner locally and run your CI/CD yaml against it. It
| works "fine," as long as you're okay with the runner it creates
| being different from the runners it actually uses in their
| environment (and especially your self-hosted runners).
|
| In GitHub Actions, you can use Act to create a Dockerized
| runner with your own image which parses your YAML file and does
| what you want. This actually works quite well and is something
| that threatens Dagger IMO.
|
| Other CI systems that I've used don't have an answer for this
| very grating problem.
|
| Another lower-order problem Dagger appears to solve is using a
| markup language to express higher-level constructs like loops,
| conditionals, and relationships. They're using CUE to do this,
| though I'm not sure if hiring the creator of BCL (Borgmon
| Configuration Language) was the move. BCL was notoriously
| difficult to pickup, despite being very powerful and flexible.
| I say "lower-order" because many CI systems have decent-enough
| constructs for these, and this isn't something I'd consider a
| killer feature.
|
| I also _also_ like that it assumes Dockerized runners by
| default, as every other CI product still relies on VMs for
| build work. VMs are useful for bigger projects that have a lot
| of tight-knit dependencies, but for most projects out there,
| Dockerized runners are fine, and are often a pain to get going
| with in CI (though this has changed over the years).
| jacques_chester wrote:
| Concourse has `fly execute` which makes the commit-push-curse
| problem go away. It's had it since 2015 or so.
| speed_spread wrote:
| Concourse also has `fly hijack` which is the
| baddest/funniest command of the decade. It's also very nice
| to use, instantly logging you in the remote container of a
| failed build so you can poke around an see what actually
| went wrong and try to run it interactively before fixing
| and re-executing. Much better than poking at things in the
| dark until you hit another issue...
| christop wrote:
| Jenkins lets you replay a Pipeline, with changes, which is
| massively useful -- removing the need to change things
| locally and commit.
| joseph wrote:
| My "workaround", if you can call it one, is to design things
| so they don't need the CI/CD server to get a
| build/test/deploy feedback loop. I should be able to do any
| stage of the pipeline without the server, and thus no code is
| committed until I know it is working. The pipeline is
| basically a main() function that strings together the things
| I can already do locally. If I need anything intelligent to
| happen at any stage of the pipeline, I write a tool to do it
| using Go or Python or something that I can write tests for
| and treat as Real Software. After fighting with this for many
| years, this approach has worked best for me.
|
| I didn't dig deeply into the docs, but Dagger appears to be
| doing a multi stage pipeline locally. If that is the case, I
| wouldn't want that either. I use Concourse, which has very
| good visualizations of the stages, and if I used Dagger
| there, it would consolidate those stages into one box without
| much feedback from the UI. Also, with Concourse you can use
| `fly execute` to run tasks against your code on the actual
| server, without having to push anything to a repo.
| fideloper wrote:
| I run a CI application for Laravel developers (Chipper CI).
|
| It turns out, the gap between "works locally!" and "works in
| CI!" is not negligible, especially when you're not sure "about
| all that server stuff".
|
| Getting this working locally with a fast cycle time, and then
| being able to easily move that into a CI environment of your
| choice sounds exciting to me.
|
| Furthermore, the _majority_ of our customer support is "I
| can't reproduce this locally but it's broken in CI". Everyone
| blames the CI tool, but it's almost never the CI tool - just
| drift between environments. A way to debug locally is a killer
| feature.
|
| Is it worth an entire, funded company? I'm not sure, but I'm
| excited for them to exist!
| austinshea wrote:
| Always nice to have new players in the space, but that gap
| isn't even addressed here.
|
| Same old problems with configs/secrets, integration with
| internal/external services, and the details required by your
| cloud provider.
|
| This is the sort of solution I have regularly been hired to
| untangle, after a company entrenches itself.
| dopidopHN wrote:
| My main takeaway was the ability to debug a build or deploy
| pipeline locally.
|
| That would come handy once or twice per month
| rendall wrote:
| Paul Graham just highlighted it on Twitter:
| https://twitter.com/paulg/status/1509209579414040588
| wdb wrote:
| Interesting, I should check out if this works with Gitlab CI. As
| Gitlab CI is quite a pain to debug (guess all CI environments).
| kristianpaul wrote:
| Just another CI/CD pipeline...
|
| Not saying we should write one [1] but CI/CD are a commodity
| these days.
|
| [1] https://aosabook.org/en/500L/a-continuous-integration-
| system...
| chubot wrote:
| Sigh ... this page is unreadable in Firefox on iPad, and
| unreadable without JavaScript. Either no text, or black on black
| zefrvtgevtrb wrote:
| Could you share more details please ? It would be awesome to
| open an issue on dagger repo :
| https://github.com/dagger/dagger/issues.
|
| Thanks :)
| shykes wrote:
| Sorry about that. We'll fix it asap.
| diggs wrote:
| There's so much more to CI/CD than the build definitions (e.g.
| dashboarding, log viewing/retention, access control, manual
| interventions, secret management, test reporting, etc.) and while
| some of your points resonate very strongly with me (e.g. local
| builds), I can't help but wonder what the endgame is here?
|
| You've raised $27m to what? Give away a free tool for engineers
| to define their builds in? While they continue to use and pay for
| another service to actually run the builds? I assume you intend
| to replace the CI service at some point and move up the stack to
| monetize?
|
| Without more transparency its easy to imagine something like...
|
| Step 1. Drive uptake of your tool by selling people on the
| pitfalls of "CI lock-in" Step 2. Introduce your own CI solution,
| which people can now easily switch to Step 3. Lock people in
| shykes wrote:
| Good question. No, we don't intend to replace the CI service.
| We think of CI as infrastructure, and there are plenty of great
| infrastructure providers out there. And you are absolutely
| correct that for Dagger to succeed, it _has_ to remain agnostic
| to infrastructure providers, which means we _cannot_ build an
| infrastructure provider business ourselves. If the experience
| of hosting Dagger becomes so bad that it affects the developer
| experience, we might ship such a feature as a stopgap and even
| charge for it - but it is not a strategic priority. The sooner
| other infrastructure providers offer a fantastic hosting
| experience, the better. That is not where we see the biggest
| opportunity for Dagger.
|
| However, there is a great opportunity to help businesses manage
| their software supply chain. What is running where, and how did
| it get there? Who is authorized to run which pipelines on which
| machines, and which which inputs? What is the diff between this
| dev environment and staging? Staging and production? Etc. Keep
| in mind this is not limited to your production pipeline; there
| are staging environments, development environments, all running
| pipelines all day long. It's hard to simply know everything
| that is going on.
|
| Each time Dagger runs your pipeline, it can produce telemetry
| about every aspect of your supply chain. Git SHAs, docker image
| checksums and cryptographic signatures, specific versions of
| kubernetes and cloudformation configurations, before and after
| templating, test results, monitoring status... It also
| integrates with your secrets management backends, your
| developer's local source code. Basically every node in your
| supply chain can be a node in your DAG if you want it to. The
| logical next step is to give you a centralized place to send
| all this telemetry, and tools to extract insights from it. You
| could also perhaps manage the configuration of your various
| Dagger engines in one location.
|
| Another product that is often requested is a _visual DAG
| debugger_. When a pipeline break, you want to know why, and
| staring at your CI logs is definitely not the best experience
| for that. With a web UI, there 's a lot we can do there.
|
| The business opportunity boils down to this: if CI/CD pipelines
| are software, there ought to be a platform for developing that
| software, and an ecosystem of developers creating value around
| that platform. Dagger aspires to create that missing developer
| ecosystem. If we succeed, there will be no shortage of business
| opportunities. If we fail, none of the other features will
| matter.
| laurentb wrote:
| Hey Solomon,
|
| question coming from a sales guy among this crowd... Who does
| this impact the most and what outcomes does that help them
| achieve? At the end of the day, you need to pay the bills,
| for you and your team, who is going to sign the dotted line
| and tell you that yes "I need to invest $100k on this because
| it solves a major pain!"?
|
| Whilst I can see this as a nice to have, I'm having a hard
| time understanding who your target market is going to be...
|
| At the end of the day, I presume this is going to end up
| becoming a full fledged company, with its sales team,
| marketing and what not. What are you thinking about in terms
| of revenue channels so far ?
| stingraycharles wrote:
| > Each time Dagger runs your pipeline, it can produce
| telemetry about every aspect of your supply chain.
|
| To me, that still doesn't seem to go to the core of what
| value it brings to table. I understand that Dagger can do all
| this, and that businesses would like to know what runs where
| and how everything interacts, but... it doesn't explain to me
| the CI / build pipeline angle?
|
| How does knowing the telemetry around my build pipelines
| translate to better software, or cost savings, or other
| improvements?
|
| If there's a registry to exchange "components" of a build
| pipeline, what does that bring me what a regular python /
| java / etc package can't?
|
| Don't get me wrong, I think there are plenty of problems with
| CI (I have a ~200 job CI pipeline on fire in front of me), I
| just can't seem to connect the dots here. :)
| taude wrote:
| I didn't spend a lot of time with the site, but I interpreted
| Dagger as the Terraform of CICD definitions....
| shykes wrote:
| Yes, that's a reasonable approximation.
| henrydark wrote:
| Seems like more stuff that Nix solves out of the box
| [deleted]
| speed_spread wrote:
| Is there a thing that Nix can't solve?
|
| This is a CI/CD system. It's not made to configure a single
| machine from coded configuration, but to build, test, publish
| software for multiple targets and then manage deployment to
| different clusters.
| n42 wrote:
| Nix builds, tests and publishes more than 80,000 packages,
| continuously on every commit, for multiple target platforms,
| via one monorepo.. without Docker.
|
| It's not just for building machines, that is NixOS and its
| module system, which is a library you can use with the build
| tool.
|
| It's possible to build your own Nix-based monorepo internally
| at your company, too, and still ship containers in production
| (that are also built by Nix, again, without Docker).
|
| Dagger is not a CI provider, it is a build tool and task
| runner. Integrating Nix with CI is the same deal as
| integrating Dagger with CI.
|
| Enter my biased opinion, as someone leading a team through
| solving the problems this tool purports to solve, but by
| using Nix: this tool will suffer from the same fatal flaws
| that Docker does, by being built on its foundation
| (Buildkit). It is abstracting dependency management at the
| application build level, whereas Nix solves it at the
| fundamental system dependency level.
|
| I would like to be proven wrong, so best of luck!
| politician wrote:
| I'm ready to be an early (&& paying) adopter of this today, but I
| have real work to get done so it would be extremely helpful if
| you could describe what doesn't work well about your solution
| today.
| xmlninja wrote:
| "A developer's dream is a devops engineer's nightmare"
|
| Well, the opposite is very much true as well.
|
| A devops engineer's dream is usually a developers nightmare.
| shykes wrote:
| That is sadly often true today, but not inevitable! When you
| are overwhelmed with complexity, a natural defense mechanism is
| to restrict choice. But with better tooling, you can better
| manage and reduce complexity, which allows you to say "yes" to
| developers without compromising on reliability and security.
| That is what we are trying to enable with Dagger.
| goodpoint wrote:
| So much for "devops" merging together development and
| operations.
|
| "A developer's dream is a devops engineer's nightmare"
| underines the failure of devops.
| Xenixo wrote:
| Ci/CD is transforming right now but I don't think dagger is
| solving those things or I don't understand it well enough yet.
|
| GitHub action is a game changer. CICD finally has a proper UI
| were it belongs: on the repository.
|
| GitHub actions are also much easier than gitlab runner. The basic
| actions are great and probably solve 90% of all normal use cases.
|
| Then we/I see a big trend of going away from self build
| pipelines. Provide Standard Features through convention over
| configuration: the docker image build action checks if it need to
| run on the pipeline and runs isolated by default.
|
| The build packs are providing something like this but they are to
| clumsy/complex.
|
| Then you have the real issue no one is solving:
|
| - fast sync between stages and builds. 1 gb of source code, maven
| cache etc. Still need effort to do it fast. Preferable a
| filesystem which is fast and allows to share snapshots you can
| mount ober the network (it's not a big issue but puts a latency
| limit on how fast you can provide build results and I like it
| fast)
|
| - standard building blocks with retry mechanism. When a 3h build
| fails, restarting is annoying like hell. Pipelines have very
| little resilience.
|
| - unified CICD output: lots of plugins support Jenkins Replacing
| Jenkins needs a new UI (GitHub has it now) which still is not
| unified. There is a thing going on to univy on GitHub through
| GitHub checks.
|
| I think in the next few years GitHub will have solved those
| issues for us. They do exactly what you are looking for. Would
| love to work with them.
|
| They even now inject the repository secret for you. They are slow
| (hello personal access token alternative we are waiting...) But
| steady.
|
| Btw. Before someone says gitlab: the Auto DevOps thing was shitty
| and bad supported. A default case with java and postgres was/is
| not fixed for 3 years.
|
| And those vulnerabilities. The last few years? Nope nope nope.
| lyime wrote:
| How is this different from buildkite
| andrewmcwatters wrote:
| shykes, can you explain or have the folks working on the website
| explain how is this different or better than just using Jenkins?
| shykes wrote:
| Sure, it's better for 3 reasons:
|
| 1. You can use the same pipelines in development and
| automation. Dagger runs on Jenkins, but it also runs on the
| developer's laptop. This solves a common problem which is drift
| between the automation available in dev and CI.
|
| 2. You're not stuck with Jenkins. If you want to migrate a
| Jenkins-native pipeline to another CI, you have to rewrite
| everything. Dagger on the other hand can run on any major CI.
| This makes migrations much easier, as well as supporting teams
| with heterogeneous CI setups (which is quite common in larger
| organizations).
|
| 3. You can debug and test your pipelines locally. In theory
| this is possible with some CI systems. But in practice, the
| experience is very different. You can actually iterate on your
| Dagger configuration like you would on regular code: make a
| change, run, try again, run again. It's quite fun and
| addictive.
|
| EDIT: there is a 4th reason, which is that Dagger uses buildkit
| under the hood (the hidden backend for "docker build"). So
| every action in the DAG is automatically cached when possible.
| This means that your pipeline is tuned for optimal caching out
| of the box: no manual tweaking necessary.
| andrewmcwatters wrote:
| Thank you, I appreciate the brief summary very much.
| zamalek wrote:
| I'm not touching anything Docker anymore.
|
| Here's the scenario: you're the unfortunate soul who received the
| first M1 as a new employee, and nothing Docker-related works. Cue
| multi-arch builds; what a rotten mess. I spent more than a week
| figuring out the careful orchestration that any build involving
| `docker manifest` needs. If you aren't within the very fine line
| that buildx assumes, good luck pal. How long has `docker
| manifest` been "experimental?" It's abandonware.
|
| Then I decided it would be smart to point out that we don't sign
| our images, and so I had to figure out how to combine the `docker
| manifest` mess with `docker trust`, another piece of abandonware.
| Eventually I figured out that the way to do it was with
| notary[1], another (poorly documented) piece of abandonware. The
| new shiny thing is notation[2], which does exactly the same
| thing, but is nowhere near complete.
|
| At least Google clearly signals that they are killing something,
| Docker just lets projects go quiet. I'm now looking at replacing
| everything I've done for the past 4 months with Podman, Buildah,
| and Cosign.
|
| How long before this project lands up like the rest of them?
| Coincidentally, we were talking about decoupling our CI from
| proprietary CI, seeing this was a rollercoaster of emotions.
|
| [1]: https://github.com/notaryproject/notary [2]:
| https://github.com/notaryproject/notation
| shykes wrote:
| I can sympathize, but want to clarify that, although I did
| found Docker, I left 4 years ago. Dagger is not affiliated with
| Docker in any way.
|
| Also: Dagger can integrate with Podman, Buildah and Cosign :)
| pc86 wrote:
| Are you me?
|
| I just started a new job a few weeks ago, and guess what, I'm
| the first one (in my small group of ~5 or so) to have an M1.
| First it was just figuring out what the issue was, then trying
| UTM and manual port forwarding and getting horrible
| performance, then Colima which seems passable, but the default
| 2GB of RAM is useless for anything like kafka, so I end up
| having to allocate half the RAM of my machine just to docker
| containers.
| 411111111111111 wrote:
| I find it hilarious that you're both seemingly blame _docker_
| for these issues.
| zamalek wrote:
| The whole M1 thing is definitely Apple's fault, but it did
| lead me to a huge amount of bitrot in Docker tooling.
| pc86 wrote:
| I don't blame anyone for the issue. Well, _maybe_ Apple
| itself but even then blame is pretty harsh. But from
| someone coming from 12+ years of software development on
| Windows, with occasional Mac usage personally, to suddenly
| doing development on a Mac, it was a big shock (mostly in a
| good way).
| vhanda wrote:
| I'd recommend renting / buying a powerful (x64?) machine
| accessible over the internet.
|
| Ever since I did that docker on Mac became bearable. That
| coupled with vscode remote development even allows me to
| easily mount volumes.
|
| Plus I'm now less concerned about malicious 'npm post install
| scripts' which could potentially nuke all my data.
| zamalek wrote:
| You might also want to try Rancher Desktop. At the time I was
| solving things RD had some annoying bugs (so we are using
| colima for now), but it has come a long way and seems to
| handle networking a bit better than colima.
|
| I plan to post a blog about the whole lot. Shoot me an email,
| or a tweet, or something and I'll ping you when I get round
| to it.
| billconan wrote:
| > Run on any Docker-compatible runtime.
|
| so does its worker run on a native machine, because we need to
| build docker images using CI/CD, which we can't do easily within
| a container?
|
| also, is there a gui?
| shykes wrote:
| > so does its worker run on a native machine, because we need
| to build docker images using CI/CD, which we can't do easily
| within a container?
|
| `dagger` is a client tool, which uses buildkit as its worker.
| Buildkit itself can run on a native machine, or inside a
| container. Either way, building docker images is one of the
| most common actions performed by Dagger, and it is very well
| supported.
| shykes wrote:
| > also, is there a gui?
|
| Not at the moment, but we plan on offering one (optionally) as
| a web service.
| nunez wrote:
| Hell yes. A build system that I can test locally without hacks
| (like act) is literally the dream.
| tofflos wrote:
| Have you given IDEs any thought?
|
| I feel like it isn't write once run anywhere until I can use the
| same configuration inside the IDE, from the command-line, and in
| CI.
|
| The IDE would need to understand what the dependencies are in
| order to provide things like auto-completion, debugging and
| syntax-highlighting.
| Svoka wrote:
| So if it is a "devops engineer" why not call them just "ops" if
| they don't do software engineering?..
| philipswood wrote:
| It's just software engineering with more obstinate, less
| tolerant components and without an IDE.
| mdaniel wrote:
| > without an IDE.
|
| There's an IJ plugin for cuelang but I'm not at my desk to
| know if it would be helpful for use with Dagger or not
| bckr wrote:
| Because the point of their role is to not do the same manual
| work over and over but to gain leverage over time by _dev_
| eloping software to automate _ops_
| bovermyer wrote:
| The name war continues unabated.
|
| I'm a "devops engineer." Besides pipeline code and
| maintaining/building out infrastructure, I also write CLI
| applications to handle complex infrastructure tasks. At what
| point is a "devops engineer" not a software engineer?
|
| [p.s., I prefer the term "platform engineer," personally]
| chadcmulligan wrote:
| So you spend all your time looking after dev infrastructure?
| Can I ask how many devs there are where you work?
| dijksterhuis wrote:
| Historically the devops name comes from the methodology where
| "dev" and "ops" teams got merged so there'd be at least 1x
| dedicated ops person for each team of developers, only focussed
| on that team's operational needs.
|
| Point was to break out of the Developers versus Operations
| silos and actually be more productive as a single team.
|
| Back in the silo'd days... the ops teams would just flat out
| refuse to release _any_ code on a Friday. Which doesn 't work
| if you're twitter. Hence the switch.
|
| A lot of my time as a "devops" engineer is reiterating that I
| can fix and automate as much as you want me too, but if your
| developers don't think about the infra the code eventually runs
| on in prod then your software is just gonna keep breaking ---
| "but it worked fine on _my local machine_ ".
|
| So, while I agree the "Devops Engineer" name is a bit
| ridiculous, and not in the spirit of the original methodology,
| half of my job is writing software that tries to help
| developers not break production and the other half is
| attempting to change the _culture_ , even slightly, towards
| "oh, wait, will this run _in production_? ".
| nickysielicki wrote:
| I'm highly jealous of teams that can containerize all their build
| tools.
|
| If you deal with proprietary toolchains that are tens of
| gigabytes (Windows WDK, Xilinx Vivado/Vitis) it's just untenable,
| and that's before even mentioning licensing. Even Azure doesn't
| have a great solution for WDK development. It's hard to feel like
| we're not being left behind.
|
| Bind mounting the tools into the container is an option but at
| that point you're using the container just because that's what
| your CI expects, not because it's any better or more reproducible
| than a raw Jenkins shell script.
| jaaron wrote:
| 100% this.
|
| I feel like there's a ton of innovation in the cloud/container
| CI/CD space, but next to nothing elsewhere. In fact, some of
| the innovations in CI/CD make things _more_ difficult for those
| of us developing in other environments (such as game
| development).
|
| There's a lot of low hanging fruit for improving things
| elsewhere.
| lazypenguin wrote:
| Yes! I'm responsible for a game development CI/CD pipeline
| that needs to run on windows and I feel like it's harder than
| it needs to be in 2022.
| jeremy_k wrote:
| Do you work in game development? I would love to hear a
| little bit more about your experience. I play and follow Apex
| Legends and I'm always so curious about how bugs and
| regressions seem to make it into every one of their patches.
| As well as tons of new information that gets data-mined.
|
| To me, its like they don't have branches (tons of new code
| not accessible in the game is released in a patch) and they
| don't have unit tests that catch bugs (a certain ability has
| 2 modes, with 2 different activation times. In the most
| recent patch, the activation times ended up the same) but
| it's possible that developing a game is far different than
| developing a SaaS product from a coding perspective. Or it
| could be that studio just has weird practices.
| judge2020 wrote:
| > As well as tons of new information that gets data-mined.
|
| Client side encryption is basically impossible; the
| decryption key has to be in the executable you ship
| somewhere, or at least sent to the client after a
| connection with the server. Perhaps some encryption tiering
| system could work to keep unreleased locked content locked
| for longer but I don't think anyone has gone through that
| trouble yet.
| theraven wrote:
| You can add Apple toolchains to that list also
| bnert wrote:
| preach
| simulate-me wrote:
| In my personal experience, Windows development is absolutely
| awful. Nothing about it is ergonomic, and the management of the
| development environment is just painful. Obviously you don't
| always have a choice, but creating a nice development
| experience is clearly not a priority for Microsoft. One really
| basic example: I was deploying files to a Windows Server box
| using scp, but my internet connection died midway through the
| transfer. All subsequent deployments would fail until I could
| connect via RDP to kill the old SSH process holding an open
| file handle. Another example: I bought into the WSL hype,
| hoping it would solve all my problems. But you can't easily
| interact with files on the system outside of Linux subsystem,
| which made it more or less useless for my purposes. If I wanted
| to interact only with Linux files, I would have used a Linux-
| based server. Maybe I could get better at Powershell, but it
| felt overly verbose while also being less composable /
| powerful. The simple posix shell primitives were sorely missed.
| Running things on startup was also insane. I think I needed to
| add a triggered job that ran on user log in, and then configure
| the host to login a specific user when the host starts. Very
| crazy...
| judge2020 wrote:
| > But you can't easily interact with files on the system
| outside of Linux subsystem, which made it more or less
| useless for my purposes.
|
| It's possible in file explorer via \\\wsl$, but that is not
| always supported by applications so it's not 100%.
| Aeolun wrote:
| This bait and switch in the docs when trying to see an example
| for anything other than Github feels a bit off to me:
|
| > If you would like us to document GitLab next, vote for it here:
| dagger#1677
|
| If you don't have an example for a specific tool, just don't add
| it to your documentation.
| vasco wrote:
| Disagree, it's a perfect way to learn about interest from
| customers by counting votes or just traffic to that page. This
| way they can learn which things to implement first with very
| little impact to you, while giving you a way to help them know
| you want a Gitlab integration. What's the alternative, you want
| to have to email them to ask about it or wonder if they're
| considering it vs Bitbucket or a bunch of alternatives?
| shykes wrote:
| To clarify, you can run Dagger on Gitlab today. It just
| requires some manual configuration that we would like to
| automate away, for convenience. We have only done this for
| Github so far, and would like to do it for more.
|
| We will look for a way to make this more clear in the
| documentation.
| BillSaysThis wrote:
| What about Bitbucket?
| shykes wrote:
| The only dependency for running Dagger is buildkit - which
| itself requires a docker-compatible container runtime. This
| could be containerd, runc or of course Docker engine. If
| your environment can run Docker containers, it can run
| Dagger.
|
| There's another aspect which affects performance: cache
| persistence. Buildkit needs a place to persist its cache
| across runs. By default, it relies on local storage. If
| that is wiped between runs (common in CI environments),
| everything will still run, it will just be slow - possibly
| very slow. Luckily, buildkit supports a variety of cache
| backends, including a Docker registry. In practice,
| persisting the cache is the most labor-intensive part of
| integrating in your CI, even though it's technically not a
| hard dependency. This is the part that we want to automate
| away for users, but it requires additional engineering work
| - hence our asking the community for input on what to
| prioritize.
| [deleted]
| cube2222 wrote:
| TLDR this basically looks like portable GitHub Actions +
| Workflows if I understand correctly.
| shykes wrote:
| Yes that is a good summary. It has other advantages beyond its
| portability - but that is the central feature.
|
| Also, importantly, it can run _on_ Github Actions itself!
| daenz wrote:
| >Write your pipeline once, run anywhere
|
| Ha, finally. The timeline for a pipeline that I wrote recently
| looks like this: 1. Write local test/deploy
| script 2. Promote scripts to hosted CI system 3.
| Local scripts rot 3. CI system down, need to use local
| scripts again 4. Re-write scripts to be current 5.
| Force CI system to use my local scripts
| abledon wrote:
| I'm getting "AmongUs" vibes from this site. Its called 'Dagger',
| and the illustrations are an anonymously clad astronaut running
| around a space station.
| smrtinsert wrote:
| I remember reading about composable pipelines a while ago. I
| remember thinking this is going to take the devops world by
| storm, hopefully Dagger brings us closer to that dream.
| gregmac wrote:
| Your Windows instructions/process needs work.
|
| > curl https://dl.dagger.io/dagger/install.ps1 -OutFile
| install.ps1
|
| This uses the `curl` alias, which is really `Invoke-WebRequest`.
| It also makes the incorrect assumption I haven't fixed this dumb
| Microsoft mistake to be an alias to actual curl.exe.
|
| > [Windows] We try to move the dagger binary under
| C:\Windows\System32
|
| Ack, please don't do this! This is similar to installing
| something under /usr/sbin/. Malware is the only modern thing that
| would attempt to deploy to that folder.
|
| > but in case we miss the necessary permissions, we'll save
| everything under <your home folder>/dagger > C:\<your home
| folder>\dagger.exe
|
| I'm glad you have a non-admin fallback, but also: yuck. I don't
| want this polluting my home folder (more importantly: I don't
| want 100's of other things like this also polluting my home
| folder).
|
| The "Windows way" is to install system-wide to
| %ProgramFiles%\dagger\ (eg c:\Program files\dagger\dagger.exe),
| or to install to %LocalAppData%\dagger\ (eg:
| c:\Users\shykes\AppData\Local\dagger\dagger.exe). The latter is
| kind of the closest equivalent to $HOME/.dagger on linux. Add
| whatever folder to the user's PATH environment variable to make
| it easy to run.
|
| Honestly, providing just the .zip is better: then Windows users
| can muck up their own system however they like. Alternatively,
| package it with something like Scoop [2] which is a fairly
| popular dev tool, and provides a fairly easy way to get a sane
| install with updates, versioning and path stuff all handled.
|
| [1] https://docs.dagger.io/
|
| [2] https://scoop.sh/
| scubbo wrote:
| > This is similar to installing something under /usr/sbin/
|
| As someone who's trying to get to grips with the Linux
| filesystem conventions, would you mind elaborating on a) why
| that's wrong, and b) what you would suggest instead? This
| reference[0] suggests that `/usr/sbin` is for "general system-
| wide binaries with superuser (root) privileges required" (and
| `/usr/bin` for those that don't require root privileges). I've
| therefore been using them in my homelab for binaries like the
| Cloudflare Tunnel Client[1]. Where instead should I have
| installed it?
|
| * If to a "well-known location" that is commonly-used by
| convention, how should I find out what that is?
|
| * If to a custom location of my choosing, how should I
| communicate its location to scripts/tools that _use_ that
| binary? I see later in your comment that you suggest "Add
| whatever folder to the user's PATH environment variable to make
| it easy to run.", but that doesn't seem like a scalable
| solution for a multi-user environment?
|
| [0] https://askubuntu.com/a/308048
|
| [1] https://github.com/cloudflare/cloudflared
| Nullabillity wrote:
| Generally, `/usr` is for stuff packaged by your distribution,
| while `/usr/local` (so, `/usr/local/bin` and so on) is for
| your own custom stuff. Both the `/usr` and `/usr/local`
| equivalents will be on your $PATH by default in most distros.
|
| For stuff that isn't just a self-contained executable,
| consider installing it to the folder `/opt/$MY_APP` and
| either symlinking the main binary into `/usr/local/bin` or
| putting a wrapper script in there (if the thing doesn't like
| being symlinked outside of its primary install dir). The
| wrapper can be as simple as: #!/bin/sh
| exec /opt/my-app/bin/my-app "$@"
| pc86 wrote:
| OT but is Scoop competing with Chocolatey? I'm having trouble
| seeing the difference and I've never heard of Scoop before
| shykes wrote:
| Thank you for the feedback! I referenced it in an issue here:
| https://github.com/dagger/dagger/issues/1946
| vardrop wrote:
| EDIT: You can now directly install dagger with scoop by
| running scoop install dagger
|
| I have opened an issue and PR with scoop [0], see also [1].
| You can directly install dagger with scoop meanwhile by using
| scoop install https://gist.github.com/vardrop/a25e0c8e2dc055f
| 86a3ff4dd7a7de309/raw/0b7f1e29454d7d2cbe1ae9d6c807ddfcdacb7fe
| b/dagger.json
|
| [0] https://github.com/ScoopInstaller/Main/issues/3460
|
| [1] https://github.com/dagger/dagger/issues/1946#issuecomment
| -10...
| no_circuit wrote:
| During the past few months I wrote a CI/CD pipeline using GitHub
| actions, Terraform, and Kubernetes. I'm not too sure Dagger would
| have saved that much time for me unless there was already a fully
| setup service already being sold.
|
| The main issue see so far is that I'd also want bringing up a CI
| instance of the service the same was as new prod (backup)
| cluster. All of that is written in Terraform already (and GitHub
| Actions). Why rewrite it at this point? If I need stats I could
| probably push them to Prometheus.
|
| So yeah, ha, sounds like just writing something like Dropbox
| would be easy on Linux. But it could be great if I wouldn't need
| to drop down to Bash Run all the time, since a lot of official or
| community extensions would need to be created:
| https://docs.dagger.io/1202/plan#plan-structure. Somewhat reminds
| me a bit of Earthly: https://earthly.dev/.
| icecap12 wrote:
| Glad someone mentioned GH Actions. Recent SaaS issues aside,
| one of the value-props to me with GH Actions is that I can
| leverage it in a multi-cloud environment if I'm using GitHub
| for SCM, which my organization is. There are some other
| benefits I specifically care about that others might not as
| well.
|
| Infra is still largely a separate concern (as in, it doesn't
| matter what I use, I have options). I suppose I need to look
| into Dagger more to understand the value-prop.
| simulate-me wrote:
| What is the benefit of using a new bespoke syntax vs. running
| docker containers directly? Drone.io does this very well.
| shykes wrote:
| One way or the other, you will have to write a declarative
| configuration describing which containers to run, and how to
| interconnect them. Then you'll want to add templating to that
| configuration, perhaps a schema verification tool, some sort of
| test scaffolding, etc. Our philosophy is that, if you're going
| to do all those things, it's better to use a dedicated platform
| with a complete syntax, API and integrated tools, instead of
| cobbling it together yourself.
___________________________________________________________________
(page generated 2022-03-30 23:00 UTC)