[HN Gopher] Dagger Python SDK: Develop Your CI/CD Pipelines as Code
___________________________________________________________________
Dagger Python SDK: Develop Your CI/CD Pipelines as Code
Author : yurisagalov
Score : 83 points
Date : 2022-11-10 17:23 UTC (5 hours ago)
(HTM) web link (dagger.io)
(TXT) w3m dump (dagger.io)
| meling wrote:
| This looks awesome! I've built similar CI logic using git and
| docker apis for an autograder system. Looks like I can throw out
| some of this code... and replace it with dagger.
| tatoalo wrote:
| This is something I'd love to try on my personal projects.
|
| I recently used act[0] to locally test my GitHub actions
| pipelines and worked okay, the fact that I could interact with
| the Dagger API via a Python SDK could be even more convenient,
| will definitely try!
|
| [0]: https://github.com/nektos/act
| epgui wrote:
| Even if this was a great, well-built tool, which I am sure it
| is...
|
| Doesn't this introduce a whole dimension of extra stateful
| complexity compared to configuration YAML?
| rytis wrote:
| But when we have control structures built on top of yaml (like
| ansible's `loop`, or `when`), we might as well just use proper
| programming language.
| xedx wrote:
| This is an absolute god send for data engineers! We often get
| blocked full stop with CI/CD headaches and the alternative of "no
| CI/CD" is just unbearable at times. This needs to catch on quick
| !
| bilalq wrote:
| Haven't heard of Dagger before, but using IaC as code to model
| your pipelines (including pipelines of infra that itself is
| modeled as IaC) is absolutely the right way to do things. And
| doing it in a real programming language rather than some limiting
| yaml schema is so much nicer. Places I've worked at in the past
| would refer to these as meta pipelines or self-mutating
| pipelines.
|
| For this reason, I love using the AWS CDK. Being able to model
| things in TypeScript was so much nicer than the janky flow of
| always feeling lost in some Ruby/JSON/YAML IaC template
| monstrosity.
|
| Curious how Dagger differentiates itself from AWS CDK or cdktf.
| shykes wrote:
| Dagger and IaC tools (AWS CDK, Terraform, Pulumi) are very
| complementary. Here's a livestream of Nic Jackson (devrel at
| Terraform) using Dagger and Terraform CDK together to build and
| deploy an application:
| https://www.youtube.com/watch?v=iFNe5W1o2_U
|
| The main difference is that Dagger focuses on running your
| entire CI/CD pipeline as a DAG of operations running in
| containers; whereas IaC tools focus on managing infrastructure
| state. The typical integration is Dagger would run your IaC
| tool in a container, and integrate that into a broader pipeline
| with well-defined inputs and outputs. Dagger itself is
| stateless except for its cache: infrastructure state management
| is handed off to your IaC tool.
| travisgriggs wrote:
| Years ago, I had inherited a complicated C code base with some
| clever Makefile jujitsu. None of the recent maintainers
| understood it at its core. They just put a little lipstick here,
| and a little there to keep things running.
|
| I sat down and extracted the compiler/link flags, and then wrote
| a Python script to do the build. The code was smaller, and built
| faster.
|
| Every "build" engine evolves from being a simple recipe processor
| to the software equivalent of a 5 axis CnC mill. Some things
| should not succumb to one size fits all.
| pm wrote:
| So far I've been using Earthly for structuring builds, and it's
| been brilliant for the most part. Dagger was interesting but a
| no-go without Python, so curious to see how it stacks up.
| chologrande wrote:
| This post is for the python sdk, but the golang SDK is fantastic
| too.
|
| Complete gamechanger to be able to move away from jenkins groovy
| CPS hell
| kpen11 wrote:
| Here's the demo video linked in the blog post:
| https://www.youtube.com/watch?v=c0bLWmi2B-4
|
| It goes step by step through the getting started guide from the
| Dagger Python SDK docs
| helderco wrote:
| Hey, lead developer for the Python SDK here.
|
| I have a few examples in https://github.com/helderco/dagger-
| examples and I plan to add more.
|
| There's also reference documentation in https://dagger-
| io.readthedocs.io/ so you can get a birds eye view on what's
| possible.
| JackMcMack wrote:
| The api looks very simple and easy, that's a very good sign.
| We've been doing similar stuff in python, mainly in combination
| with Pulumi automation.
|
| You can do some very fancy things, and I'm sure Dagger would be
| a nice addition to that. But honestly the thing I'm missing
| most is a nice UI above all that. How to present realtime and
| historic logs (and state) for multiple concurrent actions is
| still not as easy as I think it could be.
| anentropic wrote:
| Haven't tried it yet, but I like the features!
| shykes wrote:
| Hi everyone, Dagger co-founder here. Happy to answer any
| questions.
|
| We also released an update to the Go SDK a few days ago:
| https://dagger.io/blog/go-sdk-0.4
| ath3nd wrote:
| It looks awesome! I have a clarification question on this one:
|
| > Using the SDK, your program prepares API requests describing
| pipelines to run, then sends them to the engine. The wire
| protocol used to communicate with the engine is private and not
| yet documented, but this will change in the future. For now,
| the SDK is the only documented API available to your program.
|
| Does it mean the sdk is making a round trip to the dagger API
| remotely somewhere, or is the round trip to a locally running
| docker container?
| dolanor wrote:
| Hi, on the locally running engine.
| pydry wrote:
| A) Why async in the user code? Is it really necessary?
|
| B) Can you mock pipeline events?
| bennine wrote:
| Hey there, this looks interesting. What's the scope of this
| project? Is it meant to be closer to developers or do you see
| this being used in production?
|
| How is progress of builds, observability, etc being tackled?
| rekwah wrote:
| Curious if cuelang just ended up being too much of a hurdle for
| onboarding. I like it and have used it quite a bit but there's
| something about the syntax that makes it impenetrable for many.
| shykes wrote:
| There's some of that. CUE is incredibly powerful, but it can be
| polarizing. But the fundamental problem is that developers
| don't want to learn a new language to write CI/CD pipelines:
| they want to use the language they already know and love.
|
| So, no matter what language we had chosen for our first SDK, we
| would have eventually hit the same problem. The only way to
| truly solve the "CI/CD as code" problem for everyone, is to
| have a common engine and API that can be programmed with
| (almost) any language.
| dith3r wrote:
| how this compares to buildbot or gajapipelines
| mkesper wrote:
| Might be an interesting middle ground between the nightmare of
| everthing-as-plugins-but-flexible Jenkins and something like
| tekton.
| diarrhea wrote:
| I've long wished to be just writing (ideally) Python to define
| and run CI/CD pipelines. YAML is simply hell, `if` keys in such
| data description languages are cruel jokes.
|
| Round-trip times for GitHub Actions are too high: sometimes
| you're waiting for 10 minutes just to run into a dumb typo, empty
| string-evaluated variable or other mishap. There's zero IDE
| support for almost anything beyond getting the YAML syntax itself
| right.
|
| We have containerization for write-once-run-anywhere and
| languages like Python for highly productive (without footguns as
| bash has them), imperative descriptions of what to do. The main
| downside I see is it getting messy and cowboy-ish. That's where
| frameworks can step in. If the Dagger SDK were widely adopted,
| it'd be as exchangeable and widely understood/supported as, say,
| GitHub Actions themselves.
|
| We currently have quite inefficient GHA pipelines (repeated
| actions etc.) simply because the provided YAML possibilities
| aren't descriptive enough. (Are Turing-complete languages a bad
| choice for pipelines?)
|
| What's unclear to me from the article and video is how this can
| replace e.g. GitHub Actions. Their integration with e.g. PR
| status checks and the like is a must, of course. Would Dagger
| just run on top of a `ubuntu-latest` GHA runner?
| ripperdoc wrote:
| I can only agree, sitting with this right now and it's all so
| brittle. Of course, it's not just YAML, it's Javascript,
| Powershell and Bash embedded into strings plus various hacky
| methods of escaping, templating and sending values between
| steps. It's really a mess.
| blowski wrote:
| Think about it from the perspective of the provider of the
| build platform (e.g. CircleCI, GitLab, GitHub). There are way
| fewer edge cases parsing YAML files than allowing Turing
| complete languages.
| pydry wrote:
| >Are Turing-complete languages a bad choice for pipelines?
|
| I'd say that non turing complete languages are a bad fit for
| pipelines. Even mildly complex pipeline will eventually have
| loops and conditionals.
|
| Better python than some originally-a-config-YAML language
| turned into an imperative monstrosity with loops and
| conditionals bolted on.
| courgette wrote:
| Not a snarky question : what about having data and config in
| yml, and some high level tool like ansible or terraform and
| the occasional bash?
|
| You loop and branch in ansible/terraform
|
| I hate that setup but also I have a hard time thinking of
| something else
| duped wrote:
| I don't think there's a meaningful distinction between
| configuration and code in a CI pipeline. This is what
| people try to do, and it's frankly a massive waste.
|
| The problem is the attempt to make a distinction when there
| cannot be one.
| chologrande wrote:
| Looping in ansible/terraform is exactly the problem. Logic
| and control flow in YAML/hcl is a nightmare. Plus there's
| no debugging! You can't set breakpoints in a yaml file or
| HCL.
|
| Adding more YAML to be parsed by other YAML is just
| terrible at scale.
| lmm wrote:
| Something like Dhall might be viable for pipelines. I wonder
| if there are any efforts in that direction.
| shykes wrote:
| > _What 's unclear to me from the article and video is how this
| can replace e.g. GitHub Actions. Their integration with e.g. PR
| status checks and the like is a must, of course._
|
| You guessed correctly: Dagger does not replace Github Actions,
| they are complementary. The Dagger project itself uses Github
| Actions and Dagger together :)
|
| > _Would Dagger just run on top of a `ubuntu-latest` GHA
| runner?_
|
| Sure, you can do that. The only dependency to run Dagger is
| Docker or any OCI-compatible runtime. So, assuming the `ubuntu-
| latest` runner has docker installed, you can just execute your
| Dagger-enabled tool, and it should work out of the box.
|
| For example here's our own github workflow for testing the
| Python SDK (you can look around for other workflows):
| https://github.com/dagger/dagger/blob/main/.github/workflows...
|
| Note that the word "dagger" doesn't even appear, since Dagger
| is embedded as a library (in this case, using the Go SDK). As
| far as GHA is concerned, it's just executing a regular binary.
| wmichelin wrote:
| I don't know if this is just scoped to my company, but drone
| seemingly supports starlark, which is just python
| madjam002 wrote:
| I had a look at Dagger, but I'm having a lot of success recently
| just building CI/CD pipelines with Nix. My `.gitlab-ci.yml` file
| is then just a thin wrapper of `nix build` commands, and
| pipelines can be run locally in the exact same environment as
| CI/CD.
| mkoubaa wrote:
| While I prefer python over yaml, I tend to think imperative build
| systems are inherently more brittle.
| shykes wrote:
| Dagger remains a declarative DAG-based system. The trick is
| that you can control it with a (also declarative) API. This
| gives you the best of both worlds: a robust declarative
| environment to run your pipelines, accessible from your
| favorite language (imperative or not).
| aclatuts wrote:
| This makes a lot of sense but would it prevent developers from
| using the default diagrams already built into most CICD websites.
| I assume migrating to this would make the CICD pipeline look like
| one step from gitlab, bitbucket, and githubs perspective.
| shykes wrote:
| It's up to you how granular you make your CI configuration.
| Much of it depends on the context and how your team works.
|
| If you've already found yourself integrating a Makefile in a CI
| job, and figuring out the best mapping of Make rules to CI
| job/step/workflow: this is exactly the same. Ultimately you're
| just executing a tool which happens to depend on the Dagger
| engine. How and when you execute it is entirely up to you.
|
| For example, here's the Github Actions job we use to test the
| Dagger Python SDK. It executes a custom tool written in Go. hht
| tps://github.com/dagger/dagger/blob/bd75d17f9625f837d7a2f9...
| pianoben wrote:
| (NB: No relation to the widely-used Android/Java DI library)
___________________________________________________________________
(page generated 2022-11-10 23:01 UTC)