[HN Gopher] Kargo, a multi-stage application lifecycle orchestrator
___________________________________________________________________
Kargo, a multi-stage application lifecycle orchestrator
Author : KenCochrane
Score : 98 points
Date : 2023-09-18 14:38 UTC (8 hours ago)
(HTM) web link (akuity.io)
(TXT) w3m dump (akuity.io)
| esafak wrote:
| I'm not a CI/CD specialist, so help me out. Is this something
| that is more suitable for large, complex setups or is it
| something smaller shops should be considering too? How does it
| compare with competitors like Spinnaker?
| Havoc wrote:
| Sounds promising. Vaguely terraform like in its application to
| outside subscription services.
|
| Can't say I'm a fan of "we call it freight" though. Artefact is a
| perfectly fine word.
| therealkrancour wrote:
| Freight often contains multiple artifacts. Container would have
| been a nice word, but it's taken. ;)
| morey-tech wrote:
| I tend to agree, the abstraction from common language (e.g.
| artifact) is fun because it stays on theme with the "kargo"
| name but can make it difficult for new users to understand what
| is happening. I've always loved that in Argo CD, it's just an
| Application. No special name for something users are often
| going to refer to.
| manojlds wrote:
| Kargo (cargo) and Freight makes sense no?
| hermanb wrote:
| If Kargo requires R/W access to GitHub, and auto updates
| charts/images, isn't that asking for your production environment
| to be infected by a change prepared and cultured in your dev
| environment and then auto updating / hiding itself into prod
| freight?
|
| We disallow writing back to GitHub to avoid this issue, and
| manage stages through branches, combined with directories for
| overlays. Things can get out of sync, but comparing branches is
| easily automated.
| cced wrote:
| Couldn't this be mitigated against by using something like
| codeowners and only allowing access to manage versioning files?
| KenCochrane wrote:
| For those who are curious here is the Github repo
| https://github.com/akuity/kargo
| freedomben wrote:
| As of writing this, there are 5 total comments, 4 of which are by
| very inactive or brand new accounts with no activity, adding
| unhelpful/low-effort posts like:
|
| > Something like this is definitely needed in the GitOps
| space...always felt like something was missing between promoting
| things and rolling them out
|
| > Interesting. Will share with my team and try it out.
|
| > There is a webinar tomorrow with Kelsey Hightower! Here is the
| link if you want to join https:...
|
| > Looks promising, I'm definitely going to take it for a spin!
|
| Something tells me this isn't organically gaining traction...
| [deleted]
| [deleted]
| pvg wrote:
| If you suspect abuse, email the mods since this sort of meta
| tends to go to thread hell.
| [deleted]
| [deleted]
| narrator wrote:
| I mistakenly read that as "Kargo, a multi-stage .... lifestyle
| orchestrator" and thought, damn, that sounds interesting!
| spacelad101 wrote:
| Environment promotion is an area that has been severely lacking
| with GitOps tools. I've hacked together CI pipelines to do
| environment promotion with Argo CD before and it was far from my
| preferred approach, but I could never find a good GitOps way of
| handling it. Might check out the webinar tomorrow to see if this
| is something worth trying out.
| [deleted]
| tonnydourado wrote:
| > Unlike CI, Kargo deployment pipelines are not generic "jobs"
| (...). Instead, Kargo Stages are used to model your environments
| (...)
|
| I didn't see an explanation of how Stages are different than
| jobs. Every single usage of Stage could have been replaced with
| job and the meaning would stay the same.
|
| The data and DevOps marketing people really need to drop the
| buzzwordism.
| nickjj wrote:
| What about having a git repo that has a Kustomize base which has
| overlays for different environments?
|
| This way each environment is in its own directory which can have
| its own patches such as using a private load balancer instead of
| public for a staging environment or setting whatever environment
| variables that need to be different.
|
| Then at the Argo CD level, Argo CD running in prod will look for
| an Application in the prod/ directory and Argo CD running in
| staging will look for an Application in the staging/ directory.
|
| All in all you end up deploying the same artifact across
| environments and all of this is managed and exposed by Argo CD.
| For example you might want AutoSync enabled for staging so
| developers can auto-deploy their code but prod requires a tech
| lead or manager to push the sync button to deploy it.
|
| The above works well in practice and isn't too complicated to
| pull off.
| therealkrancour wrote:
| I am a lead on Kargo. That's a great approach you've described
| and Kargo helps you make that approach work even better. It's
| not trying to re-solve problems that are already solved well by
| config management tools or by Argo CD. It provides a higher
| layer that integrates with those to orchestrate the progression
| of changes from one environment to another...
|
| So in your example, suppose you changed something in your
| Kustomize `base/` directory. With _just_ Kustomize and Argo CD,
| those changes roll out to all environments all at once. With
| Kargo, you can progress the change and (re-)validate it
| environment-by-environment.
| nickjj wrote:
| > With _just_ Kustomize and Argo CD, those changes roll out
| to all environments all at once. With Kargo, you can progress
| the change and (re-)validate it environment-by-environment.
|
| In prod it wouldn't get rolled out until someone manually
| clicks the sync button for each app in that environment. But
| yes, in staging it would get AutoSync'd to all of the apps. I
| get what you're saying though.
|
| It will be curious to see how Kargo works across multiple
| clusters in the staging vs prod example where both clusters
| each have their own Argo CD instance running but you want to
| view and promote an artifact across N clusters.
|
| In any case, I'll give it a look when it's available to see
| how it can improve my current workflows. Overall I'm really
| happy I started using Argo CD ~2 years ago.
| therealkrancour wrote:
| > In prod it wouldn't get rolled out until someone manually
| clicks the sync button
|
| That's a great point and if that works for you, awesome!
| Note though that relies on the state of the prod
| Application to temporarily contradict the source of truth
| that is your repo. If you lose that prod Application and
| re-create it, prod will be in a state you hadn't yet
| blessed.
| nickjj wrote:
| Indeed. That use case hasn't been something I encountered
| yet since we generally operate at a scale where our apps
| at the Argo CD level rarely change once they've been
| created.
|
| But thanks for pointing out the mismatch on the source of
| truth. Definitely something to consider.
| spacelad101 wrote:
| This approach definitely works, but its not the most convenient
| imo. At least in my experience, this always resulted in
| separate commits to dev/stage/prod as you progress through each
| part of the promotion process. It works, sure, but depending on
| your exact situation it can get cumbersome pretty quickly (for
| example, start adding in multiple prod regions, or maybe you
| want to canary one prod region first).
|
| With Kargo, it looks like it lets you define your preferred
| promotion process, and then lets you release each promotion
| with a single click. I think the part that is the most
| interesting to me though is that it writes the changes it is
| making directly to the git repo.
| rjzzleep wrote:
| I've used Flux, Flux2 and Argo for GitOps, and they all had their
| quirks. In general it seems that most CloudNative projects were
| kinda set in their own vision with very opinionated devs and a
| lot of corner cases that are not accounted for and won't be
| accounted for.
|
| I have a little bit of hope that this one is going to be
| different with the lessons they got from Argo, but I'm not
| holding my breath.
|
| Ps. when I first saw Argo, I thought this is it. The solution to
| all my problems.
| msm_ wrote:
| I miss Flux (not flux2). It did everything I needed, and was
| simple to understand. It had its problems, but I feel like they
| could be resolved without a complete rewrite/revamp the Flux2
| is. I understand the need for Flux2, but for my simple use-
| cases it's too complicated.
|
| And Argo and now Kargo are even more complex.
| greatfood230 wrote:
| Agreed. But I do feel Argo gives a good foundation; something
| on top is highly anticipated.
| candiddevmike wrote:
| Akuity is getting their lunch eaten by Harness and need some
| kind of IP to sell besides Argo-as-a-Service. I don't really
| see a path forward for them in the very crowded CI/CD space, so
| I would wait to see how they will monetize this and who the
| first movers are before attempting to use it.
|
| From a business standpoint, Akuity had their Series A in 2022
| and raised $25M. They have yet to show up on anyone's radar,
| IMO. Maybe Kargo is their PMF but I wouldn't move my CI/CD over
| yet.
| greatfood230 wrote:
| There is enough space for innovation for tools. Harness
| product is stretched thin and trying to do everything like
| throwing Pizzas to the wall to check if it sticks in order to
| boost the valuation. I doubt it will do everything well and
| make sense to all the users.
| morey-tech wrote:
| Some of that is a challenge of product design.
|
| Devs tend to be opinionated on these projects because without
| that you end up with feature sprawl to the point that projects
| become unmaintainable.
|
| On the other side, new projects need to focus on their specific
| segment and solve that problem well without worrying about
| corner cases.
|
| Either way, any solution for a corner case needs to be
| implemented in a way that solves it broadly for any user.
| wg0 wrote:
| I don't understand one thing about GitOps.
|
| Imagine 10 apps deployed. All are actively deployed let's say
| few times a day.
|
| You want to go back 10 days for App A. But in doing so you
| would have reverted whole state and all apps as they were 10
| days ago.
|
| Only way is to cherry pick particular commit and revert it.
|
| No? I mean how git can be useful in rolling back singel
| components back and forth?
| skywhopper wrote:
| You only roll forward. For live apps with evolving state this
| is the only safe method anyway, regardless of your deployment
| strategy.
| hamandcheese wrote:
| You use git revert to make a revert commit (or commits, as
| many as you need). You don't reset to 10 days ago and
| redeploy that.
| benterix wrote:
| If you use GitOps to manage apps, you better isolate them
| somehow, for example put each app in a different directory.
| In this case, a revert for App A wouldn't cause problems for
| B and C.
|
| But frankly, GitOps works best with stateless apps. Managing
| stateful apps is possible but you need to take care of state
| yourself.
| avianlyric wrote:
| Personally I like to treat the Git in GitOps as an
| implementation detail. The underlying principle of GitOps is
| that your entire environments setup and config is written in
| code, and version controlled. So you can in theory pick any
| version of your entire env, throw it at blank slate, and
| reliably get the environment specified by that git hash.
|
| Then there's the whole constant reconciliation of your
| version controlled env specification, and the actual env, and
| how you automatically resolve differences. With the most
| important principle being that the version controlled
| code/config is absolute truth and something needs to figure
| out how to bend the world to match.
|
| But importantly in all of this, Git isn't that important.
| Version control is important, infrastructure as code is
| important, but Git isn't. Arguably Git isn't a great tool for
| GitOps due to issues like the ones you mention. But the huge
| ecosystem around Git makes the pain worth it.
|
| I would argue the "correct" solution to your problem is a
| tool that automatically creates the correct cherrypicks and
| reverts for you based on a request to rollback application X.
|
| Treat git as a dumb version control system, and broadly
| ignore "good practice", because at lot of those good
| practices are designed for software development, not
| infrastructure development. We need to develop new working
| practices, built on top of Git fundamental components, rather
| than trying to rationalise existing working practices against
| the new problems that appear in GitOps.
| hdjjhhvvhga wrote:
| > So you can in theory pick any version of your entire env,
| throw it at blank slate, and reliably get the environment
| specified by that git hash.
|
| The trap here is this only works for stateless
| infrastructure. If you do it with stateful resources,
| you'll lose all data. Your gitops tool will happily
| recreate EC2 instances, S3 buckets and RDS instances, all
| empty/initialized to whatever you defined.
| anomaloustho wrote:
| There are 2 different thoughts here I think. If using
| GitOps in Kubernetes, then application and set up (Pods)
| aren't associated with Nodes (EC2). And both can be torn
| down and rebuilt without state issues. When state is
| required, then PVCs and Stateful Sets come into play.
|
| For managed services like S3 and RDS, there are other
| GitOps tools like Crossplane.io which you can use for
| similar GitOps management. But the paradigm shift might
| also be that you add GitOps config to perform regular
| backups, and also add config to ensure that if it is
| being recreated, it restores from a backup.
| mplewis wrote:
| To restore app A to a given state, you add git commits that
| specify the desired state of app A.
| yebyen wrote:
| This one is easy. I say this in spite of the spate of
| different answers that say otherwise... it should be easy?
|
| You version your apps, of course, and you would publish some
| artifact that represents the release. Historically this has
| been a Helm chart, but for Flux we are seeing many people use
| OCI repositories for this now. They give many of the benefits
| of Helm repositories without most of the drawbacks, the way
| that Flux uses them you retain traceability to the Git commit
| that started your release, and even Helm itself has already
| adopted OCI repositories in the current version, (just
| waiting for many chart publishers to catch up, we are getting
| there!)
|
| The app manages its own manifests in the app repo, the app
| devs deploy from the main branch or a dev branch on their own
| app repo, but everyone else that uses the app will deploy
| from a release artifact. Those artifacts are tagged with
| semver numbers, so you can automatically move to the next
| version of the app as soon as its published with a valid
| signature.
|
| If your app devs are the only ones using the app, then this
| should not change anything as they are building for
| production it should be versioned and managed like any
| production concern - whether it's for distribution or not,
| you still do releases.
|
| It's not any more complicated than what you are already doing
| with `docker build` and `docker push` I assure you, it's
| nearly the same. And since those OCI manifest release tags
| all logically come from a git tag, there's traceability and
| it is still GitOps in every important sense of the word.
|
| Automation as policy directives state declaratively that an
| app is always on the latest published version at any given
| time, a `spec.semver` with a wildcard accomplishes this very
| simply with a one-liner addition to your config in Flux.
|
| When you need to roll back app A, you remove the automation
| (in Flux the easiest way is a wildcard) and pin your gitops
| config for that one app to the particular version that you
| wanted in git, the cluster repo, the app is pinned to the one
| version that doesn't have an issue. Then as the issue is
| resolved, you may remove the pin and put the automation back
| in place.
|
| As an added benefit, you get a permanent history that shows
| when incident response began, how long the automation was
| disabled, and what version was pinned at that time, so you
| can calculate metrics like "MTTR" and "MBTF" that we love so
| much.
| abuckenheimer wrote:
| I've only skimmed the quickstart docs
| https://kargo.akuity.io/quickstart/ but this snippet immediately
| clicks for me as something that I'd want to try
| cat <<EOF | kubectl apply -f - apiVersion:
| kargo.akuity.io/v1alpha1 kind: Promotion
| metadata: name: prod-to-${FREIGHT_ID}
| namespace: kargo-demo spec: stage: prod
| freight: ${FREIGHT_ID} EOF
|
| Have enjoyed using argo in general in the past, its got a great
| model for k8s native workflows / events but never got to using it
| for CD.
| sleepybrett wrote:
| Yeah we use argo workflows to power some e2e testing and
| rollouts for canary blue/green, and their notification server,
| but don't actually use their cd product.
| distortionfield wrote:
| I used Argo for some e2e test suites I wrote and it was a god
| send. Being able to trigger off of GitHub labels was so
| insanely powerful for building developer tooling.
| k8svet wrote:
| Another project announcement, another situation where I would
| literally pay a coffee's worth to cut to the chase -- why is
| there a need for another GitOps tool when several already exist?
|
| Tighter integration with other Argo products? Why is this not
| simple a new component of Argo CI/CD? How much is "thought-
| leadership" a part of it, it is Kubernetes-adjacent, after all.
|
| So far even the answers in this thread leaving me asking these
| questions even more strongly? If Argo CD is a "continuous
| deployment" tool, why isn't the priority making staged-rollouts a
| first class feature in Argo CD itself?
|
| This is coming from a place of curiosity, not simply being a
| miser, I promise.
| wg0 wrote:
| There's fluxcd and Argo. What other Gitops tools one should
| keep an eye on?
| k8svet wrote:
| https://github.com/weaveworks/awesome-gitops but also, like,
| a shell script?
|
| I guess, gitops + gating/staging is compelling, but again, I
| don't see how this is a distinct product.
| therealkrancour wrote:
| Depending who you ask, CD stands for one of two things:
| Continuous _Deployment_ or Continuous _Delivery_ and they're
| not actually the same thing.
|
| Want to go camping? You go on Amazon and order a tent. An
| Amazon truck _delivers_ it to you. Not the same as deployment.
| Deployment is when you pitch the tent.
|
| Argo CD and Flux are great at Deployment. "Make this run in my
| cluster."
|
| Neither of them addresses the other need -- delivery -- getting
| artifacts from point A to point B with adequate quality gates,
| etc. along the way.
| edvinbesic wrote:
| Isn't that Argo Rollouts?
| therealkrancour wrote:
| That's also deployment. It's a drop-in replacement for the
| Kubernetes Deployment resource type. It doesn't do anything
| to help you get things from point A to point B.
| k8svet wrote:
| I don't know what this means, to be honest. Everything I
| build ends up as a cached Nix derivation output. Delivery is
| a glorified http get w/ a pre-configured public key. Beyond
| trivial, and even more so now that the containerd-nix-
| snapshotter [1] is available. (This is where my shade at the
| k8s ecosystem comes in, there are millions of dollars in VC
| funds invested in Golang projects that are obliviated by
| just... using Nix).
|
| Though, I guess maybe I just need Kargo and not the rest of
| Argo? Can someone confirm that? I'm here and bothering
| commenting because I do still see the value of "pipelines"
| and gating. Just, incredibly skeptical of new K8s-related
| products using k8sy buzzwords. And this announcement blogpost
| is _dripping_ in them.
|
| [1]: https://discourse.nixos.org/t/nix-snapshotter-native-
| underst... (aka, just deploy nix paths as native containers
| and you get the power of the nix + nix-store)
| [deleted]
| bingemaker wrote:
| Is this tool similar to https://dagger.io/?
| KenCochrane wrote:
| I think they are similar but a little different. From Dagger's
| website "Dagger is a programmable CI/CD engine that runs your
| pipelines in containers" Kargo is a tool that works with your
| existing CI and CD (Argo CD) to help promote your code from one
| stage (Dev, UAT, QA, Prod, etc) to the next.
|
| Kargo has native support for Argo CD today, but one of it's
| goals is to be agnostic and could probably work with Dagger or
| other tools in the future.
| spacelad101 wrote:
| I'm looking forward to seeing support for other GitOps
| tooling. Argo CD makes sense initially since this is by the
| same creators, but it would be nice to see agnostic tooling
| that can support other GitOps tools like Flux. I hope this is
| something that we will actually get to see and isn't just an
| empty promise.
| nickstinemates wrote:
| Cool to see you around :)
| KenCochrane wrote:
| Hey Nick, it has been to long, I hope you are doing well.
| :)
| therealkrancour wrote:
| Not much overlap. Dagger wasn't on my radar until now, but it
| reminds me of something else I used to work on and it frankly
| looks awesome!
|
| But it looks to allow you to defines pipelines using your
| preferred imperative programming language -- which is awesome.
| I would totally use that for CI.
|
| Kargo is really about divorcing CI and CD and creating
| something that's actually CD-focused and compatible with
| established GitOps principles.
| pm90 wrote:
| I swear to god I thought this was dagster and was confused for
| a bit.
| jmorenoamor wrote:
| Wow, looks interesting.
|
| You make per project makefile-like python pipelines that are
| executed by an installed runtime/docker container?
|
| Then I will give it a try, I've been looking for a lightweight
| local ad-hoc jenkins substitute.
| [deleted]
| theptip wrote:
| This looks interesting, codifying and automating promotion looks
| cool.
|
| In my experience it gets pretty hairy to build automated
| conditional promotion in a GitOps pipeline, especially if you
| don't go all the way to continuous delivery and might have
| different versions at different stages between environments.
| InfamousSoup744 wrote:
| Something like this is definitely needed in the GitOps
| space...always felt like something was missing between promoting
| things and rolling them out
| ThinkBeat wrote:
| I can almost follow this but I am at the edge of where I see how
| it could sound like buzzword gibberish.
|
| I dont blame the author(s) but things are getting more
| specialized and more terms are created. (which usually maps to an
| existing term which also maps to an existed term which also maps
| an existing term and so on.
|
| I have though about trying to create a "terminator"(hah) where
| you can paste in something from a new product and it will map the
| terms to existing terms as a sort of translator or term machine.
|
| "" We are thrilled to announce Kargo, a multi-stage application
| lifecycle orchestrator for continuously delivering and promoting
| changes through environments. Kargo, brought to you by the
| creators of the Argo Project, is a reimagining of CD pipelines
| for the cloud-native era, with first-class GitOps support,
| progressive delivery features, and is 100% open source. ""
| what-no-tests wrote:
| Weird thing is, when I read that quote it makes perfect sense.
|
| But I understand how it could look like the undecoded bytefall
| in the Matrix for those outside the know.
| robertlagrant wrote:
| Every time there's a new tool, it's worth considering if it locks
| in previous choices. E.g. ArgoCD means I can't move off
| Kubernetes. Will Kargo mean I can't move off ArgoCD or
| Kubernetes?
| AaronM wrote:
| Using Kubernetes locks you into kubernetes. ArgoCD doesn't lock
| you into Kubernetes because its just a wrapper around managing
| kubernetes manifests
|
| Its like arguing that cloudformation locks you into aws
| pm90 wrote:
| I feel like Im missing something obvious but...
| cloudformation does lock you into aws, doesn't it?
| hdjjhhvvhga wrote:
| That's the point. In other words, ArgoCD is a tool for
| Kubernetes, just like CF is a tool for AWS.
| outworlder wrote:
| It totally does. And it's an interesting argument when
| things like Terraform exist.
___________________________________________________________________
(page generated 2023-09-18 23:00 UTC)