[HN Gopher] GitHub Actions could be so much better
___________________________________________________________________
GitHub Actions could be so much better
Author : woodruffw
Score : 282 points
Date : 2023-09-22 14:21 UTC (8 hours ago)
(HTM) web link (blog.yossarian.net)
(TXT) w3m dump (blog.yossarian.net)
| rurp wrote:
| It's hard for me to articulate exactly why but I really dislike
| the information layout in GH Actions. Compared to other CI/CD
| tools, it's harder to debug problems or get a sense of the state
| of a pipeline. Part of the problem is the number of clicks it
| takes to look at various step logs. Some of the controls are
| unintuitive, but maybe that's just me.
| whalesalad wrote:
| This could be said for a lot of GitHub. The fact that you need
| to open a disclosure menu to edit a PR is wild.
| mock-possum wrote:
| Their comment system just doesn't feel right compared to
| Bitbucket's either - I think it's the distinction between
| comments that are part of a review, versus comments that are
| 'just' comments
| kaikoenig wrote:
| Been through that git commit; git push; repeat cycle too much as
| well until i discovered https://github.com/mxschmitt/action-tmate
| which gives a shell in between steps, which does not help with
| all problems but sure it's makes it less painful at times.
| justinclift wrote:
| Thanks, that looks super useful. :)
| vladaionescu wrote:
| Founder of Earthly here - besides the build debugging difficulty,
| I would add that modern CI/CD repeats a lot of steps:
| downloading, installing and configuring dependencies, making
| things much slower than they should be.
|
| We built Earthly [1] to tackle these two problems specifically.
| We're open-source (10k stars).
|
| [1]: https://earthly.dev
| solatic wrote:
| GitHub Actions is a horrible CI/CD system. You cannot run steps
| in parallel on the same VM; container-based workloads are a
| second-class citizen. The first problem means that setting up
| local credentials and other environment dependencies cannot be
| parallelized (I'm looking at you, google-github-actions/setup-
| gcloud, with your 1m+ runtime... grrr), the second makes it quite
| difficult to put a Dockerfile in a repository to represent setup
| of the CI environment, and have both (a) the CI rebuild the
| container images when the image would change, pausing workflows
| depending on that image until the image is rebuilt, (b) not
| attempting to rebuild the image when its contents did not change,
| and immediately running workflows inside that image, including
| all dependencies already installed.
|
| No, in GitHub Actions, you will attempt to repopulate from cache
| on _every single run_. Of course, sometimes the cache isn 't
| found, particularly because there's a 5 GB cache size limit
| (which cannot be enlarged, not even for payment) which cycles out
| FIFO. So if you go over the 5 GB cache, you might as well not
| have one.
|
| I _deeply_ miss Concourse. Too bad Pivotal dismantled the team
| and it barely gets even maintenance work. Parallel tasks. Custom
| pipeline triggers. SSH into the CI environment to debug. Run one-
| off tasks in CI without needing to set up a pipeline. Transfer
| directory contents from one job to another without needing to
| create artifacts (which cost money to store if you 're not
| careful about how long they should stick around for).
|
| GitHub Actions is a bastardized toy CI/CD system that only got
| popular because GitHub make it as simple as uploading a file to
| .github/workflows in any repository - no additional signup, no
| in-house ops required, everything you could want is just _there_.
| So let 's be very clear about what GitHub Actions is good and
| what it's bad at - it's good at getting people to sign up, but
| it's not nearly powerful enough to be the "best" system once you
| start to outgrow the early features.
| couchand wrote:
| > Of course, sometimes the cache isn't found, particularly
| because there's a 5 GB cache size limit (which cannot be
| enlarged, not even for payment) which cycles out FIFO. So if
| you go over the 5 GB cache, you might as well not have one.
|
| Looks like I can move on that "build caching mysteriously
| broken" issue now. Thanks for the heads up!
| nthState wrote:
| Most of my problems would vanish if there was an official way to
| run workflows locally, something like:
|
| shell: run_workflow name=MyJob in=MyWorkflow.yml params={}
| avtar wrote:
| I guess you have to ask yourself what's in MyWorkflow.yml that
| can't be in a script and run locally?
| mason55 wrote:
| Well, that's why the word "official" is in there. Obviously
| you could mock up the entire github actions scaffolding
| locally and have it inject environment variables and support
| all the actions and everything, but keeping that up to date
| will be a nightmare if you're doing it on your own.
|
| Obviously you could put the whole CI/CD into a bash script
| and not use any features or functionality provided by github
| actions but there are plenty of nice things that it does and
| it would be a shame to not use any of them.
| avtar wrote:
| I didn't mention it in my reply but I was referring to the
| complaint in the article that the author wants to be able
| to debug locally without having to push changes to GitHub
| first. The top comment by MoreQARespect and others
| highlight the benefits of scripting as much as possible so
| that processes like builds and cutting releases can be run
| and tested locally.
|
| > Obviously you could mock up the entire github actions
| scaffolding locally
|
| Not sure if anyone's advocating for implementing GA
| entirely. You can at least automate project-specific bits
| as much as possible using scripts, and then have the CI
| environment use the same automation. That allows for more
| local debugging than overusing GA.
| k8svet wrote:
| I mean, it's _embarrassing_ how bad it is.
|
| - (unrelated) build failures just randomly notifies the latest
| maintainer who happened to merge something? (Imagine you finding
| this out when your newly added maintainer pings you on Matrix and
| tells you 1: about this behavior, and 2: _that your update
| /builds have been failing for a week without you knowing?!?!_)
|
| - The cache action is horribly, trivially observably broken with
| seemingly no maintainer?
|
| - Can't tail the build log of a step if their UI poops or your
| tab unloaded after it started?
|
| - The complete lack of abstraction that _might actually make
| workflows portable across worker nodes_? pfft.
|
| - the default image is a _travesty_. I thought it was obnoxious
| how bloated it was and then I started digging in and realizing
| "Oh, some Microsoftie that didn't know Linux was put in charge of
| this". (saying this as a former FTE that knows). And there's no
| effort to allow a more slimmed down image for folks that, you
| know, use Nix? Or even just Docker?
|
| I'm in the process of migrating off GitHub and it's mostly
| because Actions shocked me to my senses. Too bad MS can't figure
| out how to retain virtually any Linux talent, and not just their
| cuffed-acqui-hires or Windows-devs-cosplaying. Even the well
| compensated ones head for the door.
|
| And I'll just say, I don't program in YAML because YAML is a
| disgrace wrought upon us by Go+Yaml enthusiasts that don't know
| any better fueled by senseless VC money shoveled at an overall
| ecosystem incognizant of modern, actually useful technology.
|
| edit: removing some of the blatantly identifying former-FTE
| trauma. Knowing what I know I should sell all my MSFT, but the
| market thinks differently.
| chrissoundz wrote:
| Struggling with this today, and numerous other days. It's so bad.
| Stop trying to build an operating system out of YAML. I'll always
| use and recommend Gitlab from now on.
|
| And what the earth is an "actions" anyway? How on earth is simple
| bash functions not just as suitable here? Instead you have some
| weird YAML scripting language. It's so bad. Why. Somebody please
| tell me. I'm losing my mind. It is a good reflection of the rest
| of the world though and why the worlds infrastructure is
| crumbling in many places.
| giobox wrote:
| Executing bash statements/scripts/functions is the thing I
| struggle _least_ with in GH actions personally, it 's
| remarkably easy to execute shell steps. If you really want your
| entire build to be a shellscript the action executes, you can
| do just that with very little YAML.
| atomicnature wrote:
| This is a _real_ problem. I think the problem stems from the
| unasserted assumption that declarative YAML is not really coding
| /debugging. The root problem is the assumption that
| infrastructure specifications are second-class citizens when it
| comes to managing a software ecosystem. Due to this, one rarely
| sees any sort of strong tooling support for creating, updating,
| debugging and extending various infrastructure activities. The
| unpleasant truth is that we live in the dark primitive days of
| infrastructure management.
| elischleifer wrote:
| Another tool that tackles GitHub Action debugging head on:
| https://www.ci-debugger.io/
|
| Wrap your GH Actions in a debugger breakpoint and connect into
| the live broken GH Action to inspect the machine, re-run commands
| etc..
|
| At it's heart - realtime debugging for GitHub actions.
| watermelon0 wrote:
| My number one complaint is that they still don't have managed ARM
| runners. Considering how overpriced runners are (compared to
| cloud instances), I'm sure they would make a ton of money here.
|
| My choices here are either to use a different CI solution (but I
| quite like Actions, and they are better integrated in the Github
| UI than alternatives), or to use custom self-hosted autoscaling
| runners (but there doesn't appear to be a good solution available
| at the moment, and I definitely don't want to managed infra for
| CI.)
| xjia wrote:
| Check out https://dime.run/ they manage workers for you
| varsketiz wrote:
| It's not really context switching. Author is debugging github
| actions, thats the context in which a few window changes seem
| minor. Context switching would be to get dragged to meeting, or
| getting asked by a colleague to explain how to set up github
| actions for their project.
| bob1029 wrote:
| My experience with GitHub Actions is mostly managed by way of
| certain Azure Portal interactions these days.
|
| When I create a new Function app, it gives me the opportunity to
| enable GitHub integration. The experience for this is flawless,
| IMO. You provide your GH credentials, select the org/repo/branch,
| and then it will create the workflow file for you and push it
| automatically. It will _also_ update the secrets in your
| repository settings to match what Azure expects on its end for
| deployment.
|
| By the time you get to look at your GitHub repo, the action is
| already running and will 100% complete successfully if you
| followed a standard/default project structure. The automatically-
| generated workflow files aren't perfect, but they're so close
| that it becomes trivial to tweak for additional build args or
| project arrangements. Just getting the secrets & related
| boilerplate configured makes the difference between me doing it
| _right now_ vs maybe never. The consequence of always having
| proper CI /CD from day zero, even for the most trivial projects,
| seems profound to me.
|
| There exist some really happy paths now wrt GH actions, but you
| gotta be willing to get pretty hammered on the Microsoft koolaid
| to explore them.
| cwales95 wrote:
| This is my normal CI/CD workflow -- even for GitLab. Debugging
| why CI/CD doesn't work is one of the worst headaches you can
| imagine.
| ctoth wrote:
| One thing that actually makes this slightly less horrible is the
| Gh tool. You can use gh run watch to tail your logs in the
| terminal and at least don't have to click through a million
| things just to see output.
| rane wrote:
| I wrote a command-line tool that streamlines retrieving test
| results from GitHub Actions even further. Essentially parses
| jest/tsc/eslint errors in GHA jobs' logs for the current's
| branch PR. https://github.com/raine/ghtool
| nafizh wrote:
| What's the technology behind the blog? It's clean, minimal and
| beautiful.
| junke wrote:
| Jekyll
| woodruffw wrote:
| It's a Jekyll site. I originally built it off of a popular
| theme (back in 2014 or so), but these days the theme is just a
| custom thing that I've cobbled together.
|
| (So, in a sense, there's no real technology "behind" it. It's
| just Markdown with a little HTML templating, with Jekyll as the
| SSG.)
| nafizh wrote:
| Thanks for the reply. Is the code available by any chance?
| fooker wrote:
| Right click -> view page source.
| woodruffw wrote:
| Unfortunately not -- it used to be in a public repo, but I
| also track drafts and other things now that I don't want to
| be public until they're ready.
|
| I'll see if I can clean it up into a public repository, but
| no guarantees :-)
| kitallis wrote:
| One of my biggest gripes with Actions is that it doesn't allow
| you to trigger workflows for a specific SHA. Only the branch
| HEAD.
|
| This is pretty much possible in all other CIs.
| joshstrange wrote:
| I couldn't agree more with the pain of debugging a GH Actions
| run. The /only/ tool you have is the ability to re-run with debug
| on. That's it. I have so many "trash" commits trying to fix or
| debug a pipeline and so much of it's just throwing stuff at the
| wall to see if it sticks.
|
| Very basic things, like having reusable logic, is needlessly
| complex or poorly documented. Once I figured out how to do it it
| was fairly easy but GitHub's docs were terrible for this. They
| made it seem like I had to publish an action to get any
| reusability or put it in a different repo. Turns out you can
| create new yaml files with reusable logic but make sure you put
| them in the root of the workflows folder or they won't work, go
| figure.
|
| It's just incredibly painful to work on GH Actions but once you
| have them working they are such a joy. I really wish there was
| some kind of local runner or way to test your actions before
| committing and pushing.
| hahn-kev wrote:
| Check this out. It doesn't do everything but it's decent
| https://github.com/nektos/act
| pid-1 wrote:
| If only competitors could do better...
|
| https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2797
| hv42 wrote:
| yeah... https://github.com/firecow/gitlab-ci-local is a good
| workaround but should be built-in. How do developers at
| GitLab/Github debug their workflows?
| TeeMassive wrote:
| I've tried running the GitHub runner image (or maybe an
| imitation) and it was really painful to setup and to get _some_
| things working. I just let it go after 2 days.
|
| And it's not just Github. The others big CI platform are not
| really better in terms of workflow and integration.
|
| Now I just script everything to the maximum.
| giobox wrote:
| > I have so many "trash" commits trying to fix or debug a
| pipeline and so much of it's just throwing stuff at the wall to
| see if it sticks.
|
| One tool is to use draft PRs for this - you can run changes to
| your action YAML from the draft PR. When you are happy just
| squash the commits as you see fit on a "real" PR to merge the
| changes in without the mess.
|
| I've found draft PRs for debugging/developing GH action logic
| to be pretty reasonable.
| lambda_garden wrote:
| Since some action depend on the branch / tag you are on this
| is not always possible.
| secret-noun wrote:
| Indeed. I have sometimes made release workflows, hardcoded
| to the main branch.
|
| You don't want to experiment too much on main because it
| dirties your commit history with 20 "Fix typo"-esque
| commits.
|
| Or, if you try to emulate the main branch with a fake main
| branch (so you can squash it later), you're still going to
| have some test commits when do the find-replace back to
| main.
|
| Neither are great.
| andix wrote:
| If I'm fixing CI I always put it on a feature branch and do a
| squash merge once I'm done. Because it's never just one quick
| fix, it's always 3-10 commits.
| kitallis wrote:
| > If I'm fixing CI I always put it on a feature branch and do
| a squash merge once I'm done. Because it's never just one
| quick fix, it's always 3-10 commits.
|
| The problem is GA also does not allow you to commit a new
| workflow in a branch. It must first exist on your primary
| branch and then you may tweak it in another.
| [deleted]
| vladaionescu wrote:
| This is the main reason we built Earthly: run your builds
| locally, and get consistency with the CI.
| [deleted]
| natbennett wrote:
| For validating YAML I love CUE. It takes some work to define the
| schema -- would def be nice if GitHub provided that.
|
| I found Nektos act pretty usable for local testing when I was
| doing a lot of GA work.
| Pxtl wrote:
| I mean GH Actions is basically a re-brand of Microsoft's "Azure
| Pipelines". As somebody who used all previous incarnations of
| TFS/VSTS/AzDO build and release pipelines: they are _not good at
| this_. This is not a team with a record of success. That Azure
| Pipelines is moderately usable only happened because they failed
| literally every other approach they tried.
|
| There was a project to allow you to run the pipelines locally so
| you could do the edit-run-debug loop on your own private
| environment without committing. It was, of course, canned.
|
| https://github.com/microsoft/azure-pipelines-agent/pull/2687...
|
| However, there are tools to improve QOL. For example:
|
| https://marketplace.visualstudio.com/items?itemName=ms-azure...
|
| A vscode extension that's syntax-aware.
|
| Now, I'll be a bit controversial: if they'd used XML instead of
| YAML, you could have an xmlns declaration up-top that would give
| you validation in most decent code editors _without_ user
| intervention. XML is awful, but it has a lot of useful features
| that we gave up when we threw the baby out with the bathwater.
| ZeWaka wrote:
| > GH Actions is basically a re-brand of Microsoft's "Azure
| Pipelines"
|
| Probably even moreso than most people think - a large portion
| of the AzDo team got moved over after the acquisition to work
| on GitHub Actions/Projects.
| woodruffw wrote:
| > I mean GH Actions is basically a re-brand of Microsoft's
| "Azure Pipelines". As somebody who used all previous
| incarnations of TFS/VSTS/AzDO build and release pipelines: they
| are not good at this. This is not a team with a record of
| success. That Azure Pipelines is moderately usable only
| happened because they failed literally every other approach
| they tried.
|
| I was under the impression (which might be wrong!) that GHA was
| an independent project within GitHub that was well underway
| before the acquisition. Are you saying that GHA was rebuilt on
| top of AzP, that it's just a relabeling of AzP, or something
| else?
|
| (I have no particular dog in it being one way or the other, but
| I'm curious about the history here.)
| Pxtl wrote:
| They share a lot of code. My understanding is that it was an
| MS project first, but I might have that backwards.
|
| > GitHub Workflows execute on runners. The runner code is
| essentially a fork of the Azure Pipelines code, so it's very
| similar. It's also cross-platform and you can also use hosted
| or self-hosted runners.
|
| https://learn.microsoft.com/en-
| us/dotnet/architecture/devops...
| woodruffw wrote:
| Thanks for the link -- I knew that GHA workflow runs ran on
| Azure, but I didn't know the workflow runner itself was a
| fork of Azure's runner/instrumentor. That's interesting
| context!
| evntdrvn wrote:
| if you look at the source of the GHA runner, you can see
| where they regex-replace all references to Azure
| Pipelines with GitHub Actions lol
| evntdrvn wrote:
| the original GHA implementation was shitcanned
| pxeger1 wrote:
| I would like to give a strong recommendation for https://pre-
| commit.ci (no relation, just a happy user).
|
| The idea is that you write hooks (or use pre-existing ones) that
| check (and fix, where possible) your code before you commit, and
| then once you've pushed, the CI will check again, including
| automatically committing fixes if necessary.
|
| Anyway, it works brilliantly - unbelievably fast thanks to good
| automatic caching, and using the exact same setup on your local
| development machine to in CI negates a lot of these debugging
| problems. It's free for open source.
|
| It only does checks, not things like automatic releases, but it
| does those them well.
| ulizzle wrote:
| More official or blessed packages for sure. I agree with that.
| he0001 wrote:
| We recently converted from Jenkins to GitHub Actions (complete
| rewrite) build pipeline. Jenkins had its issues and warts, but
| holy Beelzebub what a monster GitHub Actions is. It pure
| downright evil and creates all sort of headaches. Opaque, zero
| clues on what broke. Glacial slow even if you throw the biggest
| instance at it. It's all enterprise and 1000s of repositories big
| company.
|
| I'm spending more time finding bugs in the build pipeline than
| the time I spend doing other things. And it's all GitHub actions
| fault. I hate it with a passion.
| sitzkrieg wrote:
| one thing to be aware of, if you use windows, and submodules in a
| repo, ci will fail to clone over 50% of the time (with internal
| retries) with early eof errors
|
| it's apparently a fixed problem in upstream openssl beta that
| isnt bundled yet, but i think a custom runner would be better too
| (but not an option at my org)
|
| still, some days it gets in a rut and quickly fails for 10+ runs
| all billed at full rate. the entire eng group is severely annoyed
| by it because we cross test all oses on each commit so its
| becoming an expensive problem. basically 3x the billing on
| windows ci
| becurious wrote:
| You'd think they could at least train GitHub Copilot to test run
| the workflow and point out immediate issues.
|
| We have to add an action to kill duplicate runs on triggers.
|
| The caching is marginal, at least on Windows runners because it
| takes forever to expand the tar ball. Not even sure from their
| docs / issues on the cache action if they did finally move it to
| use GNU tar.
|
| Having some way to get an interactive shell on fail would be a
| big step up for debugging issues. Otherwise we are back to print
| debugging in the actions or uploading artifacts at each step so
| we can inspect them.
| riv991 wrote:
| The feature I'd love is history.
|
| I'd like to know if my builds are getting slower over time. I'd
| be able to detect flaky tests automatically.
|
| It seems basic, but I know third-party solutions exist for this.
| It's out of the box in Circle CI and Buildkite and feels like it
| should be here.
| atomicfiredoll wrote:
| I was with a team that moved from Jenkins to Actions. Jenkins
| has a lot of issues, but if you knew what you were doing, at
| least it was easy to see the history and at a glance know the
| project pipelines were green/healthy. It was super useful for
| seeing where flake was in tests or observing changes to
| coverage over time.
|
| After the team migrating to actions swept though, we ran
| without this basic stuff for years because nobody had the time
| to figure out what 3rd party tool to use, money to pay for it,
| or capacity to re-implement the functionality. It made dealing
| with test failures or flake that crept in awful.
| kevmo314 wrote:
| I wish individual actions could be specified as Dockerfiles
| instead of yaml files. That would address a lot of issues that
| I've run into around build environments being slightly different
| than locally.
|
| Yes, I can wrap it with a simple docker action. It would be cool
| if that were automatic.
| bingemaker wrote:
| The current state of GA is much better than Jenkins. I don't like
| yaml, but this is still better than mucking around Jenkins
| configs. Fin
| ZeWaka wrote:
| re: `pull_request_target` and fork PRs - You can get around the
| !Even More Fun! limitations this has (GITHUB_TOKEN being read-
| only) by having a workflow called by another workflow.
|
| For example, I wrote a set of workflows[1] to automatically apply
| a label after 2 'review approvals'.
|
| First, a 'dummy' workflow triggering off PRs (in that context)
| that uploads an artifact: the PR number
|
| Second, the 'real' workflow that runs in the context of the
| actual repository, set to be `on: workflow_run: workflows: -
| Final Review Labeler` - this pulls in the artifact, runs a
| GraphQL query, and applies the label if applicable.
|
| [1]:
| https://github.com/goonstation/goonstation/blob/master/.gith...
| giobox wrote:
| My personal wish is for the ability to attach HTML reports the
| the action runs without having to use the current actions/upload-
| artifact etc.
|
| Particularly for test builds, I very often just want to quickly
| view the output HTML report. The current approaches I am familiar
| with are the aforementioned upload-artifact, or using GH pages,
| but GH pages is not great when you have multiple different
| reporting output for the same repo, or you wanna quickly view
| historical reporting rather than latest.
|
| I'd love a simple "attach-report" action that just put a link to
| an HTML report on the job summary, and clicking on it renders the
| html.
|
| Other automated CI/CD style systems have much richer support out
| of the box for dealing with HTML report capturing and viewing.
| [deleted]
| dmart wrote:
| Granted, it's not HTML, but you can append Markdown output
| directly to $GITHUB_STEP_SUMMARY without dealing with artifact
| uploads etc.
| giobox wrote:
| Thanks, I'm aware of this option too, but if you say have an
| HTML report with the results of 200 tests, no one has time to
| write some logic to parse the results to a markdown summary
| frankly. Just give me the link!
|
| Similar too if the reporting captures video/screenshots of a
| given bug - the original report UI is far easier to deal
| with. Many test frameworks already natively put out an HTML
| report UI etc, it's just harder to get to than it should be
| with GH actions today.
| pests wrote:
| > an HTML report with the results of 200 tests, no one has
| time to write some logic to parse the results to a markdown
| summary frankly.
|
| Not sure how GHA treats it, but markdown is a subset of
| HTML so by default all HTML pages are valid Markdown
| documents. No conversion required.
| remram wrote:
| Can I trigger a workflow from a different workflow yet? That
| seems very basic, and last time I looked I had to create a token
| that gives read/write access to every repo in every organization
| to do this. And then it didn't work because the docs for the
| trigger API is apparently wrong.
| simse wrote:
| A linter can catch some obvious errors:
| https://github.com/rhysd/actionlint. But yes, I agree, it's not a
| fun debugging experience.
| bastardoperator wrote:
| act would also be helpful here in terms of debugging and
| development of Actions workflows.
|
| https://github.com/nektos/act
| woodruffw wrote:
| Yep, actionlint is great! I've used it successfully both to
| lint my own workflows, and to lint third-party workflows for
| (basic) security issues.
|
| Unfortunately, it can't lint actions themselves, only workflows
| that _call_ actions[1]. This is a substantial deficiency,
| especially for users (like me) who write and maintain a decent
| number of actions.
|
| [1]: https://github.com/rhysd/actionlint/issues/46
| plonk wrote:
| The best approach is to develop completely local build scripts
| and add a few options to plug GitHub Actions' exclusive features
| into them. Make a build script that can take a cache folder's
| path as argument. Make it take a test report output path as
| another argument. etc.
|
| So your GHA workflow is: set up secrets (AWS and co.), set up
| Python, download cache to path X if possible, run the build in X
| and send reports to Y, publish reports in Y. And put as much work
| as possible inside the build script. We have a whole Python
| project for that.
|
| This is truly painless after the first development effort.
| andix wrote:
| I'm really not sure if we are using CI correctly. Sometime i
| think all those CI Templates should be replaced by just one
| executable that does everything, like a modern alternative to
| Makefiles (and there are a lot of build tools).
|
| So the CI pipeline would only call the build tool, like "./build
| containers push-to-registry release:1.0.0 run-tests"
|
| Those scripts can be tested and debugged everywhere. Also
| migrating to a different CI platform would be really easy.
| gravlaks wrote:
| https://dagger.io/
|
| may be an alternative. You can run it in GitHub actions somehow
| as well.
| pixl97 wrote:
| How does that integrate in with every else's tooling? At least
| in enterprise there are a ton of things like "SCA, SBOM,
| Compliance report, etc" that tie in with plugins and such.
|
| Also, why would a large commercial CI want to have their
| environment too open?
| andix wrote:
| They don't want to have their environments too open, that's
| why we are at this point.
|
| Those enterprise tools i know (like sonarqube for example)
| are mostly cli tools in their core. Can be integrated
| everywhere.
| WirelessGigabit wrote:
| While this is totally possible you lose a lot of the things
| that make GitHub Actions nice. Nice logs, annotations,
| individual steps that went wrong. Being able to see the status
| at a glance. Seeing 10 steps where 1 fails and 13 pass is nice.
| andix wrote:
| All those things the custom build tool would need to handle.
| It needs more or less the same functionality as GitHub
| actions and the marketplace. It would also need to integrate
| with different CI platforms to some extent for reporting and
| user interactions.
| kathlam wrote:
| It's clear we've all wrestled with GitHub actions and Jenkins in
| some shape or form. Debugging them can feel like being in an
| escape room, without any clues.
|
| I work at Trunk.io and we have a tool called CI Debugger. It
| helps you understand your workflows. Instead of getting lost in
| logfiles and commits, you can pinpoint where things are bogging
| down or going haywire. It can give you a clearer picture of
| what's happening under the hood. Like having a heat map for the
| CI/CD processes.
| tao_at_garden wrote:
| The git commit, push, wait loop is terrible UX. Users deserve
| portable pipelines that run anywhere, including their local
| machines. I understand Act [1] goes some way to solving this
| headache but it's by and large not a true representation.
|
| There are many pipelines you can't run locally, because they're
| production, for example, but there's no reason why we can't
| capture these workflows to run them locally at less-critical
| stages of development. Garden offers portable pipelines and then
| adds caching across your entire web of dependencies. Some of our
| customers see 80% or higher reductions in run times plus devs get
| that immediate feedback on what tests are failing or passing
| without pushing to git first using our Garden Workflows.
|
| We're OSS. [2]
|
| [1] https://github.com/nektos/act
|
| [2] https://docs.garden.io
| baggachipz wrote:
| Act's incompleteness has had me barking up the wrong tree many
| times. At this point I've temporarily abandoned using it in
| favor of the old cycle. I'm hoping it gets better in time!
| frodowtf wrote:
| I don't get why GitHub doesn't adopt it and make it a
| standard. Especially the lack of caches is annoying.
| sakopov wrote:
| We need Terraform for build pipelines and God help you if you
| use Bitbucket lol
| andrewstuart2 wrote:
| Yeah, it seems like we lost a lot of the "CI shouldn't be a
| snowflake" when we started creating teams that specialize in
| "DevOps" and "DevOps tools." Once something becomes a career, I
| think you've hit the turning point of "this thing is going to
| become too complicated." I see the same thing with capital-A
| Agile and all the career scrum masters needing _something_ to
| do with their time.
| candiddevmike wrote:
| If folks just had actions target make or bash scripts instead
| of turning actions into bash scripts none of this would be an
| issue. Your CI/CD and your devs should all use the same
| targets/commands like `make release`.
| taeric wrote:
| I'm actually confused and scared on how often this isn't the
| case? What are people doing in their actions that isn't
| easily doable locally?
| ghayes wrote:
| A huge portion of my actions are for things like caching or
| publishing artifacts, which are unique to actions itself.
| taeric wrote:
| I'd assume you would be able to publish and deploy
| locally before setting up actions. Such that those are
| likely targets in your build system?
|
| Caching, I can mostly understand as unique there. Though,
| I think I'm living with whatever the default stuff in
| actions is. Slow for builds that don't happen often, of
| course, but not so slow that I care.
| duped wrote:
| Why should caching in the cloud be any different than
| caching locally?
| baq wrote:
| There isn't any _locally_ in GHA after the runner exits
| baq wrote:
| It's the cloud. Runners are ephemeral (pretend, but still)
| with no persistent storage. This makes you either rebuild
| everything in every release stage (bad) or put artifacts in
| s3 or whatever (also bad) - this is especially painful for
| intermediate artifacts like dependency bundle caches etc.
|
| As much as I like make it just doesn't work with the
| typical cloud stateless by default configs. If it works for
| you, your project is small enough and try to keep it this
| way.
| ZeWaka wrote:
| > with no persistent storage
|
| There's https://github.com/actions/cache though?
| dcow wrote:
| They're saying that unless you use actions, you don't get
| the cohesive cache and artifacts support. That
| replicating that in the cloud or locally is a PITA. Thus
| people are using the GH actions vendor specific tooling
| in that way.
| taeric wrote:
| Rebuilding at every stage shouldn't be too bad, with
| pinned dependencies. I can see problems with it, of
| course. That said, using a private code publishing
| location seems the correct path? That isn't too difficult
| to setup, is it?
|
| That said, I'm still not clear on what difficulties folks
| are worried about. I'm also not clear I care on the mess
| of commits getting things working. The initial commits of
| getting anything working are almost always a mess. Such
| that worrying about that seems excessive.
| plonk wrote:
| Just run the GitHub cache action on your build directory
| and then run make inside it?
| ramraj07 wrote:
| All the linting checks and end to end tests I don't want to
| bother setting up locally for every repo I touch.
| taeric wrote:
| Aren't these just other targets in whatever build system
| you are using, though?
| jusonchan81 wrote:
| We moved our CI to this orchestrator called Netflix Conductor.
| It's highly customized internally with workers capable of running
| different kind of tasks etc. But since we did this, CI has been a
| wonderful experience. Things run smoothly and history can be
| tracked and easy to build variants by modifying or creating new
| workflows. We aim to open source this extension soon.
| eviks wrote:
| Very good points, especially re. the security footguns
| samiam_iam wrote:
| [dead]
| Nezteb wrote:
| This is why I want projects like Earthly to succeed:
| https://github.com/earthly/earthly
|
| I want to be able to run my all of CI workflows on my local
| machine.
|
| I agree with other commenters that most of what CI does should be
| abstracted out into scripts or other non-CI tools. Unfortunately
| it's not easy to do that for large pre-existing CI setups,
| especially if a different team is the one maintaining your CI
| workflows.
| numbsafari wrote:
| > Because GitHub fails to distinguish between fork and non-fork
| SHA references, forks can bypass security settings on GitHub
| Actions that would otherwise restrict actions to only "trusted"
| sources (such as GitHub themselves or the repository's own
| organization).
|
| How is this not resolved?
|
| Easily bypassing security controls is a major security issue.
|
| Yes, you need to convince someone to use your SHA, but social
| engineering is usually the easy part.
| lijok wrote:
| People really need to stop writing scripts in yaml. Not to be
| dismissive but the issues outlined in this article, aside from
| that security footgun, are non-issues as soon as you start using
| github actions only for what it should be used - to invoke your
| scripts in parallel in response to an event. We've banned all but
| actions/checkout in our org and have a very healthy dev
| experience as a result. People are naturally guided towards
| writing scripts that they can run locally instead of for github
| actions specifically.
|
| Some real issues with github actions:
|
| 1. MacOS runners have ridiculously inconsistent IO performance
| causing 200x slowdown in some cases
|
| 2. Getting charged to the nearest minute is asinine and punishes
| highly parallelised (fast) workloads
|
| 3. GitHub OIDC endpoints constantly timing out
| theideaofcoffee wrote:
| As someone who has recently sunk a considerable amount of time in
| modernizing some pipelines, workflows, what have you, I feel this
| deep in my bones. The commit-test-fail-recommit test loop is
| painful and something I've experienced all too well. Trying to
| work around product limitations brought about by whoever
| concocted the system on GitHub's side to save customers from
| themselves and declared it good is painful. Trying to debug
| simple issues is painful. I'm glad I'm not alone.
| MoreQARespect wrote:
| There are two types of github actions workflows you can build.
|
| 1) Program with github actions. Google "how can I send an email
| with github actions?" and then plug in some marketplace tool to
| do it. Your workflows grow to 500-1000 lines and start having all
| sorts of nonsense like conditionals and the YAML becomes
| disgusting and hard to understand. Github actions becomes a
| nightmare and you've invited vendor lock in.
|
| 2) _Configure_ with github actions. Always ask yourself "can I
| push this YAML complexity into a script?" and do it if you can.
| Send an email? Yes, that can go in a script. Your workflow ends
| up being about 50-60 lines as a result and very rarely needs to
| be changed once you've set up. Github actions is suddenly fine
| and you rarely have to do that stupid push-debug-commit loop
| because you can debug the script locally.
|
| Every time I join a new team I tell them that 1 is the way to
| madness and 2 is the sensible approach and they always tepidly
| agree with me and yet about half of the time they still do 1.
|
| The thing is, the lack of debugging tools provided by Microsoft
| is also really not much of a problem if you do 2, vendor lock in
| is lower if you do 2, debugging is easier if you do 2 but _still_
| nobody does 2.
| jjice wrote:
| Absolutely agreed. Well said and I'll be stealing this
| explanation going forward. Hell, just local running with
| simplicity and ability to test is a massive win of #2, aside
| from just not dealing with complex YAML.
| EddTheSDET wrote:
| How DO you debug your actions? I spend so long in the commit-
| action-debug-change loop it's absurd. I agree with your point
| re: 2 wholeheartedly though, it makes debugging scripts so much
| easier too. CI should be runnable locally and GitHub actions,
| while supported with _some_ tooling, still isn't very easy to
| work with like that.
| MoreQARespect wrote:
| Using the same commit-push-debug loop you do. It just isnt
| painful if I do 2.
| edgyquant wrote:
| I too wish I could find a nicer way than this to debug.
| somehnguy wrote:
| Act works pretty well to debug actions locally. It isn't
| perfect, but I find it handles about 90% of the write-test-
| repeat loop and therefore saves my teammates from dozens of
| tiny test PRs.
| dmtryshmtv wrote:
| Good perspective. Unfortunately (1) is unavoidable when you're
| trying to automate GH itself (role assignments, tagging, etc.).
| But at this point, I would rather handle a lot of that manually
| than deal with GHA's awful debug loop.
|
| FWIW, there's nektos/act[^1], which aims to duplicate GHA
| behavior locally, but I haven't tried it yet.
|
| [^1]: https://github.com/nektos/act
| dlisboa wrote:
| > Unfortunately (1) is unavoidable when you're trying to
| automate GH itself (role assignments, tagging, etc.)
|
| Can't you just use the Github API for that? The script would
| be triggered by the YAML, but all logic is inside the script.
|
| But `act` is cool, I've used it for local debugging. Thing is
| its output is impossibly verbose, and they don't aim to
| support everything an action does (which is fine if you stick
| to (2)).
| plorkyeran wrote:
| Yeah, I've done quite a bit of Github scripting via octokit
| and it's pretty simple. Using GHA's built-in functionality
| might turn a five line script into a one-liner, but I think
| being able to run the script directly is well worth the
| tradeoff.
|
| The main thing that you can't decouple from GHA is pushing
| and pulling intermediate artifacts, which for some build
| pipelines is going to be a pretty big chunk of the logic.
| lambda_garden wrote:
| Option 1 is required if you want to have steps on different
| runners, add approval processes, etc.
|
| I always opt for option 2 where possible though.
| xeromal wrote:
| Github actions calling make commands is my bread and butter.
| woodruffw wrote:
| This is a great perspective, and one I agree with -- _many_ of
| the woes associated with GitHub Actions can be eliminated by
| treating it just as a task substrate, and not trying to program
| in YAML.
|
| At the same time, I've found that it often isn't sufficient to
| push everything into a proper programming language: I _do_
| sometimes (even frequently) need to use vendor-specific
| functionality in GHA, mark dependencies between jobs, invoke
| REST APIs that are already well abstracted as actions, etc. Re-
| implementing those things in a programming language of my
| choice is _possible_ , but doesn't break the vendor dependency
| and is (IME) still brittle.
|
| Essentially: the vendor lock-in value proposition for GHA is
| very, very strong. Convincing people that they should take
| option (2) means making a stronger value proposition, which is
| pretty hard!
| MoreQARespect wrote:
| No, you're right it's not necessarily a good idea to be anal
| about this rule. E.g. If an action is simple to use and
| already built I use it - I won't necessarily try to
| reimplement e.g. upload artifacts step in code.
|
| Another thing I noticed is that if you do 1 sophisticated
| features like build caching and parallelization often becomes
| _completely_ impractical whereas if you default to 2 you can
| probably do it with only a moderate amount of commit-push-
| debug.
| progmetaldev wrote:
| Your advice is sane and I can tell speaks from experience.
| Unfortunately, now that Github Actions are being exposed
| through Visual Studio, I fear that we are going to see an
| explosion of number 1, just because the process is going to be
| more disconnected from Github itself (no documentation or
| Github UI visible while working within Visual Studio).
| riquito wrote:
| I agree overall, but you oversimplify the issue a bit.
|
| > can I push this YAML complexity into a script?
|
| - what language is the script written in?
|
| - will developers use the same language for all those scripts?
|
| - does it need dependencies?
|
| - where are we going to host scripts used by multiple github
| actions?
|
| - if we ended up putting those scripts in repositories, how do
| we update the actions once we release new version of the
| scripts?
|
| - how do you track those versions?
|
| - how much does it cost to write a separate script and maintain
| it versus locking us in with an external github action?
|
| These are just the first questions that pop in my mind, but
| there is more. And some answers may not be that difficult, yet
| is still something to think about.
|
| And I agree with the core idea (move logic outside pipeline
| configuration), but I can understand the tepid reaction you may
| get. Is not free and you compromise on some things
| peteradio wrote:
| If build scripts or configuration is shared it might be one
| of the only times a git submodule is actually useful.
| c-hendricks wrote:
| This is a whole lot of overthinking for something like
| #!/usr/bin/env bash set -ex aws send-
| email ...
| umvi wrote:
| Default to bash. If the task is too complex for bash, then
| use python or node. Most of these scripts aren't going to
| change very often once stable.
| wry_discontent wrote:
| Default to babashka.
| plorkyeran wrote:
| I think you are still envisioning a fundamentally incorrect
| approach. Build scripts for a project are part of that
| project, not some external thing. The scripts are stored in
| the repository, and pulled from the branch being built.
| Dependencies for your build scripts aren't any different from
| any other build-time dependencies for your project.
| tomtheelder wrote:
| I think they framed it accurately and you are instead over
| complicating. Language for scripts is a decision that
| virtually every team ends up making regardless. The other
| questions are basically all irrelevant since the scripts and
| actions are both stored in repo, and therefore released
| together and versioned together.
|
| I think the point about maintenance cost is valid, but the
| thesis of the comment that you are responding to is that the
| prebuilt actions are a complexity trap.
| chubot wrote:
| Exactly, I showed here how we just write plain shell scripts.
| It gives you "PHP-like productivity", iterating 50 times a
| minute. Not one iteration every 5 minutes or 50 minutes.
|
| https://lobste.rs/s/veoan6/github_actions_could_be_so_much_b...
|
| Also, seamlessly interleaving shell and declarative JSON-like
| data -- without YAML -- is a main point of
| http://www.oilshell.org, and Hay
|
| Hay Ain't YAML -
| https://www.oilshell.org/release/0.18.0/doc/hay.html
| [deleted]
| [deleted]
| hk1337 wrote:
| 1 is to build utilities for 2, IMO. It shouldn't have
| repository specific information inside and should be easily
| useable in other workflows.
| pjc50 wrote:
| I've reached the same conclusion with Jenkins. It also helps if
| you ever have to port between CI systems.
|
| A CI "special" language is almost by definition something that
| can't be run locally, which is really inconvenient for
| debugging.
| jahnu wrote:
| The main reason I aim for (2) is that I want to be able to
| drive my build locally if and when GitHub is down, and I want
| to be able to migrate away easily if I ever need to.
|
| I think of it like this:
|
| I write scripts (as portable as possible) to be able to
| build/test/sign/deploy/etc They should work locally always.
|
| GitHub is for automating me setting up the environments where I
| can run those scripts and then actually running them.
| MenhirMike wrote:
| Option 2 also makes it easier for developers to run their
| builds locally, so you're essentially using the same build
| chain for local debugging than you do for your
| Test/Staging/Prod environments, instead of maintaining two
| different build processes.
|
| It's not just true for GHA, but for any build server really:
| The build server should be a script runner that adds history,
| artifact management, and permissions/auditing, but should
| delegate the actual build process to the repository it's
| building.
| hk1337 wrote:
| Locally or if for some reason you need to move off of Github
| and have to use Jenkins or some other CI tool.
| intelVISA wrote:
| Turns out the real SaaS is Scripts as a Service.
| rayhu007 wrote:
| Totally get what you're saying. I once switched our workflow to
| trigger on PRs to make testing easier. Now, I'm all about using
| scripts -- they're just simpler to test and fix.
|
| I recommend making these scripts cross-platform for
| flexibility. Use matrix: and env: to handle it. Go for Perl,
| JavaScript, or Python over OS shells and put file tasks in
| scripts to dodge path issues.
|
| I've tried boxing these scripts into steps, but unless they're
| super generic for everyone, it doesn't seem worth it.
| wbond wrote:
| I have a few open source projects that have lasted for 10+
| years, and I can't agree more with approach #2.
|
| Ideally you want your scripting to handle of the weird gotchas
| of different versions of host OSes, etc. Granted my work is
| cross-platform so it is compounded.
|
| So far I've found relying on extensive custom tooling has
| allowed me to handle transitions from local, to Travis, to
| AppVeyor, to CircleCI and now also GitHub Actions.
|
| You really want your CI config to specify the host platform and
| possibly set some env vars. Then it should invoke a single CI
| wrapper script. Ideally this can also be run locally.
| jurassic wrote:
| Most of my frustrations with GHA arise when doing something
| useful conflicts with someone's idea of security. For example,
| branch protection rules intended to stop devs from yoloing
| commits blocking me from pushing a version bump commit during a
| release workflow.
| syndicatedjelly wrote:
| gitlab-runner allows you to run scripts locally for GitLab, but
| even that is lacking (any env secrets stored in GitLab? Script is
| going to fail).
|
| I really, really wish there was a way to clone the GitHub Actions
| or GitLab runner environment, spin up a runner locally and test.
| That would shave off 95% of the wait time
| artursapek wrote:
| GitLab's CI is so much better than GH it's not even funny
| gchamonlive wrote:
| I am thinking about porting my ci/cd pipelines from Jenkins to
| GitHub actions to leverage autoscaling self-hosted runners, which
| I don't think there is an exact equivalent for Jenkins nodes, at
| least not one that is already implemented where I work.
|
| The pipelines though, are all configured using ansible. Applying
| the same principle I did to Jenkins, using it strictly for
| execution automation, logs, permission and the likes, and leaving
| the heavy lifting to a more trusty tool, I think I can avoid both
| major vendor lock-in and idiosyncrasies and nonsenses from the
| language.
|
| It boggles my mind how unrefined the GitHub actions flow is. It
| doesn't offer a good experience by a long shot.
| bysja wrote:
| Github Actions has been a pleasant introduction to CI/CD for me.
| I used to build locally and then git push and/or rsync everything
| directly to a VPS (Digitalocean Droplets in my case). But for
| collaboration, this workflow breaks down. Then, using Github as
| the "origin" repository becomes compelling, and Github Actions
| fit very nicely with that workflow.
| darepublic wrote:
| I don't see how you love something that makes you jump through
| these hoops:
|
| > In this particular case, it took me 4 separate commits (and 4
| failed releases) to debug the various small errors I made: not
| using ${{ ... }}5 where I needed to, forgetting a needs:
| relationship, &c
| vegardx wrote:
| There's tools like Act[0] that tries to solve this, but this
| has been an issue with CI systems since they were invented.
|
| [0] https://github.com/nektos/act
| 5e92cb50239222b wrote:
| Also reused by gitea for their CI runner. I was quite
| impressed by that feat, pretty neat.
|
| https://gitea.com/gitea/act_runner
| hanniabu wrote:
| Isn't that just programming? That's not much different than
| saying you forgot a bracket and had to make another commit to
| make it work. Granted, it would be nice if they had some
| linter.
| matthewaveryusa wrote:
| This is a usecase where chatgpt works great! I usually pass
| this kind of ops DSL in gpt and it will find my mistakes.
| Github actions and graphanaQL code, oof, they just click for
| me.
| chc wrote:
| I'm not gonna post my employer's proprietary code into
| ChatGPT.
| matthewaveryusa wrote:
| Don't fixate on the details, fixate on the larger point.
| your specific situation of having code self-hosted and an
| employer that cares deeply about their proprietary code
| is a detail beside the point. LLMs shine at fixing up
| DSLs that you're unfamiliar with. chatgpt to llm is
| kleenex to facial tissue.
| 5e92cb50239222b wrote:
| Generally programming these days does not require you to
| submit build jobs or jump through other hoops to catch
| trivial mistakes. Although it depends on what you do,
| embedded developers and game programmers writing for consoles
| might disagree.
|
| It really feels like a throwback to the punch card era.
|
| I too don't enjoy writing CI scripts (despite knowing Linux
| administration and shell scripting quite well), it always
| takes an inordinate amount of time, but it also saves much
| more over the long term.
| garciasn wrote:
| We use Github Actions and we just don't have any issues with it
| outside the first time we set it up for each repo. Then we make
| 100s of commits a week and it does its thing and our work goes
| live a few seconds later. That's why I love it.
|
| Could things be better? Of course; that's how software is--and
| this should resonate with most folks on this site. But just
| because some product isn't infallible doesn't mean we can't
| love it too.
| neuromanser wrote:
| > we just don't have any issues with it outside the first
| time we set it up for each repo
|
| But that's exactly what a lot of commenters complain about.
| moralestapia wrote:
| >Then we make 100s of commits a week and it does its thing
| and our work goes live a few seconds later.
|
| Wow, computers doing what they're supposed to do. Pretty
| impressive ...
|
| The UX on GH actions is crap, other CI/CD solutions give you
| way more control and tooling. GH really needs to step up
| their game on this one.
| garciasn wrote:
| I am not sure why you're being so snarky, but I did address
| that it could be better.
|
| The thing for us is that we're already paying for GitHub
| and Actions are included. I know, for a fact, GH is/was
| working with stakeholders using their software to get
| feedback on what could be better. So they are trying!
|
| I'm not here being a fanboy by any means; I'm just saying
| it does what I need it to do and I'm an active user who is
| pleased with the service provided.
|
| YMMV.
| layer8 wrote:
| Yeah, I wouldn't want to use any automation that you can't also
| easily and quickly test locally.
| Pxtl wrote:
| Fun fact: Microsoft had a plan to provide that!
|
| They canned it.
|
| https://github.com/microsoft/azure-pipelines-
| agent/pull/2687...
| bdcravens wrote:
| Act can do most of it locally
|
| https://github.com/nektos/act
| ris wrote:
| The single thing that I most detest about GitHub Actions is how
| they, by design, completely miss the point of containers. Having
| "actions" that are just "install language foo" is barely better
| than just publishing shell scripts or even the days of Travis.
|
| As a result, few GitHub workflows benefit from the immutability
| or reproducibility guarantees that can be provided by containers,
| and most workflows I interact with spend more than half their
| time running ridiculous installer scripts.
| brabel wrote:
| Are you unaware that GH Actions can run jobs in containers??
|
| https://docs.github.com/en/actions/using-jobs/running-jobs-i...
___________________________________________________________________
(page generated 2023-09-22 23:00 UTC)