[HN Gopher] GitHub experiencing issues with actions, pull reques...
___________________________________________________________________
GitHub experiencing issues with actions, pull requests, packages
Author : Amorymeltzer
Score : 177 points
Date : 2021-02-08 16:10 UTC (6 hours ago)
(HTM) web link (www.githubstatus.com)
(TXT) w3m dump (www.githubstatus.com)
| alfiedotwtf wrote:
| Curious: are Microsoft slowly replacing the innards of GitHub
| with a Microsoft stack?
| qbasic_forever wrote:
| The main web backend is still ruby it seems:
| https://github.blog/2020-08-25-upgrading-github-to-ruby-2-7/ I
| can't remember where it was exactly but I remember recently
| stumbling on a random github-related OSS project that had
| chatter in the issues from githubbers talking about .NET and
| C#. It would not surprise me to see significant pressure from
| the now 2-year post-acquisition engineering org to get in line
| with the rest of MS's online services that are all .NET stack.
| FunnyLookinHat wrote:
| This same pattern happened last week - Webhooks reporting issues,
| then Pull Requests, etc. This one seems to be affecting the rest
| of their systems a bit more (or maybe they're just being a bit
| more detailed in exactly what is down this time).
|
| https://www.githubstatus.com/incidents/5zb8gfbl7qkt
| pestkranker wrote:
| ~10 downtime per month. What is happening at GitHub?
| hulitu wrote:
| > ~10 downtime per month. What is happening at GitHub?
|
| They belong to Microsoft. Reliability was never a feature of
| Microsoft products.
| Droobfest wrote:
| Maybe they're porting over the Azure DevOps code which also has
| 'degraded performance' issues about every other day.
| tylfin wrote:
| The last time this happened, it was after they shipped the
| phone app.
|
| I wonder if they have a big feature underway or are just
| migrating more infrastructure to Azure?
|
| EDIT: Either way, some postmortems would be appreciated before
| more customers have to look for a backup solution...
| capableweb wrote:
| If you're working on a serious project, hosting it mainly on
| GitHub via Git and don't already have a backup solution in
| place, I'm afraid you're late. But better late than never!
| Make sure you can always deploy when less reliable services
| are down, and GitHub has always been one of those. Git makes
| it incredibly easy as well, as long as you have your CI/CD
| externalized already.
| tylfin wrote:
| Yeah, this is very good advice.
|
| I think if revenue or product quality is tied to a VCS,
| having an active-active or active-passive setup is the way
| to go.
|
| Fortunately, I'm on an on-prem product so that investment
| hasn't seemed worth it yet.
|
| This doesn't mean we don't escrow our code, but rather than
| try to rebuild from source, I just take a short coffee
| break and wait for the impacted service to come back up :)
| dessant wrote:
| I'm still awaiting the promised post-mortem [1] of a
| retracted blog post [2][3] which has annnounced the
| deprecation of the GitHub Developer Program.
|
| > There must be quite a story behind this - will you be
| putting up a post-mortem ? (Post mortems of business
| "outages" are usually more instructional)
|
| > Yes. We will. Please stay tuned.
|
| [1] https://news.ycombinator.com/item?id=21718171
|
| [2] https://news.ycombinator.com/item?id=21718083
|
| [3] https://web.archive.org/web/20191205225751/https://develo
| per...
| Florin_Andrei wrote:
| > _What is happening at GitHub?_
|
| They were acquired by Microsoft a while ago, and now the
| chicken are coming home to roost.
|
| It's pretty much on par with my Azure experience.
| robbyt wrote:
| Me too. The only people who think Azure is viable, haven't
| actually used it.
| seniorThrowaway wrote:
| I recently left a project using AWS for one using Azure. I
| thought the AWS API's were inconsistent and janky but they
| look great compared to Azure. Azure is also extremely slow
| to perform actions in my experience and the documentation
| is very heavily tilted towards being sales funnels. I do
| like the keyvault service and the idea of resource groups.
| The whole tenant / subscription / roles / user mess of
| permissions not so much, but I expected that from
| Microsoft.
| bredren wrote:
| This is a big problem with GitHub Actions.
|
| It took a community drumbeat and persistence from an enterprise
| customer to get the status message to even show a problem last
| month. [1]
|
| It does suck and I do think there must be some political
| infighting going on that the service is having so many
| disruptions.
|
| There's no excuse for something this important to not only have
| so much unplanned downtime, but no resources to connect with
| the community by offering post mortems or other reasonable
| interactions.
|
| That said, I'm still all in on GA. It's amazing and the
| coupling with repos is great. It continues to be subtly
| refined. So I just hope whoever is holding this product back
| gets out of the way.
|
| [1] https://github.community/t/random-unknown-blob-error-when-
| pu...
| TruthWillHurt wrote:
| Microsoft.
| rvz wrote:
| All I know is that it doesn't seem like a wise choice to be
| locked into GitHub features or even use their tools with these
| frequent downtime episodes.
|
| If a large open-source organisation was to rely on say, GitHub
| Actions for example, well you'll probably see more and more of
| "GitHub down" posts and they'll be unable to push that critical
| patch or run that cloud CI on GitHub, and some maybe
| considering solutions like this [0].
|
| Every time this happens, you'll be completely locked in and
| ending up contacting / complaining to the CEO of GitHub for
| support via Twitter.
|
| No thanks and no deal I'm afraid.
|
| [0] https://news.ycombinator.com/item?id=23849565
| bob1029 wrote:
| Good deal for us, actually. I used to get upset about this,
| but the alternative really does suck more.
|
| You can either deal with the occasional non-productivity from
| a SAAS offering (which for GH has never lasted more than
| ~half a work day), or you can spin up all your own stack on-
| prem and generate 10 additional full-time problems in the
| quest to solve this one periodic issue.
|
| The trick is to never put yourself in a position where your
| tools absolutely must work immediately or you lose a
| customer. Why make a promise on delivering a piece of
| software until its already in hand? Also, if github goes down
| and I really wanted to get an issue comment in, I can just
| open a text editor and keep a note around in my local repo
| until everything is back up. I can even do some crazy things,
| like share my local branches with other developers over side
| channels until things get back to normal in the centralized
| system.
|
| Wasn't this kinda the entire point of talking developers into
| moving to the git model? Would be fun to rewind the clock and
| use these takes as an argument for sticking with TFS, et. al.
| yarcob wrote:
| > I can even do some crazy things, like share my local
| branches with other developers over side channels
|
| Haha, right, totally crazy, you could use git just like it
| was designed to be used! Who would do that!!
|
| (Joking aside, despite Githubs frequent outages, I don't
| recall the git service itself ever being affected)
| bob1029 wrote:
| > I don't recall the git service itself ever being
| affected
|
| The most important part of their service has never failed
| to work for me over the last 6 years.
| swagonomixxx wrote:
| I agree with you. To me it just sounds like people being
| entitled, expecting a service to never go down, ever. Shit
| happens and things go down. Design your processes around
| that fact and have procedures lined up for when it does
| happen.
|
| However, saying "just get GitLab and deploy it to your own
| server" glosses over the huge time sink it is, especially
| for small companies that are already short-staffed, to
| maintain something like that. I sure as heck do not want to
| be responsible for keeping my GitLab server up.
|
| As you say, if you're writing an issue, put it in an editor
| issues.md file or something. If you're working on code,
| even better, just commit locally.
| leesalminen wrote:
| Hosting Gitlab hasn't been a huge time sink for me and
| I'm a one-person show who is conscious about my time. I
| set up automatic updates and haven't SSH'd into that
| machine in about a year.
| bob1029 wrote:
| I agree that in many scenarios you will find that your
| approach is perfectly valid.
|
| For us, we have ~8 people that need to use the system all
| at the same time. We utilize issues very heavily (we are
| entering 5 figures), with lots of data-heavy QA content
| throughout (screenshots/videos/binaries/etc).
| Additionally, our customer environments are actually
| configured to talk directly to our GitHub repository for
| purposes of rebuilding themselves from source at update
| time.
|
| Because of the number of participants who are involved
| with our particular usage of GitHub, we find that a
| hosted solution with horizontal scalability and
| resilience to be an excellent fit. We have made the
| decision to make it Microsoft's problem to figure out how
| to eventually deal with 10k+ issues and 200+
| employees/clients trying to hit the same host all at the
| same time.
|
| If we had decided to host our own GitHub/Lab server in
| our cloud environment, we would be having to constantly
| review the capacity of the IT systems. As we add
| employees and customers, the load we put on our source
| control solution will increase linearly. Additionally,
| because of the deploy-time approach, having a solution
| that is backed by someone else's network means that we
| don't have to worry about our private network being
| slammed by outside requests. Our total checkout is
| nearing a gigabyte, so you can see how this might scale
| poorly if we operated out of our own infrastructure.
|
| I almost feel like we are abusive of Microsoft's
| generosity considering the sheer amount of content we
| have throughout our organization's account. Every day I
| wonder when I am going to get some email demanding that
| we switch to a more expensive enterprise plan because of
| how we use the service. Maybe that day will never come.
| Even if it does, I will gladly shell out for the bigger
| contract.
| GordonS wrote:
| I noticed that code search seems to be broken too - hasn't been
| working for 1-2 hours.
|
| Search still works for issues, repos etc, but not code.
| inetknght wrote:
| Github's code search has never been useful though
| mistersys wrote:
| GitHub search is not very smart, but I prefer that when
| searching code. When searching code, I'm usually trying to
| find exact tokens i.e. a variable name or an error message
| string vs. searching documents.
|
| I find it useful all the time.
| zoomablemind wrote:
| 'git grep' allows the search within the repo, assuming that
| you have it cloned locally.
| upbeat_general wrote:
| I also find it useful yet it still often misses exact
| matches.
|
| If I search for "int x = 5" and it doesn't return "int x =
| 5;", there's an issue here.
| inetknght wrote:
| I've found that `grep` with PCRE enabled is far more useful
| to find exact tokens like variable names or error message
| strings.
|
| I've never had Github's search find what I'm looking for.
| remram wrote:
| That requires you to clone. It's a minor hassle if you
| search over a single repo, but when searching across an
| organization or the whole site, using grep is not an
| option.
| inetknght wrote:
| > _That requires you to clone. It 's a minor hassle if
| you search over a single repo, but when searching across
| an organization or the whole site, using grep is not an
| option._
|
| Are you kidding? The finer granularity that searching
| over an organization or whole site makes grep the far
| better choice especially since its output can be fed as
| input to more filtering steps.
| remram wrote:
| I can't understand your point. GitHub _can_ search over
| finer granularities e.g. single repo. And did you
| understand my use case at all, about grep not being an
| option when searching at wider scales?
|
| No, I am not "kidding"... are you?
| inetknght wrote:
| I don't understand grep not being an option when
| searching at wider scales. I have yet to find a wide-
| enough scale that grep can't handle.
|
| And GitHub's search results are _literally_ useless.
|
| > _No, I am not "kidding"... are you?_
|
| Nope
| kawsper wrote:
| Code search have been broken for us for a couple of months,
| it's no longer reliable, I suspect something is wrong with
| their indexer.
| neovintage wrote:
| I'm sorry code search isn't working. I'm more than happy to
| help. If you reach out to me via my email I can see whats up.
| neovintage [at] github [dot] com.
| kawsper wrote:
| I will reach out to my team and send you an e-mail.
| j1elo wrote:
| As others have noted, GitHub Search is not particularly
| reliable. It might make you think you're finding all uses of
| some token, but that can be misleading as not all instances are
| necessarily shown:
|
| https://stackoverflow.com/questions/43891605/search-partial-...
| forlorn wrote:
| Oh, weekly _Github is down_.
| encom wrote:
| Related: Seems like every other day "$PRODUCT is down" is #1 on
| HN. Why? This can't possibly satisfy anyones intellectual
| curiosity.
| Florin_Andrei wrote:
| I thought this was obvious, but apparently it isn't - not
| 100% of things posted on this forum (or any forum, or the
| vast majority of human interactions) are driven purely by
| some abstract "intellectual curiosity" concept.
| encom wrote:
| No shit, you arrogant cretin. I referring to the HN
| submission guidelines, which specifically uses the
| "intellectual curiosity" wording.
| cronix wrote:
| Because many people here actually _use_ the affected
| services?
| hulitu wrote:
| So the solution is easy: don't use it anymore.
| encom wrote:
| I understand that. That doesn't make it interesting.
| brutal_chaos_ wrote:
| Receiving crumbs of info as to WHY a service is down is
| interesting to some. These are also great for tracking
| postmortems (unless they take awhile, then it's a
| separate post).
|
| Speculation is entertaining to many as well. Or perhaps
| this sparks an idea for someone (omg, GH is down ALL THE
| TIME, time to build a novel competitor!).
|
| And given the crap state status pages are in these days
| (stop showing green when your site is down!), these are
| great for knowing when a service is operating again.
|
| EDIT: words r hard
| judge2020 wrote:
| Probably something about centralization and relying on a
| company for keeping your business running. But, of course
| GitHub Enterprise Server deployments aren't down, so anyone
| that really needs uptime can pay for the privilege (or can
| pay for GH One which has a "30-minute SLA").
| [deleted]
| fartcannon wrote:
| What options exist for decentralized source discovery
| indexes/pages? Like the Pirate Bay used to be (might still be, I
| haven't looked in a while).
|
| Just searchable links to torrent based git repos.
|
| Googling suggests there was once (and still might be) a
| 'GitTorrent' which is a fantastic name for the service.
| qbasic_forever wrote:
| Git was designed for decentralized use. It's not as slick and
| polished as github, but it can get the job done and scales to
| infinity (the linux kernel has thousands of contributors and
| runs entirely dencentralized with git).
|
| For read-only access you can host a git repo on any static file
| store, like S3, netlify, digital ocean, etc. Just
| rsync/rclone/upload a bare repo from your machine and you're
| done: https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-
| Git...
|
| For write access it's more difficult without running your own
| server (which is super easy with gitea, gogs, etc. and just a
| couple clicks to setup on popular hosts like digital ocean).
| You could take an entirely decentralized approach and run
| things like the linux kernel--all patches (aka pull requests)
| get sent to an email list where they're reviewed, discussed,
| and integrated by the maintainer of the read-only repo.
| LinusS1 wrote:
| Azure just needed a quick reboot
| expliced wrote:
| Commits I pushed were not showing up on my pull-request in the
| UI, was wondering what was going on. This explains it.
| superkuh wrote:
| Play stupid games with centralized entities, win stupid
| centralized prizes. I can understand using github/lab/etc if
| you're forced to by work in order to earn money in order to live.
| But willingly choosing them for personal projects is just stupid.
| TobiasA wrote:
| "Using GitHub for personal projects is stupid" is a hell of a
| take.
| superkuh wrote:
| So was, "Facebook is a bad idea." in 2008. But the course of
| a proprietary social network run by a single corporation only
| has one outcome. It's just a matter of time.
| brahyam wrote:
| I believe this incident has been happening for more than 30 mins.
| I've had problems with gradle pipelines failing without reason
| for at least the last 6 hours
| alex_g wrote:
| On a related note, I have been unable to transfer a repo for two
| weeks now. The sender is able to initiate it but no notification
| ever appears on my end. If anyone knows how to get the attention
| of someone at GitHub please let me know. I wish GitHub had a paid
| support option that was reasonably priced for a single inquiry.
| [deleted]
| yarcob wrote:
| Even though the status page shows all green checkmarks, we are
| still experiencing issues. The web interface is showing stale
| data and we can't perform merges in git.
|
| Looks like some of their systems got out of sync and they aren't
| done resynchronising yet.
| robbyt wrote:
| Sounds like Azure... Solid 500s from API, but status is green.
| yarcob wrote:
| Update: After nothing happened, I rewrote the last commit and
| force pushed to trigger an update. That seemed to do the trick.
|
| I have no inside knowledge, but from the outside it looks like
| whatever outage they had broke the propagation of commit data
| from the git repo to to their MySQL database. Maybe they use
| webhooks for their internal systems as well? That would explain
| why they first saw issues with webhooks, and later issues with
| pull requests that might depend on them.
|
| It also looks like after they fixed the issue, they didn't
| replay the failed notifications. That's why I saw stale data
| even after they apparently fixed the issue. Then my push to the
| repo seemed to trigger an update, and now it's back in sync.
|
| I'm curious to hear what really happened, but I doubt this
| incident is important enough to anyone to warrant a detailed
| blog post.
| faitswulff wrote:
| After seeing this happen over and over again, I wonder if
| status pages should even be run by their own parent companies.
| freakynit wrote:
| Is anyone facing issue with github pages too? Updated content not
| getting reflected?
| exikyut wrote:
| "The world might be ending."
|
| "Update: The world is slightly closer to ending."
|
| "Update: If more thing breaks the world will implode."
|
| "Update: This issue has been resolved."
|
| Me: _[Wonders what happened]_
|
| - Every outage status tracker ever
| drewcoo wrote:
| But there was customer communication. It's checked off the
| list, see?
| gkop wrote:
| A few years ago GH actually had a fairly transparent status
| page that displayed error rates across components over narrow
| timeslices. I guess they canned it not too long after their
| $100M a16z raise.
| WrtCdEvrydy wrote:
| Because they are glorified marketing pages.
| itsjloh wrote:
| GitHub generally do monthly uptime reports that go into more
| detail about any outages they experienced, see the latest one
| here https://github.blog/2021-02-02-github-availability-report-
| ja...
| darknavi wrote:
| One day you'll learn about the one Russian submariner who saved
| all of our web services from destruction.
| cyberlurker wrote:
| Reference: https://en.m.wikipedia.org/wiki/Stanislav_Petrov
| gknoy wrote:
| I really enjoy seeing the postmortems that some teams publish
| after these kinds of things.
| edoceo wrote:
| In other news, folks who are self-hosting GitLab are in great
| shape. And once the GitHub issues are sorted we can make sure to
| push to those public end-points of convenience.
|
| Self hosting critical services (email, chat, git, etc) is not a
| terrible idea. Of course, CBA/risk factor for your team.
| cortesoft wrote:
| So when your self hosted instance goes down and you are working
| on it, do your coworkers post message saying "in other news,
| GitHub.com is in great shape"?
| edoceo wrote:
| No, cause chat is down too. Solvlem Probbed!
___________________________________________________________________
(page generated 2021-02-08 23:00 UTC)