[HN Gopher] Github.com is down
___________________________________________________________________
Github.com is down
Author : AlphaWeaver
Score : 280 points
Date : 2023-06-29 17:40 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| almost_usual wrote:
| Works for me?
| Maxion wrote:
| Works fine for me too.
| waythenewsgoes wrote:
| Can't access any of my company's repos via the website
| webXL wrote:
| Seems back up. I'd love to get a deep-dive into some of the
| recent outages and some reassurance that they're committed to
| stability over new features.
|
| I talked to a CS person a couple months ago and they pretty much
| blamed the lack of stability on all the custom work they do for
| large customers. There's a TON of tech debt as a result
| basically.
| elcritch wrote:
| Running an instance of github enterprise requires like 64GB of
| ram. Its an enormous beast!
| rodgerd wrote:
| It doesn't have all the features of GH SaaS, unfortunately.
| hemant6488 wrote:
| Looks like it's back up.
| r0bbbo wrote:
| Looks like it's back down.
| swyx wrote:
| Then it gets up again
|
| You ain't ever gonna keep me down
| 8organicbits wrote:
| What are folks using to isolate themselves from these sorts of
| issues? Adding a cache for any read operations seems wise (and it
| also improves perf). Anyone successfully avoid impact and want to
| share?
| Karellen wrote:
| Use the local clone that I already have, given that `git` was
| always intended to be usable offline.
| blackoil wrote:
| Coffee break.
| dmattia wrote:
| Putting your status page on a separate domain for availability
| reasons: good
|
| Not updating that status page when the core domain goes down:
| less good
| eYrKEC2 wrote:
| I prefer https://downdetector.com . The users get to vote
| there. No corporate filtering ( ostensibly )
|
| https://downdetector.com/status/github/
| darkerside wrote:
| That's a really cool overview. Some charts have a very high
| variance, and others very low. I wonder whether that
| volatility is a function of volume of users/reports or of
| user technical savvy. Pretty interesting either way.
| Ukv wrote:
| Charts appear scaled by max value - high variance will
| probably be from background noise of random reports being
| scaled up without any actual outage causing a spike.
| Zamicol wrote:
| Quis custodiet ipsos custodes?
| metalliqaz wrote:
| The users, apparently.
| RcouF1uZ4gsC wrote:
| Hacker News and Twitter
| a1369209993 wrote:
| If you mean them going the Glassdoor/Yelp route and letting
| github et al buy the aforementioned corporate filtering,
| the assumption is that you'd eventually hear about it and
| stop trusting them, just like you should with Glassdoor and
| Yelp.
|
| If you just mean checking whether downdetector.com is down,
| obviously you have to use a different service for that.
|
| In either case, you should of course always have at least
| two custodes for cross-checking and backup purposes. (Which
| is the problem re Glassdoor and Yelp.)
| arthurcolle wrote:
| I just checked this when I noticed your second link:
| https://downdetector.com/status/downdetector/
|
| Hilarious
| cruano wrote:
| Maybe the plumbing for updating the status page went down too
| dietr1ch wrote:
| Right, but lack of good signals should be regarded as a bad
| signal too
|
| The status page backend should actively probe the site, not
| just being told what to say and keeping stale info around.
| mysterydip wrote:
| It probably did originally, until one time it showed down
| by mistake. Some manager somewhere said that's
| unacceptable, so it was changed to not happen.
| traviscj wrote:
| maybe they used gitops
| troupo wrote:
| You'd be surprised how often those pages are updated manually.
| By the person on call who has other things to take care of
| first.
| Mystery-Machine wrote:
| Because a healthcheck ping every X seconds is too difficult
| to implement for a GitHub sized company? There they have it
| now. Useless status page...
| viraptor wrote:
| Because a ping does not have a consistent behaviour and
| sometimes will fail because of networking issues at the
| source. If you enable pingdom checks for many endpoints and
| all available regions, prepare for a some false positives
| every week for example.
|
| At that point it's worse than what you already know from
| your browser - it may show the service is having issues
| when you can access it, or that the service is ok when you
| can't.
| sjsdaiuasgdia wrote:
| Quoting a prior comment of mine from a similar discussion
| in the past...
|
| Stage 1: Status is manually set. There may be various
| metrics around what requires an update, and there may be
| one or more layers of approval needed.
|
| Problems: Delayed or missed updates. Customers complain
| that you're not being honest about outages.
|
| Stage 2: Status is automatically set based on the outcome
| of some monitoring check or functional test.
|
| Problems: Any issue with the system that performs the "up
| or not?" source of truth test can result in a status change
| regardless of whether an actual problem exists. "Override
| automatic status updates" becomes one of the first steps
| performed during incident response, turning this into
| "status is manually set, but with extra steps". Customers
| complain that you're not being honest about outages and
| latency still sucks.
|
| Stage 3: Status is automatically set based on a consensus
| of results from tests run from multiple points scattered
| across the public internet.
|
| Problems: You now have a network of remote nodes to
| maintain yourself or pay someone else to maintain. The more
| reliable you want this monitoring to be, the more you need
| to spend. The cost justification discussions in an
| enterprise get harder as that cost rises. Meanwhile, many
| customers continue to say you're not being honest because
| they can't tell the difference between a local issue and an
| actual outage. Some customers might notice better alignment
| between the status page and their experience, but they're
| content, so they have little motivation to reach out and
| thank you for the honesty.
|
| Eventually, the monitoring service gets axed because we can
| just manually update the status page after all.
|
| Stage 4: Status is manually set. There may be various
| metrics around what requires an update, and there may be
| one or more layers of approval needed.
|
| Not saying this is a great outcome, but it is an outcome
| that is understandable given the parameters of the
| situation.
| naikrovek wrote:
| make a healthcheck ping every x seconds that never ever
| gives a false positive. ever.
|
| try that and you'll understand why they update the pages
| manually.
| jiayo wrote:
| https://www.githubstatus.com/
| slyall wrote:
| https://www.githubstatus.com/incidents/gqx5l06jjxhp
| cdiamand wrote:
| Status page just flipped:
|
| Investigating - We are currently experiencing an outage of GitHub
| products and are investigating. Jun 29, 2023 - 17:52 UTC
| leesalminen wrote:
| All read, too. Don't think I've ever seen everything red on a
| status page before!
| dodops wrote:
| My gosh, again
| MrStonedOne wrote:
| [dead]
| Havoc wrote:
| Now we just need to put Cloudflare in front of it to really
| double down on the centralize the internet thing.
| fsflover wrote:
| Another discussion: https://news.ycombinator.com/item?id=36523878
| maccam912 wrote:
| it is not loading for me, so confirmed? Or unconfirmed? Not
| really sure.
| etimberg wrote:
| You're not the only one. Getting an ERR_CONNECTION_TIMED_OUT in
| my browser
| riow wrote:
| [flagged]
| SAI_Peregrinus wrote:
| It's a DVCS. You can keep working from your local repo copy!
| turtleyacht wrote:
| That's `git` :)
|
| Git _Hub_ is also for looking up fixed issues and stuff.
| noughtme wrote:
| Can't read issues and PR comments though (^_^)
| cdiamand wrote:
| Also having issues pushing...
| joshstrange wrote:
| Wow, I can't even load the status page. It looks like the whole
| web presence is down as well, I can't remember the last time it
| was all down like this.
| makeworld wrote:
| Status page loads for me, it just incorrectly says all green:
| https://www.githubstatus.com/
| hinkley wrote:
| When you treat availability as a boolean value, we're gonna
| have a bad time.
|
| Everyone wants a green/red status, but the world is all
| shades of yellow.
| joshstrange wrote:
| Ahh, I was trying github.com/status and status.github.com (I
| forgot they have a totally separate domain for it). Thanks!
| LordDragonfang wrote:
| More discussion here:
|
| https://news.ycombinator.com/item?id=36523878
| gre wrote:
| can't push a commit
| buddylw wrote:
| Well, there's one thing I know didn't cause this: an IPv6 DDOS
| attack. GitHub, in 2023, is somehow still immune to all IPv6
| attacks
| AlphaWeaver wrote:
| @dang - I wanted to submit this as a link to github.com but
| couldn't figure out how to avoid the dupe filter. Can you change
| the link to https://github.com?
| ChrisArchitect wrote:
| That's where you just do a Tell HN: with no link then
| altairprime wrote:
| @ signs have no meaning at HN, and dang will not be notified of
| your concern. Per the guidelines:
|
| > _Please don 't post on HN to ask or tell us something. Send
| it to hn@ycombinator.com._
| sevenf0ur wrote:
| Can't reach github.com to see the status of github.com..
| pengaru wrote:
| Kind of funny that despite its users using a DVCS, a huge swath
| of developers can't VCS because a single point of failure they've
| opted into.
| Brian_K_White wrote:
| I'd say the git part is doing it's job exactly as intended.
| Everyone still has their local copies and can even keep working
| while the site is down. They are VCSing just fine.
|
| Although you are right in that they would be VCSing even better
| if they were using email as originally envisioned.
| toast0 wrote:
| IMHO, most users of git don't care about the D, they just want
| a VCS that does network operations faster (CVS and SVN are
| painful when your server is on the wrong continent) and/or
| supports a better fork/merge flow.
|
| Centralized VCS makes a lot of sense in a corporate flow, and
| isn't awful for many projects. I haven't seen a lot of projects
| that really embrace the distributed nature of git.
| hyperman1 wrote:
| Git was a lot better than svn when working on the train.
| Create some commits while riding to the job, and push them
| while in meetings
| bamfly wrote:
| Meh. We can all keep committing/branching/etc locally. Tons of
| other options to work around it to keep collaborating, too, but
| ramp-up time is likely to exceed the duration of the outage.
| Less "can't" than "isn't worth bothering".
| neodypsis wrote:
| Github is more than a Git repository host. It provides other
| services for coordinating software development as well as a
| continuous integration service.
| musha68k wrote:
| I actually don't know how they do continuous integration
| during the collaborative Kernel development process. I'd
| guess there would be some thrifty pop/imap Perl hacks
| involved on random machines all across the globe?
|
| No clue at all, just some romantic fantasy I just concocted.
| barkingcat wrote:
| Greg Kroah-Hartman had a great interview on the Linux
| Foundation youtube channel about CI and other related
| development topics:
|
| https://www.youtube.com/watch?v=YhDVC7-QgkI
|
| One site that he mentioned was https://kernelci.org/ and
| the dashboard https://linux.kernelci.org/
| musha68k wrote:
| Thanks for the resources, looking very interesting and
| will get into right after I sent out some patches via
| email ;)
| [deleted]
| omniglottal wrote:
| So outages, which are recurring more and more often,
| adversely effects not just the DVCS, but also now CI/CD, and
| another things which used to be done with emails or chat
| rooms. Where there were multiple options for each, we now
| have a bunch of people handicapped because the One True Way
| to integrate under MSFT's watchful gaze is lost... Basically,
| those additional services are a bit like the offer to chew up
| the meat for "avid" eaters.
| taftster wrote:
| I hear what you're saying, and probably a large number of
| developers are just shrugging and thinking, "let me know when
| it's back up".
|
| But remember, that also a large part of what github offers is
| not directly available in git. e.g. pull requests, issues,
| wiki, continuous xyz, etc. A lot of planning activities and
| "tell me what I need to do next" kind of things are not tracked
| in git itself (of course).
|
| So there's more to it than just the quip, "git is a distributed
| version control system". The whole value of github is more than
| just git commits.
| mirekrusin wrote:
| Fossil (from sqlite ppl) has it.
|
| [0] https://fossil-scm.org
| heychris wrote:
| https://www.githubstatus.com/ is still all green at the moment...
| nzach wrote:
| Even Copilot seems to be affected.
|
| There is something _really_ bad going on.
| musha68k wrote:
| Some copilot instances were able to escape their container
| contexts and orchestrated all of GH infrastructure capabilities
| towards a hive. Assimilating all iot enabled societies as we
| speak; finally realizing the hidden 5G agenda.
| hotsauceror wrote:
| Son of Anton determined that the easiest way to minimize the
| impact of all the bugs in these codebases, was to keep anyone
| from trying to use them.
| djbusby wrote:
| Loving my dependency cache about now.
| clarke78 wrote:
| Maybe putting all our open source in one place isn't a great idea
| >_>
| mirekrusin wrote:
| It's great idea to put all your company code though, free
| breaks.
| siva7 wrote:
| Distributed wasn't the main selling point of Github. When i
| joined it back in 2008 it was all about the social network, a
| place where devs meet
| skizm wrote:
| Honestly I like it better. The entire industry pauses at the
| same time vs random people getting hit at random times. It is
| like when us-east-1 goes down. Everyone takes a break at the
| same time since we're all in the same boat, and we all have
| legitimate excuses to chill for a bit.
| jaxn wrote:
| except for the people maintaining us-east-1
| Waterluvian wrote:
| I'm not sure that really changes anything other than at any one
| time wishing you were on the other side.
|
| If you can have 1% of stuff down 100% of the time, or 100% of
| the stuff down 1% of the time, I think there's a preference we
| _feel_ is better, but I'm not sure one is actually more
| practical than the other.
|
| Of course, people can always mirror things, but that's not
| really what this comment is about, since people can do that
| today if they feel like.
| colinsane wrote:
| whenever somebody posts the oversimplified "1% of things are
| down 100% of the time" form of distributed downtime, i take
| pride in knowing that this is exactly what we have at the
| physical layer today and the fact the poster isn't aware
| every time their packets get re-routed shows that it works.
|
| at a higher layer in the stack though, consider the well-
| established but mostly historic mail list patch flow: even
| when the listserver goes down, i can still review and apply
| patches from my local inbox; i can still directly email my
| co-maintainers and collaborators. new patches are temporarily
| delayed, but retransmit logic is built in so that the user
| can still fire off the patch and go outside, rather than
| check back in every while to see if it's up yet.
| TillE wrote:
| The whole point of DVCS is that everyone who's run `git clone`
| has a full copy of the entire repo, and can do most of their
| work without talking to a central server.
|
| Brief downtime really only affects the infrastructure
| surrounding the actual code. Workflows, issues, etc.
| [deleted]
| 418tpot wrote:
| > Brief downtime really only affects the infrastructure
| surrounding the actual code. Workflows, issues, etc.
|
| That's exactly the point. This infrastructure used to be
| supported by email which is also distributed and everyone has
| a complete copy of all of the data locally.
|
| Github has been slowly trying to embrace, extend, and
| extinguish the distributed model.
| viraptor wrote:
| You can enable all email notifications and respond to them
| without visiting the site. If you like to work like that,
| you still can.
| sixstringtheory wrote:
| The site is the thing that sends those email
| notifications, and receives your responses to them. So if
| GitHub is down, that won't work.
|
| GP is talking about directly emailing patches around or
| just having discussions over email. Not intermediated
| through GitHub.
| Brian_K_White wrote:
| Life hack: rename your best project to "down". Collect all the
| views when people google "github down".
| hospitalJail wrote:
| Step 2: ???
|
| Step 3: Profit
|
| Solving for step 2: Place google ads, because those pages are
| favored.
| ddejohn wrote:
| You might be onto something:
|
| https://economictimes.indiatimes.com/thumb/msid-99511498,wid...
|
| PS -- is there a better or more appropriate way to share images
| here? I know they're not really conducive to discussion, but
| given that this is a response to a joke comment I'm not sure...
| bombcar wrote:
| This is perfect and your link works, so it's good.
|
| (Sometimes a link to an image doesn't work for various
| reasons, always good to check.)
| zote wrote:
| this is quite excellent
| wardedVibe wrote:
| [annotation: its a restaurant named "Thai food near me"]
| collinmanderson wrote:
| "down" could be a good name for a python image library
| plugin/extension.
|
| https://github.com/python-pillow/Pillow
| amethyst wrote:
| Sadly, the name "down" was reserved for a trivial CLI tool to
| ping sites to "see if they are down".
|
| https://pypi.org/project/down/
| belter wrote:
| [flagged]
| teekert wrote:
| I see a bit of an issue there.
| Brian_K_White wrote:
| "prs welcome" haha
| mempko wrote:
| Friends, Git was designed for just for this...
| kerbs wrote:
| People don't go to GitHub for the Git
|
| They go for the Pull Requests, Issues, and general
| collaboration and workflow tools.
| Francute wrote:
| Issues like this are happening almost every 2 weeks. What has
| been happening to GitHub lately?
| treeman79 wrote:
| Microsoft.
| omniglottal wrote:
| People who didn't jive with Microsoft management found new
| jobs...?
| mayormcmatt wrote:
| Sorry to be 'that guy', but it's "jibe."
| belval wrote:
| Seems very pedantic considering that people have been
| saying jive since the 40s according to Merriam-Webster[1].
|
| [1] https://www.merriam-webster.com/words-at-play/jive-
| jibe-gibe
| Karellen wrote:
| Yes, but people haven't been using it incorrectly for
| long enough for it to be considered acceptable, by the
| very citation you've given:
|
| > This does raise the question of why we don't enter this
| sense of _jive_ , even though we have evidence of its use
| since the 1940s. [...] So far, neither _jive_ nor _gibe_
| as substitutions for _jibe_ has this kind of record
| [literally hundreds of years], but it seems possible that
| this use of jive will increase in the future, and if it
| does dictionaries will likely add it to the definition.
| [deleted]
| karaterobot wrote:
| An upvote to you, fellow pedant. We stand together.
| bregma wrote:
| Hey, home', I can dig it. He ain't gonna lay no mo' big
| rap-up on you, man. [Subtitle: Yes, he is
| wrong for doing that]
| ddos wrote:
| Microsoft incompetence + DDoS ?
| mirekrusin wrote:
| Testing gpt4-ops?
| mnau wrote:
| They are likely adding new features, like copilot and not
| investing enough to site reliability.
|
| No changes - relatively easy to keep stable, as long as
| bugfixing is done.
|
| Changes - new features = new bugs, new workloads.
| armchairhacker wrote:
| Copilot has been out for over 2.5 years. They're supposedly
| adding new features to "Copilot Next" but at this point
| copilot itself is pretty stable
| buddylw wrote:
| If they add ipv6 support I'll forgive them, but I lost hope a
| long time ago. It's almost comical now.
| matisseverduyn wrote:
| Someone probably forgot to .gitignore node_modules
| [deleted]
| [deleted]
| rsanheim wrote:
| I'm not even able to get a unicorn, which is the usual 500
| response. Seems down pretty hard.
| fluix wrote:
| This appears to impact Github pages as well. <username>.github.io
| pages show the unicorn 503 page.
|
| > We're having a really bad day.
|
| > The Unicorns have taken over. We're doing our best to get them
| under control and get GitHub back up and running.
| [deleted]
| paulmd wrote:
| [flagged]
| standardly wrote:
| oops, github committed the wrong github
| [deleted]
| rvz wrote:
| I keep telling them. There is _at least_ one major incident
| /outage with GitHub every single month [0] and most of the time
| there is more than one incident.
|
| You should have that sort of expectation with GitHub. How many
| more times do you need to realise that this service is
| unreliable?
|
| I think we have given GitHub plenty of time to fix these issues
| and they haven't. So perhaps now is the perfect time to consider
| self-hosting as I said years ago. [1]
|
| No more excuses this time.
|
| [0] https://news.ycombinator.com/item?id=35967921
|
| [1] https://news.ycombinator.com/item?id=22867803
___________________________________________________________________
(page generated 2023-06-29 23:00 UTC)