[HN Gopher] Tell HN: GitHub is down again
___________________________________________________________________
Tell HN: GitHub is down again
Yet somehow https://www.githubstatus.com is ALL GREEN! smh
Author : pupdogg
Score : 358 points
Date : 2021-11-27 20:40 UTC (2 hours ago)
| spapas82 wrote:
| Can't they at least fix their status page?
| https://www.githubstatus.com/ It returns `All Systems
| Operational`. I mean what's the point of having a status page if
| it returns wrong info?
| ranklord wrote:
| It's updated already. Relatively fast... :/
| res0nat0r wrote:
| These things don't/can't get updated instantly. I was doing
| some work as of ~5 minutes ago and it was working fine, and is
| unavailable now. If it is a major outage it will likely be
| updated shortly.
| MrStonedOne wrote:
| My status page for an open source multiplayer game updates
| the status page for each server on a minute by minute bases.
|
| They can do better, they just don't want to.
| derefr wrote:
| No, I don't think they can. An MMO is a very simple system,
| in that there's only one Service Level Indicator (SLI) that
| devs, shareholders, and players all agree on. That SLI is
| "can a player connect to the server, and perform regular
| gameplay actions, without a ridiculous amount of per-action
| latency."
|
| GitHub, meanwhile, is a million different things to a
| hundred million different people. Users of e.g. Homebrew,
| with its big monolithic ports system hosted as a github
| repo, have a very different SLI for Github than do users of
| some language-ecosystem package manager that allows you to
| pull deps directly from Github; than do people who depend
| on GitHub Actions to CI their builds on push; than do
| people doing code-review to others' PRs; than do people
| using Github mostly for its Wiki, or Issues, or downloading
| Releases, or Github Pages, or even just reading single-
| page-with-a-README repos, ala the various $FOO-awesome
| projects.
|
| For many of these use-cases, Github _isn 't_ degraded right
| now. For others, it is.
|
| If you ask for Github (or any service with this many
| different use-cases and stakeholders) to measure by the
| _union_ of all these SLIs, then the service would literally
| never be not-degraded. In systems of sufficient scale,
| there 's likely no point where every single component and
| feature and endpoint of the system is _all_ working and
| robust and fast, all at once. Never has been, never will
| be.
|
| And anything less than just going for the union of all
| those SLIs, is asking Github to exercise human judgement
| over which kinds of service degradation qualify as part of
| their own SLOs. Which is exactly what they're doing.
|
| Certainly, internal to services like this, there are all
| sorts of alerting systems constantly going off to tell SREs
| what things need fixing. But not all of those things
| immediately, or even quickly, or even _ever_ , translate to
| SLO violations. There are some outlier users whose use-
| cases just break the system's semantics, where those use-
| cases just aren't "in scope" for the SLO. As long as those
| users are only breaking the system for _themselves_ , the
| degradation they experience won't ever translate to an SLO
| breakage.
| thih9 wrote:
| You seem to be applying different rules to MMOs and
| Github, and I don't understand why. I'd say that there
| are many ways of looking at this; there exist complex
| MMOs; and one could look at Github from the point of view
| of an average user.
|
| E.g., a bit tongue in cheek:
|
| > An MMO is a very simple system, in that there's only
| one Service Level Indicator (SLI) that devs,
| shareholders, and players all agree on. That SLI is "can
| a player connect to the server, and perform regular
| gameplay actions, without a ridiculous amount of per-
| action latency."
|
| Wouldn't you say that in an MMO of sufficient scale
| there's likely no point where every single component and
| feature and endpoint of the system is all working and
| robust and fast, all at once?
|
| > In systems of sufficient scale, there's likely no point
| where every single component and feature and endpoint of
| the system is all working and robust and fast, all at
| once. Never has been, never will be.
|
| Couldn't we redefine SLIs as "can the user connect to the
| server and perform regular user actions without a
| ridiculous amount of per-action latency"?
| res0nat0r wrote:
| Large orgs (like Github) don't want or use automated status
| updates. There is usually always some service having issues
| in complex systems and immediately updating a public status
| page does more harm than good, including false alarms which
| may not affect anything public facing.
| Dylan16807 wrote:
| If you don't want to report sufficiently small issues,
| you can put that into the code, can't you?
|
| Besides that, how are you going to cause "more harm than
| good"?
| derefr wrote:
| More harm than good to the company's own long-term
| reputation.
|
| A status page is a kind of PR. Think of it like a policy
| for a flight attendant to come out into the cabin to tell
| everyone what's going on when the plane encounters
| turbulence. That policy is Public Relations -driven. You
| only do it if you expect that it's _positive_ PR,
| compared to not doing it -- i.e. if telling people what
| 's going on is _boosting_ your reputation compared to
| saying nothing at all.
|
| If a status page just makes your stakeholders think your
| service is crappy, such that you'd be better off with no
| status page at all... then why have a status page? It's
| not doing its job as a PR tool.
| mjw1007 wrote:
| It seems to me you are now agreeing with the original <<
| They can do better, they just don't want to. >>
| derefr wrote:
| I read the line specifically as " _The employees_ can do
| better; they just don 't want to _try_. "
|
| But that's not true. The _company_ could do better. But
| the individual employees cannot. The individual employees
| are constrained by the profit motive of the company. They
| _are not allowed by corporate policy_ to set up automatic
| status updates, for about the same reason they 're not
| allowed to post their corporate log-in credentials: that
| the result would very likely be disastrous to the
| company's bottom line.
|
| (Though, really, the corporations in most verticals are
| in a race-to-the-bottom in most respects. Even if you
| treat GitHub as a single entity capable of coherent
| desires, it probably doesn't _desire_ to avoid automatic
| status updates. It _needs_ to avoid them, to survive in a
| competitive market where everyone else is also avoiding
| them. People -- and corporations -- do lots of things
| they don 't _want_ to do, to survive.)
| qwertox wrote:
| They have a banner "Investigating - We are investigating
| reports of degraded performance for GitHub Actions. -- Nov 27,
| 20:43 UTC" which should suffice for a start.
| spapas82 wrote:
| Yes now it has been properly updated. However it was down for
| 10-15 minutes before their status page was updated...
| drdaeman wrote:
| There is a point. Even two.
|
| 1. It clearly indicates that automatic systems are failing to
| detect the outage. 2. It also indicates that no one is aware
| about the incident to manually signal the outage (or that there
| is no manual override).
|
| Basically, it makes a difference between "yeah, shit happened,
| we know (and maybe working on it)" and "hah, they don't even
| know themselves".
| res0nat0r wrote:
| Most of these over arching status pages are manually run,
| intentionally by design.
| the_duke wrote:
| Almost no one has automatic status pages anymore.
|
| Partially because these large systems have some kind of
| ongoing issue at any given time, so it's challenging to
| provide a meaningful live status that isn't a bit misleading
| and could cause misdirected panic.
|
| Partially because you don't want to give potential attackers
| (eg ddos) any insight if/how their methods are affecting your
| systems.
|
| Partially because there are SLAs and reputation at risk, and
| you don't want to admit to any more downtime than you
| absolutely have to.
| toshk wrote:
| Yeah it seems company status page lost its infancy stage
| where companies were actually honest about their outages.
| Bit similar what happened to online reviews.
| derefr wrote:
| If you had a _really_ robust system, it 'd be fun to just
| slap a read-only mirror of your internal metrics dashboard
| onto the public Internet for anyone to browse and slice as
| they please. It'd be a brag, kinda.
|
| Of course, in the real world, I don't think there's a
| single IT admin who didn't just start nervously sweating
| and pulling at their collar after reading the above,
| imagining it as something their CEO said and encouraging
| them to target it. Nobody can really do this -- and that in
| turn says something important about how those metrics
| really look in most systems.
| MartijnBraam wrote:
| Sourcehut does
|
| https://metrics.sr.ht/graph?g0.expr=&g0.tab=1&g0.stacked=
| 0&g...
| Erethon wrote:
| Wikimedia does this https://grafana.wikimedia.org/
| Jweb_Guru wrote:
| FWIW, I've worked on systems that have internal SLAs
| orders of magnitude higher than what they promise to the
| public. I think it's more just that there's no advantage
| to doing something like this as long as none of your
| competitors are. The status quo is that people's systems
| are really opaque and vastly underpromise what they
| should be capable of, and in exchange you get to absorb
| some unplanned downtime due to issues with underlying
| systems that you have little control over.
| random_thoughts wrote:
| They are currently updating it one by one
| cube00 wrote:
| ...and firing off automated tweets per product.
| hdjjhhvvhga wrote:
| The only status pages that make sense are the ones maintained
| by a third party, not the owner. Also for technical reasons.
| cyberge99 wrote:
| And legal reasons.
| ImprovedSilence wrote:
| lolol. Clearly it's just degraded service. It seems The
| Management hasn't approved declaring it an outage, that would
| just ruin uptime metrics!
| charles_f wrote:
| It's updated now.
| crecker wrote:
| I would not be impressed if they hear about the down for a
| trending thread on HN.
| ryantgtg wrote:
| I don't work for github, so on a personal note I was about to
| create an issue in a repo, and it wasn't loading. My go to
| "is my router messed up?" check is to load HN because it's so
| reliable and fast. And lo, the top post was about github
| being down!
| crecker wrote:
| Ha. I checked HN for the same reason, I was not able to
| reach Github via university network and I thought that was
| the time my university messed up with DNS.
|
| HN seems to be going down for the massive amount of
| requests!
| JohnFen wrote:
| Honestly, HN is the best status indicator on the net right
| now.
| danjac wrote:
| There's lies, damn lies and SAAS status pages.
| Flankk wrote:
| Status page is also down and is returning a false positive. If
| you check the status page status page it shows the status page
| is down.
| chippiewill wrote:
| Status pages are almost never automated these days, they cause
| more problems than they solve.
|
| To be fair, redditstatus.com is quite nice with their sparkline
| headline metrics. It at least lets you know _something_ is
| happening even if they haven't yet declared an incident.
| yellow_lead wrote:
| Maybe companies should alert on an increased amount of traffic to
| their status pages.
| scrollaway wrote:
| That's honestly a good idea. Does anyone do that?
| gdubicki wrote:
| My company does. :)
| cranekam wrote:
| In my last job our main "is everything okay?" monitoring
| dashboard had RPS, latency, 500 rate etc and a line for the
| number of user issues reported per N requests served. We
| didn't alert on this but it was a useful way to detect issues
| that were subtle enough not to trip up our main alerts.
| hnarn wrote:
| A telco I used to work for did this like two decades ago but
| even better, they mapped incoming support calls (customers)
| to stations, and if more than N came in during a certain
| period for the same DSLAM it triggered some kind of alert.
|
| Same thing happens (should happen) when you visit your ISPs
| website and look for registered downtime -- many requests
| from one zip code from multiple ips should trigger an alarm
| if the isp is competent.
| rozenmd wrote:
| I'm surprised more folks don't monitor their vendors via uptime
| checkers - worth knowing if you'll be able to get anything done
| that day.
| mlazowik wrote:
| I've seen alerting based on the number of Twitter mentions too
| colpabar wrote:
| I think this is how downdetector works, it just looks for
| tweets that are complaining about something not working.
| PaulBGD_ wrote:
| I'm fairly confident they just count a site as down based
| off the popularity of the page.
| kelnos wrote:
| That might be a good idea as a last-resort measure, but if
| you're only finding out problems because customers are telling
| you about them (even indirectly through a signal like this),
| your monitoring and alerting is woefully inadequate.
| szundi wrote:
| fun fact here that no matter how short the automated system
| is set up to alert, customers are faster - no one knows how
| is that possible
| therein wrote:
| Yup, that's another reason why it pays off having a direct
| line of communication to at least some of your top
| customers.
|
| Can't tell you how many times I've had a customer write to
| me, only to be receiving some automated alerts 30-40
| seconds later.
| lanstin wrote:
| And the best customers for this are the pickiest annoying
| ones
| hnarn wrote:
| It's not last resort, you should do both because you're
| alerting on different things.
| qubyte wrote:
| Close that loop: Once the traffic goes above a threshold
| transition it to red!
| [deleted]
| hk1337 wrote:
| Homebrew is also not working since it relies on GitHub a lot.
| [deleted]
| gime_tree_fiddy wrote:
| Is it possible for GitHub to mirror the releases in multiple
| different places(they likely do that, but I mean complete
| isolation where an outage like this doesn't break the downloads).
| Maybe like a proxy to object store, so it is a little more
| reliable(a setup such as this, should have less moving and custom
| parts).
|
| So in a moment like this, you can convert
| https://github.com/opencv/opencv/archive/4.5.3.zip to
| https://archive.github.com/opencv/opencv/archive/4.5.3.zip. Maybe
| an implicit agreement of somewhat stale data by add the sub-
| domain "archive.". They'll try to maintain low sync times on a
| "best effort basis".)
| rvz wrote:
| Again? Oh dear.
|
| I think I lost track of how many times they went down since,
| suggesting to self-host, repeatedly. [0] So they seem to continue
| to be very unreliable, I expected them to be up without any
| issues for a month.
|
| Anyway, going all in on GitHub once again makes no sense. So at
| least have a (self-hosted) backup solution to this.
|
| [0] https://news.ycombinator.com/item?id=27366397
| saltmeister wrote:
| it's not
| lol768 wrote:
| Why does their status page have components only for issues/pull
| requests? What about browsing e.g. repository code, downloading
| releases etc?
|
| There should be a general "web" component marked as unavailable -
| I can't even hit the homepage.
| [deleted]
| thebradbain wrote:
| Right in the middle of replying with a lengthy comment on a code
| review, too :). Guess we'll find out if the server persisted it
| or not!
| szundi wrote:
| When it's more that 5 minutes of typing, I always select all
| and press ctrl-c. And then submit. Saved my time several times.
| brundolf wrote:
| Usually just clicking the back button, your browser will
| remember the form field contents
| zoomablemind wrote:
| I got greeted by the image of the frightened Octocat, saying
| "Oops!!! 500". Thought, that I broke something.
|
| ...Copilot must've been bored over the holidays, so probably went
| on to find a way and snacked on a few freshly updated repos,
| whichever were close by, then covered it all up as just a
| degraded perf incident... You've gotta feed da beast!
| daxfohl wrote:
| Curious to see the postmortem here. Who makes production changes
| on Thanksgiving weekend!?!
| [deleted]
| daxfohl wrote:
| Granted, it's probably the best time of the year for github to
| break, precisely _because_ nobody else is pushing production
| changes....
| bifrost wrote:
| You don't want to know the answer to this question. I'm totally
| serious too.
| riksucks wrote:
| Seems like the webhooks that populate most of the pages don't
| work, and hence leaves you with an incomplete github page. For
| some, no page at all
|
| But it also seems like that apart from the webhooks, the APIs and
| the pages served are slower too.
| EugeneOZ wrote:
| Dear Github developers, I brought some love and hugs for you. And
| a hot coffee with chocolate.
|
| Not everyone hates you today, don't be upset about toxic posts
| like this one.
| daptaq wrote:
| > Not everyone hates you today, don't be upset about toxic
| posts like this one.
|
| No, do get upset. You have contributed to the centralization
| and increasing dependence on a single entity, while
| piggybacking of the advances that distributed version control
| have brought about. You have blurred the difference between Git
| and your service, to the point that people don't know about the
| former, and routinely abuse it. You have normalized the
| fundamentally inconvenient "pull request" workflow as the
| default for a lot of software development (introducing the
| artificial steps of "forking" and proposing a "pull request"),
| to such a degree that people now complain when they are
| confronted with anything else. This is not a one-off issue, and
| isn't fixed when the site is operational again. You deserve all
| the criticism, and no not coffee chocolate.
| EugeneOZ wrote:
| One day you will wonder how much you were upset about not so
| critical things.
|
| The lifespan of our civilization is just a glimpse. Do you
| really think your toxic comment will make it brighter for a
| moment?
|
| Look at this image:
| https://www.nasa.gov/feature/goddard/2019/hubble-
| astronomers...
| raro11 wrote:
| Did you think yours would?
| activitypea wrote:
| Imagine letting toxic HN comments get to you lmao
| Stampo00 wrote:
| By labeling them toxic, it implies the existence of non-toxic
| HN comments. Fringe researchers have been searching for
| years, but mainstream scientists regard non-toxic HN comments
| as a form of cryptid.
| samuelroland wrote:
| it seems to be up again...
| culi wrote:
| same for me but the status page doesn't say so
| dgellow wrote:
| Oh wow, EVERYTHING is red or orange in the status page. That's
| unusual.
| rglover wrote:
| https://cheatcode.co/opinions/control-your-user-data
| arendtio wrote:
| I have a website hosted on Github pages which uses ServiceWorkers
| to be accessible when being offline (cached). Sadly, Github
| serves some error page currently and kills the ServiceWorker
| functionality that way...
| gray_-_wolf wrote:
| Github.com does not work properly (borderline on "at all") in a
| browser, and I cannot download a release of one project via curl.
| This sucks. And is definitely not just "degraded performance for
| GitHub Actions".
| cube00 wrote:
| It's funny seeing the description say "degraded availability"
| but if you hover over the red icon you get "major outage"
| noir_lord wrote:
| degraded covers everything from 1 in 100,000 requests is slow
| through to "the data centres all simultaneously caught fire".
|
| It's weasel words in their purest form.
|
| Right up there with "We apologise for any inconveniences you
| may have experienced".
|
| If you are apologising it's because you _know_ you
| inconvenienced someone.
|
| I'd genuinely have more respect if instead they just said "We
| fucked up, we'll do better".
|
| It's at least honest.
| staticassertion wrote:
| Yeah I noticed because I'm trying to pull down a protobuf
| release binary.
| veltas wrote:
| Reminder that git is distributed(TM)
| [deleted]
| revskill wrote:
| Rails is hard to scale.
| activitypea wrote:
| onejoke.png
| robertwt7 wrote:
| omg it is still down!
| bredren wrote:
| There goes my docs PR for `drf-turbo`.
|
| Guess I'll work on my stuff.
| albeebe1 wrote:
| Here i am on a Saturday afternoon learning how to programatically
| interact with GitHub using go-git, banging my head on my desk
| because the code should work, but im getting cryptic errors. I'm
| searching stackoverflow, nobody seems to have encountered the
| errors i'm getting (that typically means i'm doing something
| wrong)...... oh GitHub is down. To GitHubs credit, they've been
| very reliable for me over the years, it didn't even cross my mind
| that they could be down.
| busymom0 wrote:
| Ah! I am brand new to learning flutter and was trying to change
| the Flutter channel with the command:
|
| flutter channel master
|
| And it kept failing with:
|
| ------
|
| git: remote: Internal Server Error.
|
| git: remote:
|
| git: fatal: unable to access
| 'https://github.com/flutter/flutter.git/': The requested URL
| returned error: 500
|
| Switching channels failed with error code 128.
|
| ------
|
| thought I was doing something wrong and spent some time
| troubleshooting.
| vanusa wrote:
| _It didn 't even cross my mind that they would be down._
|
| As soon as you start banking on "the cloud" to get useful work
| done, the very first thing that should enter your mind is:
|
| "Any of these nifty services can just turn into a pumpkin at
| any time"
| dlsa wrote:
| The real kicker is that they're still other people's
| computers. There's definite benefits as well as costs.
| Ignoring either is not a good idea.
|
| Centralisation into just a few large providers is another bad
| idea we're heading towards or have already arrived at.
| NaturalPhallacy wrote:
| "The cloud is just other people's computers." is the most
| succinct way I've heard it.
| mindcrime wrote:
| Much like we have the "Fallacies of Distributed
| Computing"[1], we probably need (if somebody hasn't already
| created) a "Fallacies of The Cloud".
|
| Fallacy #1 - The cloud is reliable
|
| [1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_c
| ompu...
| chrisco255 wrote:
| Can't access Vercel because I used Github for authentication. I
| guess I'm done with using centralized authentication services.
| incognito_mode wrote:
| Try logging in via email. It'll send you a code to cross-check.
| Voila!
| dmitshur wrote:
| In case you find it interesting, I've had luck removing an
| authentication SPOF1 by using IndieAuth on my personal site2. I
| wish it were an option on more of the sites where I need to
| sign in.
|
| 1 https://twitter.com/dmitshur/status/1223304521767669761
|
| 2
| https://github.com/shurcooL/home/commit/bb504a4ef0d7c552d363...
| aliswe wrote:
| The error message for me, when trying to push, was "Make sure you
| have the correct permissions" and also something about repo not
| found ...
| hk1337 wrote:
| At least CodeSpaces is working
|
| https://i.imgur.com/x4IFTwY.png
|
| *EDIT* Now CodeSpaces is gone
| sidarape wrote:
| All red now.
| ejanus wrote:
| Not only GitHub from my side. I can't access Telegram from
| browser.
| ronyfadel wrote:
| Explains why I had less app downloads today.
|
| (My app binaries are hosted on Github; not sure if it's the way
| to go but hey it works).
| Stampo00 wrote:
| Not today it doesn't. ;-)
| brw wrote:
| At least this outage allowed me to discover this cute pixel art
| octocat loading icon on the activity feed. Never noticed it
| before because I believe it always loads near instantaneously, or
| maybe I just never paid enough attention to it.
|
| https://i.imgur.com/6Uwlwh7.gif
| 13415 wrote:
| Of course it's down. That is the _one and only_ time I wanted to
| check something on it on Saturday evening.
| tisryno wrote:
| This is exactly what's happened to me also, oh well, back to
| procrastinating
| rlucas wrote:
| I also had intermittency on 2021-11-27 between 1-2 PM PT. Would
| work one minute, not the next, retry and it works. Feels round-
| robin-y kind of Heisenbug.
| axiom92 wrote:
| Looks like it's up now.
| zapf wrote:
| Wish we could stop using such msft run nonsense, and build a p2p
| decentralised PR system. What's the challenges we need to address
| to make it happen?
| pshc wrote:
| That would be great. But Github is a massive coordination point
| with network effects and free tiers so it's not so easy to
| compete. Some other challenges:
|
| - despite being decentralized, you still need moderation and
| access control
|
| - open source projects typically aren't great at frontend UI/UX
|
| It would be neat if I could publish git repos on IPFS and
| receive patches from people. It's just hard to compete with a
| centralized, CDN'd service where you can click and get a result
| in 100ms.
|
| edit: cribbing from sibling comment, it looks like Radicle has
| thought hard about decentralized git:
| https://radicle.xyz/blog/radicle-link.html
| vimda wrote:
| > p2p decentralised pr system
|
| You mean Git? The Linux Kernel for e.g. does this just fine.
| dataflow wrote:
| > What's the challenges we need to address to make it happen?
|
| I imagine the ability to resist $billion offers when you're
| successful would be one.
| gnubison wrote:
| Hmm, great idea. We could build on Git as a storage medium, and
| we'd need a durable and reliable way to exchange patches.
| Something like ...email? $ man -k patches
| git-am(1) - Apply a series of patches from a mailbox
| git-format-patch(1) - Prepare patches for e-mail submission
| git-imap-send(1) - Send a collection of patches from stdin to
| an IMAP folder git-send-email(1) - Send a collection of
| patches as emails
|
| Ironic that you're pondering Git as a solution to Github's
| self-created centralization problem.
| pshc wrote:
| I think they're talking more about the Pull Request interface
| with nice diffs, comments, automated tests and builders,
| status checks and so on...
|
| You totally could develop in the kernel workflow, everything
| done over email, but are you willing to?
| gnubison wrote:
| Yes :)
| sodality2 wrote:
| > p2p decentralised PR system
|
| My first thought was https://radicle.xyz/.
|
| What a cruel twist of fate. Their downloads page is hosted on
| Github Pages, which is down right now!
| https://radicle.xyz/downloads.html
|
| In any other situation I'd recommend it since it really was the
| tool of choice for decentralized git repos with lots of cool
| features. (Despite the "Empowering developers with DeFi
| building blocks like NFTs, token streams, and more" mentioned
| on the home page which throws me off)
|
| Edit: It seems like the download page is up and working now.
| slmjkdbtl wrote:
| HN for the most time is a better status page for big sites like
| Github.
| jerbearito wrote:
| Thankfully command line / desktop app still work
| minedwiz wrote:
| Thanks for pointing that out. I needed to clone a new repo.
|
| EDIT: nope, command line is down, too.
| jerbearito wrote:
| Odd, sorry. I'm able to push and pull from one of my private
| repos. [Edit: no longer accurate. everything has stopped
| working]
| Gsydvdndh12876 wrote:
| Would be interesting to know PRE-MS and POST-MS acquisition
| statistics on downtime...
| deeblering4 wrote:
| The tone of this post could be improved by describing the problem
| objectively, and there is no need to say "down again"
| dylan604 wrote:
| Is saying "down again" inaccurate?
| colejohnson66 wrote:
| It's _technically_ correct, but, at least for me, it has the
| connotation of it being a common enough thing, such as
| something that happens every week. For something with over
| 99.9% (99.99?) uptime, it 's just another fluke occurrence.
| noir_lord wrote:
| I vividly remember the last time they really went down, I
| was in the middle of an important deploy and since "we" (it
| predates me) didn't consider "github been on fire" a
| critical dependency it all went a bit sideways.
| julianlam wrote:
| Not to mention the vitriol over a manually updated status page
| not updated the instant a service is down.
|
| I mean, the moment it went down I'm sure Github's SREs were
| paged. Give them a minute to process it first, jeez.
| MrStonedOne wrote:
| I have a status page that is automatically _and_ manually
| updated and it costs 10 bucks a month to run, updates within
| 2 minutes of a downtime.
|
| Github has no excuse.
|
| We can and will hold them accountable for their decision to
| hide information from their paying customers and any other
| case they prioritize their ego over their users.
| Dylan16807 wrote:
| Mockery, not vitriol. Nobody asked them to make it manual.
| MrStonedOne wrote:
| Are you paying pupdogg? Because you have no right to demand
| somebody be perfect and professional about something you aren't
| paying them to do.
| [deleted]
| midasuni wrote:
| They should run it in the cloud /s
|
| (My self hosted gitlab is working fine)
| Longwelwind wrote:
| I would wager that your self-hosted GitLab has less uptime than
| GitHub (or the quality of the deployment). We typically
| underestimate the downtime of the software we deploy ourselves,
| probably because we're attached to it, or because we're busy
| fixing it, instead of having to wait.
|
| I'd take the occasional downtimes of cloud solutions, if it
| means not having to use some of the self-hosted software I had
| to to use at work.
| raro11 wrote:
| Nope. Been self-hosting for four years without any issues.
|
| We only do security upgrades or wait for x.1 upgrades. Never
| x.0
|
| The hosted .com however I wouldn't recommend.
| js4ever wrote:
| I'm managing a Gitlab instance for 10k users since 3 years,
| we had zero unplanned downtime in the last 3 years, I just
| had to apply the upgrade every 3 months and it's running
| smoothly ever since
| belltaco wrote:
| I'm sure the remote masters are keeping the uptime high.
| https://therecord.media/gitlab-servers-are-being-exploited-i...
| gime_tree_fiddy wrote:
| This seems to be a big one. Even the pulls are failing. I just
| discovered there isn't not Github mirror for OpenCV code.
| lordnacho wrote:
| It's not that hard to push to multiple remotes. Perhaps more
| projects ought to have multiple GitHub/Bitbucket/GitLab/etc
| repos. Then it wouldn't matter if one of them went down now and
| again.
| axiom92 wrote:
| Hmm explains why the following line is suddenly causing my jobs
| to crash:
|
| > nltk.download("punkt")
|
| Apparently NLTK uses Github for hosting their sentence
| tokenization models.
| egberts1 wrote:
| That's why I have a THREE-tier GIT repository.
|
| - Github
|
| - My Git server
|
| - My local git
|
| No need to worry about failures. :-)
| seirl wrote:
| If you urgently need to retrieve a piece of software, it's likely
| archived in Software Heritage:
| http://archive.softwareheritage.org/
| davegauer wrote:
| This is fantastic! It also has some projects I thought had been
| lost to the mists of time when Bitbucket stopped hosting
| Mercurial repos.
| withinboredom wrote:
| I ended up here because automated cluster deployments failed
| trying to download releases from GitHub... I wonder if that
| software is served there and I can update the URLs before
| GitHub fixes their issues :)
| eugenhotaj wrote:
| Let me guess, someone pushed a bad config.
| xfbs wrote:
| It's amazing how much stuff breaks when GitHub goes down. I'm
| doing some Rust coding right now, the rust-s3 crate tells me that
| to look up what feature I need to enable (tokio + rustls), I need
| to look into the Cargo.toml. Well, the repo won't load, and nor
| can I clone it. Well okay, fuck that I can use the default
| dependencies. But no wait I can't, I can't even do a cargo build
| because cargo uses a github repository as the source of truth for
| all crates. No more Rust for me today :(
| kleiba wrote:
| _It 's amazing how much stuff breaks when GitHub goes down._
|
| That's right. Someone should really come up with a
| decentralized VCS. /scnr
| catskul2 wrote:
| IMO Git is decentralization-ready but the rest of the
| infrastructure necessary to make it practical is not widely
| available/in use. The necessary peer-to-peer and networks of
| trust are still not a solved problem, or if they are they've
| for some reason are not popular enough to be in wide use.
| PaulDavisThe1st wrote:
| It's widely available and widely used, just not as widely
| used as github.
|
| Setting up your own git server is not hard, but it's not as
| easy as just getting github or gitlab to run it for you.
| Way too many people take the easier path, even though the
| harder path is not actually that hard.
|
| There are also multiple solutions.
| lostmsu wrote:
| You probably meant distributed decentralized VCS.
| birktj wrote:
| You can view the source of a crate on docs.rs (see [1] for the
| Cargo.toml of rust-s3). Also I am pretty sure cargo only
| depends on GitHub for authentication for uploading crates and
| not for the actual contents. Trying to build an empty crate
| with rust-s3 as a dependency right now seems to work fine.
|
| [1]: https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml
| ATsch wrote:
| As I understand it, the crates themselves are not stored on
| github, but the crate index is, as it uses a git repo to get
| "free" delta compression and auditing.
| anothernewdude wrote:
| I can't stand builds that just reach out to the internet for
| things.
| lanstin wrote:
| Yeah seems like basic hygiene especially given the supply
| chain attacks but also a lost cause. No one has the skills to
| do builds without the internet. Even forcing teams to use an
| allow list for the internet involving fighting a lot of angry
| people.
| contingencies wrote:
| One of the few decent uses for containers is to enforce
| proxied internet so build process artifacts can be auto-
| stored for subsequent builds.
|
| For the worst offender I am aware of, try building a
| _flutter_ project... it silent internet gets artifacts from
| _at least three_ different packaging systems (node,
| cocoapods, android packages), all of which have caused
| hard-to-debug failures.
| etra0 wrote:
| Beside the `--offline`, you can also use `--vendor` to include
| all the dependencies in a folder to be committed alongside your
| project. Useful when you don't want to rely on external fetch
| every time!
| jiggawatts wrote:
| Both should be the default.
|
| Rust has a robust memory model, but everything else about it
| insists on copying the fragility of the NPM ecosystem.
|
| The recent hoopla around a bunch of Rust mods quitting
| revealed that my suspicions are precisely true -- key Rust
| staff also sit on the NPM board!
| dom96 wrote:
| Should be fairly easy for Cargo to solve this: why doesn't it
| already mirror its source of truth to GitLab and other Git
| hosters?
| revskill wrote:
| I guess same story for Go!
| krasin wrote:
| In Go, it's customary to use 'go mod vendor' to put your
| dependencies into the main repository. While it's not
| universally recognized as a good technique, it saves the
| adopters of this approach from the downtime today.
| jen20 wrote:
| I'm not sure this is customary at all - I rarely encounter
| a vendor directory anymore.
|
| Note you also need to build differently if you go this
| route: `-mod=vendor`, otherwise the directory will be
| ignored in modern Go.
| krasin wrote:
| With more outages of our amazing, but overly centralized
| development ecosystem, the popularity of this approach
| will likely surge. It helps that Go supports the
| vendoring workflow as it makes the choice practical.
|
| As for building with a flag: true, but very minor, as
| it's rare to execute 'go build' directly. In most
| projects I've seen, it's either Bazel, or some kind of
| "build.sh".
| travisd wrote:
| I've never had to use -mod=vendor, so I just looked it
| up: -mod mode
| module download mode to use: readonly, vendor, or mod.
| By default, if a vendor directory is present and the go
| version in go.mod is 1.14 or higher,
| the go command acts as if -mod=vendor were set.
| Otherwise, the go command acts as if -mod=readonly were
| set. See
| https://golang.org/ref/mod#build-commands for details.
|
| If there's a vendor directory, it's used by default. As
| for my two cents, I use it frequently when building
| Docker images so that I don't have to pass secrets into
| the image to clone private modules (but I don't check the
| vendor directory into Git).
| shoo wrote:
| Maybe the key point is to choose consciously and pick the
| option that gives the best combinations of tradeoffs for
| your situation vs just doing what is easy or copying what
| other people are doing without understanding you're
| making a decision with various tradeoffs and
| consequences. Tradeoffs that are a good fit in other
| contexts may be a poor fit for your situation.
|
| If one of the goals of your build process is to be able
| to guarantee reproducible builds for software you've
| shipped to customers, and you depend on open source
| libraries from third parties you don't control, hosted on
| external services you don't control, then you probably
| need your own copies of those dependencies. Maybe
| vendored into version control, maybe sitting in your
| server of mirrored dependencies which you back up in case
| the upstream project permanently vanishes from the
| internet. But setting up and maintaining it takes time
| and effort and maybe it's not worth paying that cost if
| the context where your software is used doesn't value
| reproducible builds.
| avl999 wrote:
| Google takes care of storing copies of any go dependency
| you use on their proxy, there is very little reason for
| you to maintain your own via vendoring. Maybe if you are
| a big enough organization you run your own proxy as an
| extra layer of safety above google but still I don't see
| the value of vendoring these days.
| avl999 wrote:
| These days with go mod and the go team maintaining their
| proxy there is very little benefit to vendoring and any
| benefit is not worth blowingup up the size of your repos
| and messing up code reviews on PRs that introduce new
| dependencies.
| jlouis wrote:
| Go uses a proxy for downloading modules, so there's no Github
| involved. And you could run your own proxy-cache if you
| wanted. In addition, your work machine has a local proxy
| cache of the modules you already downloaded.
|
| Go doesn't use a repository as a single source either, which
| is another problem in of itself.
| ggktk wrote:
| I think Go dependencies should still work, thanks to Google's
| module mirror[0] (enabled by default), which has cache.
|
| [0]: https://proxy.golang.org/
| fowlie wrote:
| I was working on my Nix config. I had just added a small
| command line utility and wanted to install it, but then got 504
| errors from github.com. Annoying!
| dandotway wrote:
| In the 1970s-80s we managed with email lists and tarballs on
| multiple mirrors. Git itself decentralizes source control, and
| yet we all want to use single-point-of-failure Github.
|
| Anyone remember when Github took down youtube-dl?
|
| I wonder how much Big Brother data Microsoft is gathering on
| Github developers. "Oh, as part of our hiring process, just
| like we scan your FB and Twitter, we also scan your Github
| activity to evaluate your performance as best as our machine
| learning overlords can assess it. Do you create Github projects
| and then abandon them when unfixed issues accumulate? Our
| algorithm thinks you're a bad hire."
| adeelk93 wrote:
| > Anyone remember when Github took down youtube-dl?
|
| I would blame the law (DMCA), not those forced to abide by
| the law (Github)
| roastedpeacock wrote:
| It was an unfortunate situation. I don't entirely know how
| GitHub handled things before the Microsoft acquisition but
| it also wouldn't surprise me if Microsofts legal department
| is more risk adverse compared to GitHub internal.
|
| > I would blame the law (DMCA), not those forced to abide
| by the law (Github)
|
| It seems even in the context of US-based companies that
| some companies are a little more "trigger-happy" with DMCA
| and other claims compared to others.
| skeeter2020 wrote:
| Especially because they worked to try and make sure this
| wouldn't happen again.
| [deleted]
| kevin_thibedeau wrote:
| The law also permits content to be restored immediately
| upon receipt of a counternotice. Somehow the data silos
| never get around to supporting that.
| anchpop wrote:
| twitch actually does do that IIRC, not sure about others
| zinekeller wrote:
| GitHub can't, until at least the the public support is in
| their favour, because legally it's a hot water. This is
| _not_ your ordinary copyright DMCA Complaint, this is the
| section 1201 concerning circumvention, which is held to a
| different standard. This is essentially RIAA telling
| GitHub "if you're not stopping this, we'll see _you_
| (not the YT-DL developers) in court ".
| roastedpeacock wrote:
| > GitHub can't, until at least the the public support is
| in their favour
|
| Public support on which grounds?
| dandotway wrote:
| Laws need teeth to be enforced. It used to be illegal to
| harbor runaway slaves, or to be a Catholic priest in
| Protestant England. So people built cleverly hidden Priest
| Holes[1] in houses to hide. But you can't really build the
| equivalent of a Priest Hole within Github, because Github
| is not your house, but rather your lord's castle.
|
| [1] https://en.wikipedia.org/wiki/Priest_hole
| Teever wrote:
| Your argument is pure sophistry.
|
| Microsoft is a member of the RIAA and works closely with
| the MPAA to further control over society with DRM and
| intellectual property restrictions like the DMCA.
|
| It is well within the power of an entity the size of
| Microsoft to oppose something like the DMCA and the fact
| that they don't indicates not that they can't as you imply,
| but that they don't want to.
| surfer7837 wrote:
| Don't create a Github account with your name then?
| asxd wrote:
| Yeah seems like this would be a simple workaround if
| companies did start adopting this practice
| tester756 wrote:
| but that data is avaliable to everyone cuz it's mostly
| public?
| pasabagi wrote:
| Cargo has an --offline option. It's actually pretty possible to
| use rust totally offline - the doccumentation can be built, for
| instance, then locally served with (iirc) cargo doc.
| jiggawatts wrote:
| Offline should be the default, not an option.
|
| Otherwise nobody will notice how fragile their workflow is
| until it is _too late to fix it!_
| xfbs wrote:
| I build and host documentation on every commit anyways in the
| CI. And yes, that is true, I eventually figured it out (had
| some issues with it at first) but it seems like GitHub is
| back up so all's well anyhow. I do however wish that there
| was some public mirror that cargo could fall back to,
| wouldn't that make a lot of sense?
| cube00 wrote:
| Watermelon status.
| donutloop wrote:
| Dam!
| activitypea wrote:
| Started learning Emacs tonight, and now I can't access Helm's
| documentation :\
| brabel wrote:
| Good time to swith to the lighter (and cooler) Ivy [1]. This
| site is up, but the actual Ivy Docs Page [2] is also returning
| a 500!
|
| [1] https://writequit.org/denver-
| emacs/presentations/2017-04-11-...
|
| [2] https://oremacs.com/swiper/
| activitypea wrote:
| Thanks for the suggestion, but for now I can only bookmark it
| for later. My priority right now is to reach basic competency
| ASAP, so I'm running Spacemacs. I love the "batteries
| included" thing, but just installing it took like an hour of
| fiddling before I just gave up and ingored the error message.
| I don't think it'd be wise to fiddle around with the settings
| if even the defaults are going haywire.
| thumbellina wrote:
| Helm might have info documentation, available offline within
| Emacs.
|
| Try `M-x info-apropos` and type "helm".
|
| Info docs form a tree: press '^' to go up a node, 'n' and 'p'
| to go forward and back on same level, '[' and ']' to go
| forward/back a page like a book.
| qwertox wrote:
| I just had a very odd thing happen to me on GitHub.
|
| I accidentally closed my browser so I reopened it with Undo Tab
| Close, and GitHub's tab title was labeled " _Your account
| recovery is unable to load_ " for a very brief moment. Then the
| GitHub error site with the pink unicorn loaded. The URL which was
| supposed to load was https://github.com/hlissner/doom-emacs which
| I had tried to load about 15 minutes earlier or so, but which
| would not load because the site is down.
| sodality2 wrote:
| It sounds like some HTML was cached by your browser and it then
| tried to load some info, which of course results in a 5xx
| error. Then the existing HTML realized that and showed the
| error.
| qwertox wrote:
| Why would something related to _account recovery_ be cached
| in such a way that it is the first thing which gets rendered?
| I really doubt that it is related to caching. Also, that tab
| was idle for a couple of minutes while I was trying to load
| GitHub in another tab multiple times before I accidentally
| closed the browser (just that window, I had other windows of
| the same browser instance open).
| sodality2 wrote:
| Just the HTML of the existing page could have been cached,
| and it may check in the background to reload account info.
| Then, if this can't be loaded, it shows the error.
| carbocation wrote:
| And no updates from https://twitter.com/githubstatus
| yellow_lead wrote:
| https://twitter.com/githubstatus/status/1464696434645741574
|
| Got one after you posted at least
| nickradford wrote:
| I mean... it _literally_ just went down.
| carbocation wrote:
| Yes, if it's a manually run account, then the fact that it
| just went down a few minutes ago would be a good reason for
| it not to be updated.
| yellow_lead wrote:
| It's been like 30 mins at least...
| nickradford wrote:
| My coworker and I just reviewed and merged a PR, at 12:38
| Pacific
| milankragujevic wrote:
| I rebooted my modem two times before I realized (with VPN) that
| Github truly is down.
|
| And I need it right now.
|
| Guess I'll be setting up a local git instance very soon...
|
| edit: by local I meant in the intranet. not on my local machine.
| maximilianroos wrote:
| > Guess I'll be setting up a local git instance very soon...
|
| The incorrectness of this highlights how useful DVCS can be --
| a git server going down doesn't affect working locally at all.
| markozivanovic wrote:
| What if he needs to pull some changes he needs to continue
| with his work?
| Jweb_Guru wrote:
| You can merge from anywhere, not just Github master. It's a
| little more work, but a coworker could share their
| repository without much difficulty. You can even apply
| patches sent over email, if you're so inclined.
| markozivanovic wrote:
| All legitimate solutions, I agree. The thing is, it
| really sucks when this happens over the weekend. Alerting
| a colleague in their off-hours to be able to continue
| with your work is not ideal. Of course, it happens
| rarely, so you're making a good point. If it happened
| weekly, that would be another story. :)
| julianlam wrote:
| Then a local git instance would preclude him from needing
| or receiving those changes anyway.
| markozivanovic wrote:
| Hmm, I think we understood 'local' as two different
| things. I understood it as a self hosted git server, like
| on premises in parent's company that he and his
| colleagues would use. Now that you mentioned it, I'm
| starting to think I understood it wrong.
| milankragujevic wrote:
| Nope, I may have misspoke, but I meant self-hosted git
| instance in the intranet.
| backoncemore wrote:
| Could have just connected your phone to data and tried it out.
| milankragujevic wrote:
| A VPN is a button click away, and modem reboot is two clicks
| :) For me personally more convenient than tethering.
| yellow_lead wrote:
| https://downdetector.com/status/github/
|
| Yep, can't even push
| pupdogg wrote:
| For record keeping: https://imgur.com/a/b6JIQT3
| svnpenn wrote:
| https://i.imgur.com/SJDJRyo.jpeg
| breakingcups wrote:
| Showing (nearly) all-red for me, 18 minutes later:
| https://imgur.com/a/plZbFky
| svnpenn wrote:
| https://i.imgur.com/3TyNW9A.png
| jmeyer2k wrote:
| Thought I might be the only one seeing this (account-related or
| something), but nope, just Github's misleading status page...
___________________________________________________________________
(page generated 2021-11-27 23:00 UTC)