[HN Gopher] HN Is Up Again
___________________________________________________________________
HN Is Up Again
Author : tpmx
Score : 732 points
Date : 2022-07-08 20:30 UTC (2 hours ago)
| cryptoz wrote:
| Obligatory "can't wait for postmortem" comment
| lnalx wrote:
| I finally started to have a social life... it didn't last.
| joshstrange wrote:
| I don't think I had ever fully internalized how often I open this
| site throughout the day. Finish a task? HN. Got frustrated/stuck
| on a problem? HN break. Waiting for something to
| install/upload/compile/etc? HN.
|
| Needless to say I opened a new tab, typed "n", and hit enter
| countless times today before my brain caught up with my muscle
| memory.
| m4tthumphrey wrote:
| I realised this today too. I imagine pretty much all of us are
| the same!
| beckingz wrote:
| Makes you appreciate the noprocrast mode feature.
| m4tthumphrey wrote:
| Tried it once, never again!
| divbzero wrote:
| I set the delay to 1 minute: Short enough that I can wait
| if I _really_ need to read a thread, but long enough to
| nudge me back to my primary task if I'm just browsing.
| function_seven wrote:
| I've been meaning to try that feature. Will probably check
| it out next week.
| Dnguyen wrote:
| I see what you did there! haha
| beckingz wrote:
| Yeah the delirium tremens are rough.
| [deleted]
| BbzzbB wrote:
| CTRL-SHIFT-n (or -p) -> n -> ENTER
| grepfru_it wrote:
| Ah the elusive PorN method
| omoikane wrote:
| Doesn't help users who are usually not logged in.
|
| ... I might have been more productive than usual today.
| Overtonwindow wrote:
| HN is where I get the bulk of my news and information about
| current thinking. I visit maybe several dozen times a day.
| robrorcroptrer wrote:
| Amen.
| smcl wrote:
| This happened last couple of times I switched laptops - my old
| habit to visit "guardian.co.uk" by typing "guar" and hitting
| enter no longer works because I've now accidentally searched
| too many times for "guar" :D
| johnnymellor wrote:
| You can make the omnibox forget about URLs and search terms
| you've used a lot by selecting them with the down key then
| pressing Shift+Delete (https://superuser.com/a/189334).
| sokoloff wrote:
| Note that that doesn't seem to work if you have a bookmark
| with that content (as it seems to find the bookmark, which
| is reasonable behavior but caught me out when I was trying
| to change the URL to an internal tool and didn't realize
| why it wasn't working to delete the auto-complete).
| GekkePrutser wrote:
| Nice, doesn't seem to work on firefox though, guess it's
| chrome only?
| bragr wrote:
| noprocrast exists for a reason :)
| slategruen wrote:
| Does it work even when you're logged out?
| timeon wrote:
| Even if it did. Second browser will help with the urge. But
| it can help anyway. Some people just need the reminder.
| teeceetime2 wrote:
| went out of my way to leave lurk mode and log in just so I
| could say "100% same experience for me"
| cheesewhizemacs wrote:
| > Needless to say I opened a new tab, typed "n", and hit enter
| countless times today before my brain caught up with my muscle
| memory.
|
| I do this too, and it's because this site is an addictive slot
| machine just like every other social networking site. I
| actually really hate this website, but I'm here almost every
| day, because I can't seem to break the habit. Neat. It's
| probably because I have a common impulse control / executive
| functioning disorder, and the way the front page works exploits
| some bug in my brain.
|
| Reddit does this to me too. I also hate Reddit.
| autoexec wrote:
| > I actually really hate this website
|
| Why hate this site? Because it contains interesting/useful
| content often enough to make you come back? That'd be a weird
| reason to hate the site. I too have a common impulse
| control/executive functioning disorder, but I don't hate the
| things that it makes me vulnerable to. If I were feeling
| resentful, I'd have to put the blame on my condition.
|
| I don't have to ask why you hate reddit, the valid reasons
| for hating reddit are myriad
| bbkane wrote:
| Others have mentioned browser add-ons / DNS providers who can
| limit/blacklist sites. Maybe try one of those? The thing
| that's worked best for me though is leaving my phone in
| another room for a while or taking a walk without it.
| msrenee wrote:
| If it helps, I wouldn't say it's a disorder since it appears
| that basically everyone has a habit like this. It's probably
| a byproduct of some kind of adaptive advantage, but I don't
| have it in me to speculate exactly what at the moment. The
| only variable is what exactly you do automatically. Nowadays,
| everyone has their app or web page. Before smartphones and
| the internet being available everywhere, I remember my mentor
| talking about quitting cigarettes. This was shortly after the
| non-smoking section of the restaurant became the whole
| restaurant. She said that part of why it was so hard to quit
| was that even when she meant to cut back, she'd still find
| herself a third of the way through a cigarette before she
| realized that she'd lit one. I tear at the skin next to my
| fingernails in addition to opening HN (which was what I
| switched to when it became painfully obvious that Reddit was
| both bad for me and run by bad people). I moved my ebook app
| to the first screen on my phone and moved this app to a spot
| where I wasn't used to finding it. I figured it might get me
| to read more. What actually happened is that I started
| absent-mindedly swiping to the second screen and opening up
| the app.
|
| It's a pretty universal issue. Companies are just getting
| better at using it to their advantage.
| xoa wrote:
| I reference back to it for a lot of info too, which I guess I
| should probably load more of into my own notes database. But
| still today there were a bunch of saved comments I wanted to
| re-read as reference multiple times, definitely noticeable to
| miss it. Or alternatively if I'd grabbed the URLs for
| everything I'm assuming the wayback machine probably archives
| this pretty well. Perils of depending on the HNcloud service
| :).
| christophilus wrote:
| Yep. Same. It says a lot about the quality of HN, I think.
| Also, I can't remember the last time it was down. For a while,
| I thought there _must_ be something wrong with my internet
| connection or DNS config or something.
| jwdunne wrote:
| Me too! I was, shamefully, in the middle of work so had a
| mini panic thinking my 5 min HN scroll was gonna become an
| hour long battle with my connection!
| ijidak wrote:
| Haha. A building of ours in Canada lost internet at the same
| time.
|
| When I saw hn was down, I double-checked the news to see if a
| major part of the internet had gone down.
| butterNaN wrote:
| Habitual usage doesn't necessarily correlate to quality, I'd
| say. People who use Facebook/Twitter also have this sort of
| muscle reflex developed over time.
|
| That said, HN does have quality content and the signal/noise
| is way better than sites designed specifically to keep you
| addicted.
| autoexec wrote:
| It might not correlate to quality, but if the information
| found at a website wasn't valued we wouldn't be constantly
| pulling the site up, just like facebook users do.
|
| I'd argue that this site has a good signal/noise ratio by
| design and specifically to keep you addicted (where
| "addicted" means using and constantly returning to the
| site). This site is just designed to attract people who are
| put off by the kinds of tricks employed elsewhere
| devonbleak wrote:
| Might have tried to curl it from a production system just to
| make sure it wasn't my internet.
| hinkley wrote:
| I understood this reference.
| 867-5309 wrote:
| I didn't wget it
| joncp wrote:
| Indeed. I find that it's so reliable and fast that I use it
| to check my internet connection. If I can't hit HN, then
| something's wrong on my end.
| andrepd wrote:
| I use example.com for that purpose :p
| autoexec wrote:
| Wow, it works, but it really seems like it shouldn't. I'd
| expected reserved domain names to not resolve at all let
| alone be pingable and point at a working webserver. Has
| it always had a website?
| 40four wrote:
| I assumed at first there was a problem with my mobile data
| connection, before I realized it was actually HN :)
| huevosabio wrote:
| Same here. I thought the my internet was down or I was too
| far from the router. HN being down was my last thought.
| LoveMortuus wrote:
| Ohh... I thought my phone was just being weird because I use an
| AdBlock that works through VPN and it sometimes breaks stuff...
| chrisshroba wrote:
| I legitimately thought my internet was down for thirty minutes
| until I decided to try google.
| the_af wrote:
| Same. HN is how I check my internet connection is working.
| Took me a while to realize the problem wasn't my
| connection...
| Mixtape wrote:
| I was in almost the exact same position all day. What made it
| worse though was the fact that this happened right in the
| middle of my attempts at curbing my browsing habits. Once my
| app timers for Reddit is Fun, Instagram, and Twitter were up,
| it was time for HN... except there was no HN. What that meant
| is that I was reaching for a stimulus and then not getting it,
| the same way that an alcoholic wouldn't feel satisfied by, say,
| a can of soda. It was weird to experience, but very
| enlightening. It both made me realize how subconsciously my
| addiction is reinforced and reaffirmed to me that it is, in
| fact, an addiction. I'm not going to stop using HN of course,
| but I'm definitely going to be more aware of _how_ I use it
| (e.g. passively vs. intentionally) from now on.
| selimnairb wrote:
| I picked a bad day to stop using Twitter.
| GekkePrutser wrote:
| As far as addictions go, I find HN actually one that delivers
| actual knowledge. Literally every day I read something I
| didn't know before. Unlike on Facebook that just tries to
| serve me with more of the stuff I have already seen.
|
| I like wasting time on HN because it's time not actually
| wasted :)
|
| And don't get me started on Twitter... Sure there are some
| gems on twitter but I have to wade through 1000s of tweets of
| pure nonsense to see them. No thanks. If it's something
| really great someone will post a link on HN anyway :)
| [deleted]
| [deleted]
| lemoncookiechip wrote:
| Same experience through out the day.
| akomtu wrote:
| Mr. dang, can I use the opportunity to suggest to turn the tiny
| upvote and downvote arrows into links, separated by sufficient
| distance? My fingertip is 15x larger than these arrows and it
| takes quite a bit of precision to hit the right one. I bet, half
| of upvotes and downvotes are erroneous for this reason.
| haswell wrote:
| This downtime made me realize (again) how much I appreciate the
| kind of interesting topics that show up here, the depth of
| discussion, and a general attitude of good faith that (most)
| engage with here.
|
| I realized how little of this I find elsewhere in my life -
| whether through Reddit or even my IRL friend circles.
|
| This realization saddens me - I feel like I shouldn't have to
| rely on HN so much to scratch this particular itch.
|
| Perhaps I need to get out more.
| hluska wrote:
| The beautiful part of the internet is that it provides space
| for people to share incredibly niche interests. For all of its
| problems and complications, that beauty still exists.
| telesilla wrote:
| Enjoy it while it's here. When it's gone or you have to move
| on, we'll miss you too and that's the meaningful part of life.
| rco8786 wrote:
| > This realization saddens me - I feel like I shouldn't have to
| rely on HN so much to scratch this particular itch.
|
| > Perhaps I need to get out more.
|
| Another way to look at it is that you have a particular set of
| interests and HN is the online outlet that serves those
| interests. There's nothing wrong with that, at all and you
| don't need to have multiple sources for it. No different than
| someone who likes to ride bikes owning one bike, or someone who
| likes to read going to the same local library every week for 10
| years.
| avalys wrote:
| It is very different from your examples. Even if you only own
| one bike, there are innumerable others in existence and
| companies making new ones every day, if yours is destroyed or
| lost. Similarly, there are plenty of local libraries to
| choose from, even if your favorite one closes.
|
| Whereas, if HN closes, there is no equivalent replacement
| available.
| rco8786 wrote:
| What I am saying is that you don't need to worry about any
| of that. Sure, if HN permanently shutters - you'll need to
| go find a replacement. But HN isn't going anywhere, as far
| as I am aware. You don't need redundancy for your online
| community/content consumption
| adamredwoods wrote:
| Quality, non-biased news sources are (surprisingly) difficult
| to find.
| TameAntelope wrote:
| Danger: that is decidedly not what HN is, at all.
| baby wrote:
| I've been wanting to create something similar but for
| cryptocurrencies. A place with no scam/bullshit posts and only
| deeply technical discussions about the latest trends in zero-
| knowledge proofs, consensus protocols, scalability challenges,
| etc.
|
| But I'm too lazy to write the application. I wish there was
| some SDK I could spin up, like PHPBB back in the days, to have
| something exactly like HN.
| alberth wrote:
| I'd love to learn more about HN hardware & software stack.
|
| I know old posts indicate it's running on a low core count but
| high frequency Intel CPUs on FreeBSD and no database (just flat
| files).
|
| I wonder if it's still the same.
| oblio wrote:
| Ah, that's easy.
|
| HN is running on an old laptop from Viaweb.
|
| Arc is running under the pg user and it's used as the process
| supervisor.
|
| The actual web server is a VB app running on Linux through
| Wine.
|
| The flat files have been migrated to an MS Access DB, also
| running through Wine.
| fartcannon wrote:
| My employer is taking notes.
| [deleted]
| TIPSIO wrote:
| Went to Reddit instead today more than I should (not proud).
|
| Anyone been on Slashdot lately? Checked it out too was really
| nice.
| doublerabbit wrote:
| Slashdot is a slurry of Reddit and HN. My goto when bored with
| reddit and hn...
| cpdean wrote:
| Its weird how I saw this headline, thought to myself "Oh good!
| It's back!" before realizing that I was using HN to see this very
| headline...
| ddingus wrote:
| Whew!
|
| Thanks Dang and company.
|
| I appreciate you all.
|
| @my HN peers:
|
| Have a great weekend and thank you all for being you. I learn a
| ton here and enjoy the perspectives often found on these pages.
| It is all high value.
| anttiharju wrote:
| I've been so used to using hn as my "is internet working" page
| that for a while I thought my internet was just down and tried to
| troubleshoot it haha.
| afrcnc wrote:
| I'm sorry... I tripped.
| bob1029 wrote:
| Even accounting for this outage, most other SAAS platforms still
| can't compete with HN's non-existent SLA.
|
| Thank you to everyone who keeps this thing running.
| vntok wrote:
| > Even accounting for this outage, most other SAAS platforms
| still can't compete with HN's non-existent SLA.
|
| 8 hours of downtime in a given year is 99.9%, so only three
| nines. The major SaaS platforms all are basically at least as
| resilient as this, and most have more stringent SLAs.
| thamer wrote:
| The last post[1] before this one was posted at 12:45:10 UTC. This
| current post was made at 20:30:55 UTC, so that's a gap of 7 hours
| 45 minutes and 45 seconds.
|
| [1] https://news.ycombinator.com/item?id=32026548
| TimWolla wrote:
| There's this comment which is newer:
| https://news.ycombinator.com/item?id=32026565. There's also
| comments on some test submission:
| https://news.ycombinator.com/item?id=32026568 that itself got
| deleted: https://news.ycombinator.com/item?id=32026567
| [deleted]
| rvz wrote:
| No news reports, press coverage or the majority of world caring
| about this 'site' going down for longer than usual.
|
| Does that mean nothing of value was lost?
| swat535 wrote:
| Great! Just in time for the Friday evening to kick in as well.
|
| I was worried that I may actually have to go out and do things
| instead of lurking here this weekend..
| damsta wrote:
| It was really funny to read advices to move HN to cloud from all
| those "experts".
| edgartaor wrote:
| HN is so reliable that I distrusted my wife first.
|
| "Did you unplug the router?"
| jandrusk wrote:
| Had to take a bunch of Xanex to get through the data without HN.
| droptablemain wrote:
| Hypothesis: a link from HN hit the front-page of HN and inception
| ensued. In short, HN got HN'd.
| booleandilemma wrote:
| This was a rough one. I actually opened slashdot at one point.
| ddingus wrote:
| Same
|
| Early on /. was amazing! Remember Cmdrtaco it all out, often
| taking us for the ride?
|
| Good times, frequently good discussion.
|
| HN has been better for years now, was better at inception, for
| the most part.
|
| /. has improved a bit. Good to see, or I caught it on a good
| day.
| ilikeitdark wrote:
| I've been noticing internet problems here and there all week, and
| was getting a little sketched out (is the heat waves, Russia, or
| alien attack:) so this really got me worried.
| ahepp wrote:
| I read an article recently on avoiding fallback in distributed
| systems.[0]
|
| Is it more appropriate to call the strategy in this case
| fallback, or failover? Since the secondary server wasn't running
| in production until the first one failed, it sounds like
| fallback?
|
| Perhaps higher reliability strategies would have been instead of
| having a secondary server, just have more mirrored disks on the
| main server, to reduce the likelihood of the array being
| compromised?
|
| Alternatively, to run both the primary and secondary servers in
| production all the time. But that would presumably merely move
| the single point of failure to the proxy?
|
| [0] https://aws.amazon.com/builders-library/avoiding-fallback-
| in...
| I_complete_me wrote:
| I really enjoyed the outage.
|
| Disclaimer: I did not cause it.
| CSMastermind wrote:
| A huge thank you to everyone who keeps this amazing site running
| :)
| humanwhosits wrote:
| Spent so long debugging my home network..
|
| Couldn't possibly have been HN that was the problem haha
| pbhjpbhj wrote:
| Same, I ended up updating my pihole which was long overdue.
| Just finished, loaded up HN and it worked - "huh, wonder what
| the problem was with my pihole" I thought ... well it needed
| doing anyway.
| D-Coder wrote:
| All sites broke on my machine this morning.
|
| I reset the router... and HN was _still_ down.
|
| <sniff>
| tekacs wrote:
| I always use https://isup.me
| (https://downforeveryoneorjustme.com/) in these situations, to
| find out if I'm losing my mind. :)
| sam0x17 wrote:
| > HN Is Up Again
|
| I'm not sure I believe you what is your source for this ;)
| ru552 wrote:
| grats on the reload
| onenukecourse wrote:
| I'm traveling, so I just assumed my internet had died instead and
| didn't refresh all day. I literally just found out HN was down by
| this post.
| nyadesu wrote:
| Just installed a new router and was trying to browse HN to check
| if my internet connection was working properly, lol
| anewpersonality wrote:
| This downtime is a wakeup call for many
| m0llusk wrote:
| a call to go back to sleep for others
| qwertox wrote:
| What a ride. HN should shut down once a month on purpose on a
| random working day just to allow us to recalibrate our inner
| compass a bit.
| srj wrote:
| My first thought was that my own Internet was down. My second was
| that HN was somehow dependent on Rogers. Hard to go through a
| Friday afternoon without HN!
| beckingz wrote:
| Previous post on the HN status twitter account is from June 2021.
|
| Over a year with no issues. Impressive.
|
| https://twitter.com/HNStatus
| jve wrote:
| Twitter doesn't bother with minor issues, but there were some
| for sure.
|
| This logs lesser ones: https://hn.hund.io/
| coding123 wrote:
| Damn, and I got a LOT done today.
| johnsutor wrote:
| Worst couple of hours of my life.
| ithinkso wrote:
| I've realized that I check if my internet works by opening HN...
| robotsquidward wrote:
| HN is my default 'is the internet working' site and that
| genuinely threw me for a loop across multiple devices today
| dealing with hotspots while our power was out.
| jeffbee wrote:
| I wonder if things like double disk failures and restores from
| backup make HNers of the "bare metal 4ever" tribe revisit their
| cloud hatred.
| wumpus wrote:
| No.
| onion2k wrote:
| I did some work. Please don't let this happen again.
| divbzero wrote:
| Thank you HN admins for bringing the site back online with
| everything restored from backup. Thank you also for the 99.99% of
| the time that HN just runs and runs without issue.
| O__________O wrote:
| Thank you Dang!!
|
| ____________
|
| Related:
|
| https://www.newyorker.com/news/letter-from-silicon-valley/th...
|
| https://news.ycombinator.com/user?id=dang
| jacquesm wrote:
| And all those poor project managers at the end of 2022 wondering
| what on earth they did right on the 8th of July that caused
| productivity to reach previously unthinkable heights.
| Johnny555 wrote:
| Will you be posting a postmortem?
|
| Not that I deserve or expect one from a free service, but because
| I enjoy reading postmortems from failures where both the primary
| and backup systems failed, I like to see what holes I might have
| in my own failover setup.
| abarrak wrote:
| For me, HN is one of the default pages when browser's open.
| olingern wrote:
| While the naysayers will say, "Why isn't this in the cloud?," I
| think the response times and uptime of hackernews is really
| impressive. If anyone has a write-up of the infrastructure that
| runs HN, I would be interested. Maybe startups really can be run
| off of a rasberry pi
| tpmx wrote:
| It seems like it is in the cloud (AWS) now. See
| https://news.ycombinator.com/item?id=32027091.
| NKosmatos wrote:
| If I was (re)designing this, I would keep the existing bare
| metal server but I would also put in place double (or triple)
| cloud redundancy/failover. We all love HN so much that it
| should have zero downtime :-)
| pyb wrote:
| AWS has had more outages than HN in recent times
| [deleted]
| mpyne wrote:
| AWS also operates at a significantly larger scale. When was
| the last AWS outage due to two critical disks failing at the
| same time?
| hosteur wrote:
| Failures due to increasing complexity are still failures.
| mpyne wrote:
| Sure, but you're talking about all of AWS as if every
| customer is impacted when any part of AWS suffers a
| failure. But that's not the case, which makes it quite an
| apples/oranges comparison.
|
| But even comparing the apples to the oranges, this HN
| status page someone else pointed out https://hn.hund.io/
| seems to show that HN has had more than one outage in
| just the past month. All but today's and last night's
| being quite short, but still. Sometimes you need some
| extra complexity if you want to make it to zero downtime
| overall.
|
| That's not something the HN website needs but I think AWS
| is doing fine even if that's your point of comparison.
| fartcannon wrote:
| Agree. Same with places like github.
| lkxijlewlf wrote:
| Feels snappier. :)
| zacharycohn wrote:
| I have a flakey internet connection. Because HN loads so fast, I
| use it to test if my internet is working/is back up after it
| disconnects.
|
| WELL TODAY WAS VERY INCONVENIENT LET ME TELL YOU! :)
| smn1234 wrote:
| Not S3 static site hosting + DynamoDB?
| SilasX wrote:
| I think I found the last one before the outage:
|
| https://news.ycombinator.com/item?id=32026565
|
| The ones after it are hours later and usually deleted, until this
| post (...71).
| jhatax wrote:
| It might just be my devices (iPhone, iPad), but the fonts for
| stories and comments have changed since the restore. Anyone else
| noticing this?
| clintonwoo wrote:
| When HN went down I was reading the stuff that was up before it
| went down on https://remix.hnclone.win
|
| GitHub page for that project:
| https://github.com/clintonwoo/hackernews-remix-react
| rob_c wrote:
| So, why and what lessons were learnt? We get to post about others
| having that problems
| willhinsa wrote:
| I just figured my internet wasn't working! It's a testament to
| how well this site works!
| rr808 wrote:
| Whenever I get frustrated by cloud complexity I wonder if its all
| worth it, as HN, stackoverflow, camelx3 etc are still on real
| servers. Maybe it is worth it after all.
| geoffeg wrote:
| I've wondered about this a lot recently as I've spent a lot of
| time recently fighting with Terraform, Atlantis, CircleCI,
| github and AWS. The first half of my career was deploying code
| to various UNIX machines. When there was a problem I could
| login to that machine and use various (common) tools to
| diagnose the issue. I may not have had root but I was at least
| able to gain insight into what was likely the culprit. The
| interface was immediate and allowed quick iteration to test out
| a solution.
|
| It feels like we've lost a lot of that observability and
| immediacy with the cloud. It's not as easy to quickly
| understand the larger picture. You can understand the state of
| various services with the web console or command line tools but
| tracing a path through those services is much less obvious and
| efficient.
|
| I'm kind of nervous to even discuss this as I wonder if it's
| just my age showing, especially since I see very people mention
| this as one of the downsides of various cloud solutions. Maybe
| I'm just jaded?
| bbkane wrote:
| Oh my goodness yes. I had the "great" idea to use Azure
| Functions to do a task at work. It's **ing insane how
| difficult it is to specify an Azure Function all in code with
| reasonable CI/CD, AD permissions, logging, and dev/prod
| instances. I wrote about what it takes at
| https://www.bbkane.com/blog/azure-functions-with-terraform/
| but the experience really soured me on cloud services.
| rr808 wrote:
| Lol, same boat. Not sure if I'm the old wise guy who really
| knows his sh*t about what's important, or the old useless guy
| rambling and moaning in the corner.
| jonnycomputer wrote:
| Did global productivity go up today, or were we all hopelessly
| clicking refresh to see if HN was back up?
| bawolff wrote:
| Rogers engineers too busy refreshing ;)
| caycep wrote:
| granted, this is probably the most exciting post here all day!
| hbn wrote:
| Prove it
| nmajor25 wrote:
| Phew, we're back!
| Pr0ject217 wrote:
| Yay. Thank you HN.
| Sjonny wrote:
| HN was down?
| [deleted]
| Mockapapella wrote:
| Well on the plus side I got a lot done today
| elcapitan wrote:
| Oh noes, now my productivity is in the toilet.
| bell-cot wrote:
| 127.0.0.1 news.ycombinator.com # REQUIRED for meaningful
| productivity
| Amfy wrote:
| Interesting though, HN is now hosted on AWS and no longer on bare
| metal (m5 hosting).
|
| % host news.ycombinator.com
|
| news.ycombinator.com has address 50.112.136.166
|
| and also interesting: DNS TTL is set to 1 (one).
| bestinterest wrote:
| Oh that is interesting, I guess they just spun up a beefy EC2
| instance. I'm noticing slower performance, I used to get about
| <200ms for front page. Now it's 500ms-1s? Or is this placebo
| with my bias to thinking AWS is slow?
| NetRange: 50.112.0.0 - 50.112.255.255 CIDR:
| 50.112.0.0/16 NetName: AMAZON-EC2-USWESTOR
| NetHandle: NET-50-112-0-0-1 Parent: NET50
| (NET-50-0-0-0-0) NetType: Direct Allocation
| OriginAS: AS14618 Organization: Amazon.com,
| Inc. (AMAZO-47)
| doublerabbit wrote:
| I hope it's temporary. Would hate HN to move to the "cloud"
| from bare metal.
| GekkePrutser wrote:
| This, totally.
|
| It's great that they were able to spin it up in the cloud
| for recovery purposes. But it's more legendary on a real
| server <3
|
| Yes I'm old :P
| [deleted]
| [deleted]
| jdoliner wrote:
| That was a rough 6 hours
| tpmx wrote:
| It really was.
| sabjut wrote:
| Yeah, I had to _actually work_ for once. Glad that I can
| finally distract myself again :)
| dboreham wrote:
| Someone suggested lobste.rs, which I hadn't come across before.
| It needs an invite though so read-only today for me.
| hansword wrote:
| I went for a long walk. When I came back, HN was still down. :(
| [deleted]
| peter_retief wrote:
| What happened? wasn't resolving at all.
| crorella wrote:
| it is back! I wonder what happened.
| jmt_ wrote:
| https://twitter.com/HNStatus/status/1545409429113229312
|
| https://twitter.com/HNStatus/status/1545461511870566400
|
| Disk and fallback server failure. Was definitely a long day for
| their ops team, on a Friday no less.
| ricefield wrote:
| thank god
| throw7 wrote:
| thanks!
| gruturo wrote:
| Thanks for all the daily work in running and keeping up the site,
| which we too easily take for granted, even those of us with
| similar jobs.
| function_seven wrote:
| Everyone, you're welcome. My last F5 must've jogged it loose
| finally.
| badrabbit wrote:
| Thank you for your service.
| [deleted]
| leeoniya wrote:
| are you wearing a cape, or are you not?
| hguant wrote:
| 07
| proactivesvcs wrote:
| You can't fool us, we know you're an F7.
| DamnInteresting wrote:
| A refreshing take!
| gedy wrote:
| Name almost checks out
| raydiatian wrote:
| Thank you
| dom96 wrote:
| Hurray! Is this the longest downtime ever for HN? I don't
| remember it being down for this long in the past, anyone know?
| [deleted]
| Jcampuzano2 wrote:
| My routine is usually to check HN first thing in the morning when
| sitting down at my computer before work.
|
| I definitely spent a non-reasonable amount of time thinking my
| internet had a problem trying to open HN since it's always just
| been so constant.
| ImpulseGuided wrote:
| Life felt so meaningless. Don't ever do this to me again.
| ghostoftiber wrote:
| RCA? AAR? "Ooops someone tripped over the cord?"
| pjbeam wrote:
| Just when I was starting to get some work done. Thank you!
| thebitstick wrote:
| I nearly died... of boredom...
| aasasd wrote:
| You don't say.
| collegeburner wrote:
| Ong i couldnt stand being that productive so relieved
| sillysaurusx wrote:
| HN was down because the failover server also failed:
| https://twitter.com/HNStatus/status/1545409429113229312
|
| Double disk failure is improbable but not impossible.
|
| The most impressive thing is that there seems to be no dataloss,
| almost whatsoever. Whatever the backup system is, it seems rock
| solid.
| tgflynn wrote:
| According to this comment:
| https://news.ycombinator.com/item?id=32024485
|
| each server has a pair of mirrored disks, so it seems we're
| talking about 4 drives failing, not just 2.
|
| On the other hand the primary seems to have gone down 6 hours
| before the backup server did, so the failures weren't quite
| simultaneous.
| bell-cot wrote:
| I'm extremely curious about the makes & models of the failed
| hardware...
| hunterb123 wrote:
| What was the test to determine the dataloss?
| Cerium wrote:
| I came to the same conclusion by observing that there are
| posts and comments from only eight hours ago.
| jbverschoor wrote:
| So that means dataloss.. Probably restored from backup.
|
| Good news for people who were banned, or for posts that
| didn't get enough momentum :)
|
| edit: Was restored from backup.. so def. dataloss
| joshuamorton wrote:
| > So that means dataloss.. Probably restored from backup.
|
| If the server went down at XX:XX, and the backup they
| restored from is also from XX:XX, there isn't dataloss.
| If the server was down for 8 hours, the last data being 8
| hours old isn't dataloss, it's correct.
| sillysaurusx wrote:
| Informal. My last upvote was pretty close to when HN went
| down, so I expected my karma to go down, but it didn't.
|
| Also I remember the "Why we're going with Rails" story on the
| front page from before it went down.
| [deleted]
| digitallyfree wrote:
| By second disk failure do they mean that the disks on both the
| primary and fallback servers failed? Or do they mean that two
| disks (of a RAID1 or similar setup) in the fallback server
| failed?
|
| The latter is understandable, the former would be quite a
| surprise for such a popular site. That means that the machines
| have no disk redundancy and the server is going down
| immediately on disk failure. The fallback server would be the
| only backup.
| spiffytech wrote:
| 14 hours ago HN failed over to the standby due to a disk
| failure on the primay. 8 hours ago the standby's disk also
| failed.
|
| Primary failure:
| https://news.ycombinator.com/item?id=32024036 Standby
| failure:
| https://twitter.com/HNStatus/status/1545409429113229312
| dboreham wrote:
| CP => !A
| [deleted]
| marcosdumay wrote:
| If you have an active/passive HA setup and don't test it
| periodically (by taking the active server offline and switching
| them afterwards), my guess is that double disk failures will be
| more common than single disk failures for you.
|
| Still, I see no reason for prioritizing that failure mode on a
| site like HN.
| pastor_bob wrote:
| is dang pushing changes and such on his own?
|
| sounds like it is run by one guy
| sillysaurusx wrote:
| HN will be around a hundred years. I think it's more than
| just a forum. We've seen lots of people coordinate during
| disasters, for example. Dan and his team do a good job
| running it. (I'm not a part of it.)
|
| EDIT: My response was based on some edits that are now
| removed.
| rat9988 wrote:
| You are overstimating HN way too much.
| nominusllc wrote:
| A hundred years, I give it 10 tops.
| sillysaurusx wrote:
| It's already been around since 2007. How many decades
| does HN need to be around before people realize it's an
| institution?
| grepfru_it wrote:
| Slashdot has been around since 1997 and people still rave
| about its moderation system today. However, while I have
| high hopes for HN, it could very well go the way of digg
| overnight
| GekkePrutser wrote:
| I doubt that though. Digg was hyped way too much and the
| inevitable decline that comes after a hype killed it.
| Some things are good enough to survive that phase but
| Digg wasn't. HN never had a hype phase, just slow but
| strong & steady growth. And not growing too much either.
|
| It seems the perfect circumstances to really last. It
| doesn't have an invasive business model, or investors
| screaming for ROI either. That's the kind of thing that
| often leads to user-hostile changes that so often start
| the decline into oblivion.
|
| Also, I would imagine it's pretty cheap to host, after
| all it's all very simple text, I don't think it hosts any
| pictures beside the little Ycombinator logo in the corner
| :)
| [deleted]
| jethro_tell wrote:
| The reason it's an institution is because it hasn't been
| bought by some corp trying to squeeze value out of
| eyeballs, which is why it hasn't really changed much.
|
| However, it takes money and time to keep it around in a
| not for profit way, so it will be an institution as long
| as it's funding is the same.
| GekkePrutser wrote:
| Yeah I really hope that if Ycombinator ever wants to pull
| out, that they don't sell it but let the community pull
| together to support it. I'd gladly donate to keep it
| running as it is.
|
| It would be even better if they just keep doing it as
| they are though <3
| jacquesm wrote:
| What makes you think that? That's just a tweet from an
| unrelated account.
| pastor_bob wrote:
| Nevermind, I thought the OP ran that twitter account
| swyx wrote:
| theres two people fulltime on it but dang appears to be both
| DBA and SRE
| openthc wrote:
| And Mod; hope he gets three cheques
| chippiewill wrote:
| > Double disk failure is improbable but not impossible.
|
| It's actually surprisingly common for failover hardware to fail
| shortly after the primary hardware. It's normally been exposed
| to similar conditions to what killed the primary and the strain
| of failing over pushes it over the edge.
| GekkePrutser wrote:
| Isn't that more for load balancing than failover?
|
| For load balancing I would consider this very likely because
| both are equally loaded. But "failover" I would usually
| consider a scenario where a second server is purely in wait
| for the primary to fail, in which case it would be virtually
| unused. Like an active/passive scenario as someone mentioned
| below.
|
| But perhaps I got my terminology mixed up. I'm not working
| with servers so much anymore.
| davedunkin wrote:
| > Double disk failure is improbable but not impossible.
|
| It's not even improbable if the disks are the same kind
| purchased at the same time.
| kabdib wrote:
| I once had a small fleet of SSDs fail because they had some
| uptime counters that overflowed after 4.5 years, and that
| somehow persistently wrecked some internal data structures.
| It turned them into little, unrecoverable bricks.
|
| It was not awesome seeing a bunch of servers go dark in just
| about the order we had originally powered them on. Not a fun
| day at all.
| spiffytech wrote:
| Yep: if you buy a pair disks together, there's a fair chance
| they'll both be from the same manufacturing batch, which
| correlates with disk defects.
| clintonwoo wrote:
| This makes total sense but I've never heard of it. Is there
| any literature or writing about this phenomenon?
|
| I guess proper redundancy is having different brands of
| equipment also in some cases.
| athenot wrote:
| Not sure about literature but that was a known thing in
| the Ops circles I was in 10 years ago: never use the same
| brand for disk pairs, to minimize wear-and-tear related
| defects from arising at the same time.
| AceJohnny2 wrote:
| I don't know about literature, but in the world of RAID
| this is a common warning.
|
| Having a RAID5 crash and burn because the backup disk
| failed during the reconstruction phase after a primary
| disk failed is a common story.
| eganist wrote:
| Not sure about literature, but past anecdotes and HN
| threads yes.
|
| https://news.ycombinator.com/item?id=4989579
| toast0 wrote:
| I also don't know about literature on this phenomenon,
| but i recall HP had two different SSD recalls because
| when the uptime counter rolled over, they would fail.
| That's not even load dependent, just did you get a batch
| and power them on all at the same time. Uptime is too
| high causing issues isn't that unusual for storage,
| unfortunately.
|
| It's not always easy, but if you can, you want
| manufacturer diversity, batch diversity, maybe firmware
| version diversity[1], and power on time diversity. That
| adds a lot of variables if you need to track down issues
| though.
|
| [1] you don't want to have versions with known issues
| that affect you, but it's helpful to have different
| versions to diagnose unknown issues.
| GekkePrutser wrote:
| The crucial M4 had this too but it was fixable with a
| firmware update.
|
| https://www.neoseeker.com/news/18098-64gb-
| crucial-m4s-crashi...
| davedunkin wrote:
| I hadn't heard of it either until disks in our storage
| cluster at work started failing faster than the cluster
| could rebuild in an event our ops team named
| SATApocalypse. It was a perfect storm of cascading
| failures.
|
| https://web.archive.org/web/20220330032426/https://ops.fa
| ith...
| Flott wrote:
| Great read, thank you!
| mceachen wrote:
| Wikipedia has a section on this. It's called "correlated
| failure."
| https://en.wikipedia.org/wiki/RAID#Correlated_failures
| bragr wrote:
| Yeah just coming here to say this. Multiple disk failures
| are pretty probable. I've had batches of both disks and
| SSDs with sequential serial numbers, subjected to the same
| workloads, all fail within the same ~24 hour periods.
| mpyne wrote:
| Seems like it was only a few days ago that there was a
| comment from a former Dropbox engineer here pointing out
| that a lot of disk drives they bought when they stood up
| their own datacenter had been found to all have a common
| flaw involving tiny metal slivers.
| schroeding wrote:
| Had the same experience with (identical) SSDs, two
| failures within 10 minutes in a RAID 5 configuration.
|
| (Thankfully, they didn't completely die but just put
| themselves into read-only)
| [deleted]
| GekkePrutser wrote:
| Eek - now I'm glad I wait a few months before buying each
| disk for my NAS.
|
| Not doing it for this reason but rather financial ones :)
| But as I have a totally mixed bunch of sizes I have no RAID
| and a disk loss would be horrible.
| deltarholamda wrote:
| >the failover server also failed
|
| Those responsible for the sacking have also been sacked.
| [deleted]
| jchw wrote:
| I assume this post will be flagged as off topic, but I actually
| just went to visit moments before it came back up. I didn't
| figure out that it wasn't an issue with my internet connection
| until I saw this post.
| codegeek wrote:
| This was traumatic :). Felt like I had lost a part of me lol.
| Glad to be back.
| cableshaft wrote:
| How many times did people refresh to check this today? I did at
| least four times. I might have a bit of a problem.
| beckingz wrote:
| like 40 times. Probably more.
| marcosdumay wrote:
| What? I refreshed way more than 4 times before I believed it
| was offline.
|
| At the early 00's, when Google went offline I wouldn't believe
| it, and go check my connection (even if I was fetching other
| sites at the same time). Looks like nowadays HN is in that
| place.
| timbit42 wrote:
| My RSS app refreshes every hour.
| nickmyersdt wrote:
| The muscle memory that kept opening HN was strong today. Thank
| you for bringing it back.
| Bayart wrote:
| Did you turn it off and on again ?
| MrBlueIncognito wrote:
| Top post in just 3 minutes
|
| Edit: 100 points in 4 minutes
| bevdecloud wrote:
| Didn't even notice
| erwincoumans wrote:
| Thanks for fixing the site! Keep up the good work.
| nimbius wrote:
| but why was it down though
| drpgq wrote:
| I was wondering if HN was running on Rogers
| UncleOxidant wrote:
| Same here. HN is from Canada? Who knew?
| gmiller123456 wrote:
| Thought my RSS reader was broken because I came to the site and
| saw it was up, then saw this headline.
___________________________________________________________________
(page generated 2022-07-08 23:00 UTC)