(C) Daily Kos This story was originally published by Daily Kos and is unaltered. . . . . . . . . . . The Great Daily Kos Outage of 2023 [1] ['Daily Kos Staff', 'Backgroundurl Avatar_Large', 'Nickname', 'Joined', 'Created_At', 'Story Count', 'N_Stories', 'Comment Count', 'N_Comments', 'Popular Tags'] Date: 2023-03-15 Amazon Web Services, probably Holy crap, this has been brutal, and it’s not over. But progress is ongoing. The tl;dr version of what happened is that Amazon Web Services, our web hosting service, crashed. This affected wide swaths of the web, like Reddit. But while all those other sites were able to come back online, the Amazon crash exposed bad or corrupted code on our end. The result was this weird situation in which stories created during the times of Amazon’s server issues crashed the site if they were updated. So periodically the site would come back online, only to have someone, somewhere, update a story and bring it all crashing back down again. Our engineering team has been working for over a day, with barely any sleep, to diagnose and fix the problem. The solution, which requires changing settings on our database and reconfiguring tables (or something like that, I’m not an engineer), has already begun. The first phase you’ve already seen—we took the site down for 30 minutes and cleaned up a subset of tables. Staff can now post stories, but the community cannot. Our devs want posting slowed to a trickle so they can monitor performance and ensure the fix holds. Commenting should be possible, but don’t be surprised if it’s buggy. Assuming all goes well, the site will be taken down again tonight, during our night traffic lull, and the rest of the database will be cleaned up. That should take a couple of hours. When we’re back up, everything will—knock on wood—be back to normal. In some ways, we were lucky. We weren’t hit with big news. Yes, Abbreviated Pundit Roundups and Ukraine Updates never came, and there were plenty of Republican shenanigans left unreported. But no one important died, or gave birth, or did something to change the course of history. Could you imagine this happening on Election Day? Or the day when Donald Trump is indicted? (I’m going to manifest that into reality, I promise you!) x Y'all picked a heckuva time to go down on the day Ukraine launched a major offensive, Biden resigned to pave the way for Harris 2024, and Trump was arrested. — Curtis (@RebrandNuggets) March 15, 2023 But it was still … lonely. A reminder that even at our most obnoxious, the alternative is much, much worse. Give me pie wars, all the flavors, even the “religion” one. I don’t ever want to live in a world without them. Give me all the Marjorie Taylor Greene updates, even if they’re stupid. Inform me of Trump’s latest idiocy, even if I want to pretend that I’ve moved on. Is Mitch McConnell still in the hospital for a “concussion protocol” that clearly doesn’t exist, because no one stays in a hospital for a week for a regular concussion? Please let me know! Also, how many more woke banks are going to fail for having too many women and brown people in their employ? This is important stuff! x I'm telling you, they're actually running with the "woke banks" thing. They're already using scary placeholder acronyms ESG and DEI, which to them mean "diversity." It serves to obfuscate the reality: there was a panicky bank run, frontrun by some of the GOP's biggest donors. https://t.co/Rrhxj6rTOs — Ben Collins (@oneunderscore__) March 12, 2023 And yeah, Twitter exists, and I can get that information there. But our analysis and contextualization is better, not to mention the community. It’s not even close. So I miss what the crew had to say about all those topics and more. And I missed what you guys had to add to the conversation. So yeah, it was too quiet, and too lonely, and I didn’t like it. And while I wish we were fully back up, I am glad we are on our way. Let’s not do this ever again, okay? [END] --- [1] Url: https://www.dailykos.com/stories/2023/3/15/2158359/-The-Great-Daily-Kos-Outage-of-2023 Published and (C) by Daily Kos Content appears here under this condition or license: Site content may be used for any purpose without permission unless otherwise specified. via Magical.Fish Gopher News Feeds: gopher://magical.fish/1/feeds/news/dailykos/