[HN Gopher] Southwest cancels 5,400 flights in less than 48 hours
___________________________________________________________________
Southwest cancels 5,400 flights in less than 48 hours
Author : edward
Score : 371 points
Date : 2022-12-27 15:13 UTC (7 hours ago)
(HTM) web link (www.npr.org)
(TXT) w3m dump (www.npr.org)
| boomboomsubban wrote:
| There was a news story a few months ago about some airlines
| changing the way their crew software works to stop a third party
| app employees were using to better track their hours Was
| Southwest one of those airlines?
| makestuff wrote:
| Looks like it was American
| https://news.ycombinator.com/item?id=33233975
| dang wrote:
| Recent and related:
|
| _Massive Southwest Airlines disruption leaves customers
| stranded_ - https://news.ycombinator.com/item?id=34145286 - Dec
| 2022 (102 comments)
| paultopia wrote:
| So, like, what do we think... I have a flight on Jan 3. Book a
| different airline now or pray they've gotten their heads out of
| their asses by then?
| elijaht wrote:
| I would hedge with a refundable ticket to a different airline.
| If Southwest is still having issues you might be able to get
| the other ticket comped to some degree and still have a flight,
| if they aren't having issues you can refund the other fare
| paultopia wrote:
| Ooh, good idea. Thank you.
| DrWumbo wrote:
| [flagged]
| dang wrote:
| Please don't take HN threads into nationalistic flamewar. It's
| not what this site is for, and destroys what it is for.
|
| https://news.ycombinator.com/newsguidelines.html
| josephcsible wrote:
| Because trains don't need scheduling software or crews?
| ciphol wrote:
| Trains have major weather issues much less frequently than
| planes.
| [deleted]
| milkytron wrote:
| They do. But at least there would be another option for
| traveling.
| zbentley wrote:
| In addition to the advantages of trains over flights
| mentioned by some of the other commenters, trains also:
|
| - Require less specialized crew training, and don't require
| as many crew members to be equivalently specialized (though
| you'd be surprised at the depth of training that's still
| required to crew an Amtrak).
|
| - Can stop, and stop at offloading/transfer points, more
| easily than planes. In the event of issues this further
| increases flexibility of the system.
|
| - Have considerably more built-in redundancy than planes, and
| _some_ built-in swap-ability of components during a trip by
| removing or replacing cars. I say "some" in italics because
| I've recently been ... er, very directly acquainted with how
| critical a single pantograph failure can be to an entire rail
| line.
|
| - _Might_ , if implemented via state-operated rail or state-
| sponsored monopoly, offer the ability to replace, deploy, or
| reroute entire trains in the event of unexpected
| capacity/mechanical issues. This advantage is a bit of a
| wash, though, in that we'd probably have more redundancy in
| air travel if there were fewer larger carriers (this isn't
| guaranteed and trades off with other issues, but
| redundancy/flexibility is an advantage of consolidation).
|
| - Are mechanically simpler than planes, and thus require less
| overhead before being deployed and have fewer "no-go"
| inspection conditions that can introduce unexpected
| unavailability.
| overtonwhy wrote:
| High speed rail infrastructure would be new and modern.
| sokoloff wrote:
| One of the problems with SWA's case is that cabin crews can
| work a longer duty day than flight crews and airplanes can't
| leave on passenger service without a full complement of both,
| so the airplane scheduling problem might be more complex.
| lostinroutine wrote:
| I'm guessing the GP means that there will hopefully be an
| increased interest for alternatives to planes that one can
| take when there are air travel crises like this.
| samename wrote:
| Doubt the airline lobby that controls Congress will allow
| anything like that to pass.
| LeifCarrotson wrote:
| Same thing happened on July 20, 2016:
|
| https://www.dallasnews.com/business/local-companies/2016/08/...
|
| at the time, then-CEO now-chairman Gary Kelly said:
|
| > "What's unique is the partial failure, it's never happened," he
| said. "This isn't a drill you can run."
|
| https://web.archive.org/web/20161112192103/http://www.dallas...
|
| Delta had a similar outage due to a datacenter fire, grounding
| all domestic flights. Southwest was uniquely slow in taking days
| to start up again. And if the way my American Airlines ticket
| switched my birthdate to January 1st, 2000 is any indication,
| many airlines still need to modernize.
| photochemsyn wrote:
| Modernization of equipment, hiring more pilots and other
| employees, investing in updating the code base - how can that
| be done? It's far more important to keep the stock price high
| by whatever means necessary, such as using government bailouts
| to buy back shares.
|
| Investment capitalism is really a garbage system when it comes
| to building and maintaining basic infrasctructure like
| transportation, electricity grids, roads and so on. China has
| demonstrated that convincingly over the past two decades,
| hasn't it?
| pksebben wrote:
| "this isn't a drill you can run". And yet, Netflix has chaos
| Kong do it with regularity.
|
| The difference between what's true, what some people will buy,
| and what you can get away with saying is gross, y'all.
| Supermancho wrote:
| > "this isn't a drill you can run". And yet, Netflix has
| chaos Kong do it with regularity.
|
| Netflix and airlines are so different as to make this
| comparison laughable. The cost of setup and consequence of
| problems actually being found (ie Federal Regulations) that
| are not addressable (it's not like SWA didn't know about some
| of the eventualities), easily outclasses the need for testing
| every combination of situations. Kong doesn't run anything
| that has to do with weather turning jet fuel into sludge or
| 12x pre-staffing in case of massive computer failures along
| with assessing the possible legal consequences from each
| locale. The hubris of pretending that physical services on a
| national scale, is as deterministic as a complex automated
| system, is unsurprising from a certain crowd, I guess.
| Chris2048 wrote:
| > Delta had a similar outage due to a datacenter fire
|
| They only have one geo-located data-centre?
| vrc wrote:
| Google runs Disaster Recovery Training annually (DiRT) where
| security teams are tasked with simulating these "black swan"
| events. Seems like this practice needs to expand to more
| industries.
| landemva wrote:
| This does not need Google deep pockets. It needs the
| motivation and some funding. SWA does not care.
|
| Pull the backup tapes, hand those to DR team, provide bare
| metal, and start the stopwatch. I participated in this in
| 1990s across the Mississippi.
| paganel wrote:
| Google is also a trillion dollar company, as other have
| pointed out Soutwest is a low-cost carrier which most
| probably doesn't have the luxury of hiring FAANG-level
| engineers on 500k yearly comp in order to best simulate
| "black swan" events.
| [deleted]
| 0xBDB wrote:
| They don't seem to pay competitively with banks, let alone
| FAANGs, though the benefits and culture are (or were)
| reputedly fantastic.
|
| Source: Am a local who's been headhunted by them a few
| times but never got beyond the initial discussion with the
| headhunter for this reason.
| makestuff wrote:
| This is probably the best argument for AWS/GCP/Azure even
| though it is becoming more and more obvious you don't
| really save that much money.
|
| If you have a black swan event like this and you listened
| to your solutions architect you will have a disaster
| recovery plan or even better a multi region setup. Worst
| case you have highly paid support engineers at the cloud
| providers who will do everything they can to get you back
| online.
| 0cf8612b2e1e wrote:
| This does not seem like a hardware failure scenario where
| the cloud has anything to offer. More like their
| intricate software/database systems became out of sync
| with reality and disentangling the mess is a highly
| manual process.
| makestuff wrote:
| In this case no, but I was more referring to the 2016
| delta ground stop that was due to their datacenter
| burning to the ground.
| therealjumbo wrote:
| I remember reading about the Delta incident a ways back,
| here they claim it cost them ~$150 million. https://www.d
| atacenterknowledge.com/archives/2016/09/08/delt...
|
| That's not the article I hoped to find however. I seem to
| remember there was another article where they hired a
| investigator/consultant to figure out the price to
| migrate to the cloud and ensure "this never happens
| again."
|
| My recollection of that was: their scheduling/ops team is
| also in the same city (Atlanta GA) as this datacenter,
| and that teams work was brought to a halt by the
| datacenter outage. The investigator concluded that Delta
| would need redundant copies of the ops team or the whole
| effort of moving the software to the cloud would just be
| at risk to something happening to the human team all in
| the same city. That would obviously cost to much money,
| so Delta decided to skip it.
| namdnay wrote:
| Regarding the employees, keep in mind that neither SWA (nor
| any other airline for that matter) have big software
| engineering departments. It's all outsourced to either
| generalist bodyshops for custom/peripheral systems (IBM,
| Accenture) or specialist shops for core (Amadeus, SABRE)
| adamsb6 wrote:
| At Facebook we would simulate an entire datacenter
| disappearing.
|
| When we first started doing it the datacenter would be chosen
| months in advance so that teams would have plenty of time to
| ensure their services can run without that specific
| datacenter.
|
| When I left this year, the datacenter would be randomly
| chosen on the same day it would be cut off.
| chasd00 wrote:
| That's pretty cool and ideal practice for a software firm
| but in one of the reddit threads they're talking about mass
| quits/refusal to work of ground crew at Denver because of
| the weather. I wonder how you could ever prepare for that?
| Keep a backup, airport scoped, ground crew in the waiting
| room??
|
| You can't really do hot spares for people without time to
| gear/train up and the weather event is so widespread I
| doubt there's enough spare SWA human capacity across the
| whole nation even if you had C130s on standby everywhere
| ready to take workers where they're needed most. From a
| national security perspective, situations like this is why
| the Marines exist right? Ensure a rapid response while the
| rest of the machine gets moving. I feel bad for everyone
| involved, those affected and those trying to figure out a
| solution.
| landemva wrote:
| > I wonder how you could ever prepare for that?
|
| Management could consider how pay and performance
| programs can help ensure business continuity.
|
| HR and MBA xls wizards don't understand how to manage for
| business longevity.
| manigandham wrote:
| Do you really think the MBA wizards can't figure out some
| basic pay issues?
|
| It seems you're discounting just how complex HR can be,
| especially in the face of exigent circumstances. No
| amount of bonuses will immediately staff up an entire
| terminal in the face of a massive snowstorm.
| lotsofpulp wrote:
| While they may be able to figure it out, optimizing pay
| to quality of life at work ratios to ensure long term
| employee retention and loyalty has certainly not been a
| priority.
| dpkirchner wrote:
| Pay might not really be enough. Maybe management could
| try to find folks to babysit kids/take care of parents
| trapped at home, freeing up workers to come fly. They'll
| certainly fail, but at least they'll understand the
| plight of their workers.
| drdec wrote:
| It doesn't seem that farfetched for an airline to run a
| drill where a given airport is assumed inoperable to see
| how the system reacts. The expectation shouldn't be the
| same as the data center failure but you can learn what
| you aren't doing well enough.
| [deleted]
| tomjakubowski wrote:
| Training for these scenarios may help with responding to true
| black swan events, like Rick Rescorla's WTC evacuation drills
| ahead of 9/11. But, nitpicking, if you've predicted something
| will happen, by way of simulating it, it's not a black swan
| when it does.
| adrr wrote:
| There are cost considerations. Business continuity costs
| money. Finance firms have significant capital and income to
| have empty but built out building around around airports
| for business continuity. Which doesn't even make sense
| since they can work from home as proven with covid.
| Airlines can't work from home.
| barrenko wrote:
| Personally one of the basic tenets of my adulthood is
| realizing how many companies are a _hair_ away from a similar
| scenario (differing in magnitude from an airline ofc).
| bookofjoe wrote:
| EMP FTW
| bobkazamakis wrote:
| >Seems like this practice needs to expand to more industries.
|
| I think you've mistaken this for something immediately
| increases quarterly gains with no regard to long-term
| strategy.
| 23B1 wrote:
| What's most annoying is that there's plenty of employees on
| the front line who not only care about testing for this
| sort of thing, but it actually interests them, they're
| motivated by it, and they understand the dire reality of
| what happens - to them, primarily - if the company isn't
| prepared to handle it.
|
| And you can guess what their managers' response typically
| is: "We need to focus on OKRs and QBRs and KPIs right
| now... maybe next quarter"
|
| I'm fully convinced that achieving 'manager status' is
| directly correlated to cowardice. Companies need top-down
| decision-making, but those decision-makers need to spend
| more time on the front line.
| giantrobot wrote:
| > Companies need top-down decision-making, but those
| decision-makers need to spend more time on the front
| line.
|
| This is not rewarded so it doesn't happen. Managers are
| rewarded for _line goes up_ so they only focus on _line
| goes up_. If line ever doesn 't go up it costs them money
| (advancement, compensation) even if there's little they
| could have done to make line go up.
| Analemma_ wrote:
| In that case the government should smash Southwest with a
| billion dollar fine so the cost of not doing this drilling
| exceeds the cost of doing it.
| hallway_monitor wrote:
| Asking the government to step in for additional
| regulation is rarely helpful. For this type of failure,
| the free market will determine whether processes and
| tools improve, or whether the status quo is good enough.
| heavenlyblue wrote:
| Free market?
| ceejayoz wrote:
| Regulation sometimes helps remind the free market that
| fuckups like this can come with real human costs.
| astrange wrote:
| What's an airline got to do with the free market? They're
| a extremely highly regulated business.
| dzhiurgis wrote:
| Until regulation steps in.
| kevinmchugh wrote:
| My wife and I spent a few hours this morning dealing with
| the cancellation of our return flight. Southwest has long
| been preferable for me in many cases, including my most
| flown route. Between the headache of this outage and the
| apparently dismal state of their operations there's a
| strong chance I never fly with them again.
| jdeibele wrote:
| Our family was in San Jose last week when our Southwest
| flight was cancelled.
|
| It used to be that my first priority would be to go into
| the terminal and try to talk to somebody. I figured they
| were the experts. From what I've read, the staff use an
| antiquated system that takes you from one airport to
| another, then they can try to get you from that city to
| where you want to go. That's why there's so much tapping
| of keys and why it takes so long.
|
| It's better to present them with a route that you've
| found on Google Flights or similar. The Southwest first
| flight out was supposed to be yesterday evening, the day
| after Christmas. In our case, the only thing we could
| find before Christmas was getting us from SJC to Seattle
| via Phoenix on Alaska. We ended up renting a car and
| driving home to Portland. Things got bad around Eugene -
| I stopped counting after 40 wrecked cars and semis - and
| got worse as you got closer to Portland.
| astrange wrote:
| Airlines don't have quarterly gains. They regularly go
| bankrupt and get picked up again, because the country needs
| airlines and because they have large union contracts.
|
| See any gains here?
|
| https://www.google.com/finance/quote/LUV:NYSE?window=5Y
| splistud wrote:
| [dead]
| Scarblac wrote:
| Google is made of money, and the reason they are is not
| because of DiRT. Other industries can't afford the same
| things that Google can while continuing to be a business.
| fragmede wrote:
| After canceling 5,400 flights, I can't see how Southwest
| can afford _not_ to test. Even if they only made $1000 off
| each flight, that 's still $5 million they just lost.
| Scarblac wrote:
| They probably would not have made it this far if they
| tested for every possible scenario, their margins are
| razor thin.
|
| Anyway, doesn't even the best testing only catch 40% of
| bugs or thereabouts? It's not a silver bullet.
| rumdonut wrote:
| I figure most companies are too small for that to be
| budgeted. Though, it's possibly a good selling point for
| cloud if it's capable of it.
| JustLurking2022 wrote:
| Cloud doesn't solve badly designed processes or poorly
| written software, which seem to be at play with Southwest.
| Yes, it can help provide more stable infrastructure and
| there are some (but by no means all) black swan events that
| can be mitigated simply by throwing more kit at the problem
| during a surge but it's no silver bullet.
| thriftwy wrote:
| Yandex used to run datacenter loss training every week, where
| they will nullroute one DC and see what breaks all while
| handling live taffic.
| babyshake wrote:
| > "This isn't a drill you can run."
|
| When characterized as something that can't be done instead of
| something they don't know how to do, you know exactly where
| they are on the Dunning-Kruger curve.
| splonk wrote:
| > many airlines still need to modernize.
|
| Most of the travel industry runs on old software that would
| horrify a lot of people here, especially those who've never
| worked for a large, 30+ year old company. When I used to
| interview a lot of people I made it a point to mention some of
| the more "interesting" aspects so they'd know what they were
| getting into.
|
| One example: ever tried to book a flight a year in advance? On
| a lot (almost all?) of systems you can't, because the
| underlying date format is "DEC27".
|
| Edit to address a couple comments: logistics are hard and there
| are plenty of reasons why airlines wouldn't want to support
| booking that far out. However, the reason you can book a flight
| 330 days from now but not 360 days from now is almost certainly
| due to the date format. (I believe the windows used are less
| than 365 days because it's helpful to be able to have dates in
| the recent past. I remember seeing documentation for 360, but
| AA and United seem to be in the 330-340 range on their
| websites).
| xiphias2 wrote:
| ,, Most of the travel industry runs on old software that
| would horrify a lot of people here''
|
| If you can see how it works, it horrifies me even more as a
| traveller, as from outside it just doesn't work a lot of the
| time.
|
| Also if you just look at the video, we all know how bad these
| systems are, but are not able to do anything (starting
| anything new in the airline industry has too much cost).
| grepfru_it wrote:
| Likely cannot book that far not because of the underlying
| date format, but because of jet-a fuel prices which
| fluctuate. Airlines typically hedge their near term purchases
| with longer-term futures
| [deleted]
| np- wrote:
| To be fair, I think allowing flights a year in advance is
| probably far more complicated than just updating the
| underlying date format. Even if they were able to solve that
| problem, airlines probably can't easily operationally plan
| that far ahead due to so many moving parts, i.e. committing
| to routes and schedules, planning for staffing that far ahead
| of time, ever changing government restrictions, fuel price
| fluctuations, inflation, geopolitical realities, staffing,
| etc. I mean, imagine if they did, and something like COVID
| comes along again, it would cause far, far more disruption if
| they had booked out the next few years in advance (we had no
| idea how long COVID restrictions would last while we were in
| the heat of it, it's only clearer now in retrospect).
|
| Also speaking as a software engineer myself, it's almost
| never just a software fix that will magically solve everybody
| else's problems, that always ends up being just wishful
| thinking
| CPLX wrote:
| Airlines historically have not set their schedules more than
| a year in advance and it's not clear they want to.
| Frost1x wrote:
| While this is humorous in that there are limiting assumptions
| like this baked into the system, I also have to wonder, who
| needs or even wants to book a flight a year in advance? I
| dread planning out a flight 4 months in advance and dealing
| with the almost inevitable cascade of conflicts this
| introduces of juggling and rescheduling things to make things
| align correctly. One year makes me cringe.
| LarryMullins wrote:
| My family plans the yearly family get-together at the
| yearly family get-together. A year in advance. Except
| sometimes due to scheduling deconfliction, it's actually 10
| or 11 months in advance.. or 13 or 14 months in advance.
| The exact date floats and sometimes we are planning trips
| more than a year in advance.
| Induane wrote:
| Me for:
|
| - Annual conferences or conferences that occur every-other
| year - Planning family reunions because you need that kind
| of cat-herding lead time when you have 9 uncles/aunts on
| just ONE side of the family - Periods where I have some
| spare cash I'd like to lock in a getaway with before I
| spend it or something unexpected like the invasion of the
| Ukraine drives up fuel costs and overall prices... or a
| global pandemic hits - would be sweet if I could have
| rebooked some of my trips for for 1-2 years out when the
| pandemic hit - Travel for future medical stuff; at one
| point for 2-3 years I was taking my mom to the Cleveland
| Clinic every 4 months for periodic checks and it would have
| been super nice to be able to just book that stuff way in
| advance and have it all taken care of
|
| Etc
|
| etc
|
| etc
|
| I'd bet quite a few people would appreciate that ability
| dendrite9 wrote:
| There are events that I can see purchasing flights well in
| advance for. I used to a go to a conference that was held
| every other year at the same time, it would have been easy
| to buy tickets more than a year in advance for that without
| much concern. Eclipses, certain sporting events, or
| reservations for activities with a wait list of more than a
| year could qualify as well. Despite that I am like you and
| rarely have tickets far in advance of a trip.
| steveklabnik wrote:
| I could see it for major holidays. I spent too much money
| to fly home this year because I am bad at scheduling. I
| would consider booking next year's flight during this
| year's trip just so I know it's knocked out and I don't
| have to worry about it.
| masklinn wrote:
| > I also have to wonder, who needs or even wants to book a
| flight a year in advance?
|
| Major holiday, destination wedding, event known long in
| advance (e.g. Grandma's 80s does not come as a surprise).
| kenneth wrote:
| As a fun side thing, I am also a travel agent with access to
| some of these internal systems on the booking side. The
| technology is incredibly antiquated. Most of the US runs on a
| system called SABRE, which is basically a MS-DOS system with
| a text command line interface and its own language. It's all
| ASCII text based (and all in uppercase). It's straight out of
| the 80s. Travel agents need to buy special "errors" insurance
| to cover any losses caused by fucking it up (a typo could
| accidentally cancel a ticket and cause the client thousands
| in losses rebooking it).
| thebradbain wrote:
| They actually have a GUI interface over it now with the
| ability for power/legacy users to drop into the raw shell,
| if they wish. From feedback, many of the older agents
| actually prefer the command line, because it's muscle
| memory and an experienced agent can perform routine tasks
| that would take multiple screens in the UI with one hand in
| the way we're comfortable with our text editors.
|
| Granted, the rollout across airlines is probably glacial
|
| Source: I used to work there
| yardie wrote:
| I don't blame them. Modern UX has a huge problem with
| something as simple as date pickers. Preferring you
| scroll through 90+ items when a simple textbox would
| suffice.
| tintor wrote:
| What are the reasons preventing flight booking software
| modernization?
| nikanj wrote:
| 30 years of cumulative complexity in the existing stack,
| with endless edge-cases and special exceptions
| sjm-lbm wrote:
| .. and, as we're learning, extremely high penalties if
| one of those edge cases happens to cause a cascading
| failure.
| thebradbain wrote:
| GDS -- there's really only 3 centralized stores of real-
| time flight/hotel/booking information in the entire world
| (Sabre/SABRE, Amadeus, and Travelport). Almost every
| American airline uses Sabre (American Airlines is an
| interesting case in that it does not _technically_ in a
| legal sense, but actually it spun off and sold Sabre in
| 2000, so a lot of their core systems are forks of each
| other)
|
| Complexity -- Fundamentally you're looking at a logistics
| software, except unlike packages you're dealing with
| people who aside from expected destinations have travel
| lengths and time-in-air calculated down to the minute.
| Also unlike a package, a surprise multi-day trip,
| unexpected multi-leg journey, one day delay is not
| something passengers (and crew members) will accept or be
| at all ok with. And if any one thing goes wrong there's
| going to be cascading failures down the line-- so much
| that it may break your company's entire operating
| workflow (e.g. Southwest) entirely, and no software can
| overcome that kind of organizational gap.
|
| Airlines - There's not many commercial passenger airlines
| left in the US, especially that fly nationwide. Good luck
| trying to convince one of these giant behemoths to move
| to a non-battle-tested system for core operations,
| especially when decades-old industry software and
| practices around that software exist.
|
| Entrenched - Sabre is entrenched in airlines around the
| world. They don't just provide the booking services, they
| do the flight tracking, the ticket handling, the
| upgrading, the in-flight upgrades, missed connection
| handling, the flight scheduling algorithms, the pricing
| algorithms, the pilot and flight attendant time tracking,
| ground crew management, even the terminal software at
| each gate. To replace SABRE, you would physically need to
| rip out and then replace software around the world. And
| because agents don't work from an office usually, but at
| the airport, you're going to need to conduct trainings
| and provide support around the entire service area, which
| for the largest airlines is the entire world
|
| Scale -- A lot of Sabre's revenue comes from passengers
| boarded. It depends on the airline, but I believe the
| average is that each airline pays 10cents/customer
| boarded with their software (though with increases in
| passenger volume each year, it may be less now). Because
| Sabre is so prevalent, and so many flights use them, they
| can afford such a price. A company servicing just one
| regional passenger airline would absolutely not be able
| to compete on price, at least starting out
|
| Also-- Sabre's software itself is actually reliable! As a
| corporation it is slow clunky and bureaucratic, but the
| actual functionality it provides is stable, battle-
| tested, can handle any travel edge case you can think of,
| and fast and efficient for those who know how to use it,
| while also good enough at day to day operations that it
| doesn't take too much time to train new agents on how to
| use it for routine tasks.
| coldcode wrote:
| SABRE is ancient technology, but very reliable and at the
| same time extremely inflexible. Last time I saw it
| upclose in the early 2000's much of the core was still
| coded in IBM assembler, although over the decades more
| pieces were slowly being modernized so I have no idea
| where it is now. Sabre is a horrifically un- imaginative
| company where projects are measured in years and not much
| every changes.
|
| I think though Southwest's issues are more on their side.
|
| Yeah building a new GDS today is an exercise in insanity,
| it's a huge complexity nightmare and switching probably
| impossible. I always wondered if AI could eventually
| improve things, but the existing GDSs are unlikely to
| care much to try. It's basically a (tri)monopoly you can
| never break.
| [deleted]
| mastax wrote:
| Big software projects inevitably become expensive
| boondoggles that get everyone fired so nobody wants to do
| them until they're absolutely necessary.
| cmehdy wrote:
| The typical answer for old behemoths: it was built
| because it was necessary to build it, and it won't change
| until a change is necessary too. Wanting that change is
| not enough, it has to become an almost mechanical
| constraint, and usually the constraint gets noticed when
| it far outweighs the costs (and not just a little). Or is
| a noticeable threat to the system's existence.
| TigeriusKirk wrote:
| I remember using some version of SABRE through CompuServe
| back in the day. All command line stuff over a dial-up
| modem, but it was novel and cool to be able to book your
| own flights with it. It would be very annoying to still be
| stuck on that interface, though.
| rippercushions wrote:
| SABRE dates from 1960 and is by some reckonings _the_ very
| first piece of commercial (non-military, non-academic)
| software in the world.
|
| https://en.wikipedia.org/wiki/Sabre_(travel_reservation_sys
| t...
| bombcar wrote:
| 1/1/2000 sounds like a default value when it lost the data or
| never had it. Even more obvious would be if it threw you to
| 1/1/70.
| loeg wrote:
| It's definitely some default (or "null" in a DB) value and
| that is exactly what OP is insinuating.
| monocasa wrote:
| Particularly on mainframe systems like airline reservation
| systems tend to run on where the Y2k fix in a lot of cases
| for Cobol was to simply contextually know that certain fields
| couldn't have been created before 2000, so '00' BCD is simply
| year 2000.
| Kon-Peki wrote:
| On our database systems, we have some date fields for which
| the default value should never, ever be used and if it is,
| there is a big problem. All of those dates are set to the
| dates of well-known natural disasters that happened in the
| 1800s or earlier.
|
| The thought was that it needs to be something that isn't
| believable to a non-technical user seeing it on their
| computer screen. It turns out that this is not necessarily
| useful. I listened to a guy talking about some issues with a
| record; he says "1871? What's up with that?" And then just
| moved on as if "well it came out of the computer, must be
| right" or something.
|
| I think that databases need to have the concept of NaN for
| dates and time stamps, except that this should be
| configurable to something like a poop emoji or something like
| !?. It has to be something where your grandma would look at
| it and confidently say "your computer is broken"
| SoftTalker wrote:
| Your database should not allow invalid values to exist.
| That is what check contstraints, foreign keys, NOT NULL
| constraints, etc. are for.
| rini17 wrote:
| If you're such a DBA, the rest of an enterprise will
| quickly route around you.
| CydeWeys wrote:
| > I think that databases need to have the concept of NaN
| for dates and time stamps, except that this should be
| configurable to something like a poop emoji or something
| like !?
|
| How about just NULL?
| Kon-Peki wrote:
| Making database columns nullable isn't a free ride.
|
| _In some situations_ , you are trading one known point
| of failure for a million unknown ones. Among other
| problems :)
| Gwypaas wrote:
| "The server returned an unexpected error."
|
| Now GLHF!
| NovemberWhiskey wrote:
| > _" What's unique is the partial failure, it's never
| happened," he said. "This isn't a drill you can run."_
|
| The unspoken part you have to hear there is "... within the
| economic model of the airline business".
|
| Business continuity gets exponentially more expensive as you
| chase the blackest of swans: the sheer volume of plan
| development and maintenance, developing exercises, table-top
| vs. walkthrough vs. simulation, assumptions about how many
| different uncorrelated failures you're prepared for deal with
| at once etc.
|
| I've no doubt you could run an airline to be as resilient as
| (say) USAF Air Mobility Command, but no-one could afford the
| tickets.
| Frost1x wrote:
| What's ironic here is that groups like USAF are constantly
| pressured to adopt private industry models to be more
| "economically efficient" and completely ignoring that
| resiliency is a requirement baked into the high cost. I
| understand why both take the approaches they do but it seems
| everyone holds private industry barely running with no
| resiliency optimizations above all else, which don't make
| sense in all contexts. Corner cutting is fine in many
| contexts, especially when you know the side effects of their
| failures which may be quite insignificant.
| inetknght wrote:
| > _" What's unique is the partial failure, it's never
| happened," he said. "This isn't a drill you can run."_
|
| As someone who writes some very thorough unit tests... and also
| have had to have _mandatory training_... I find "this isn't a
| drill you can run" to be _very_ wrong.
| SoftTalker wrote:
| Southwest is a "discount" airline. They do many things to
| economize, i.e. no assigned seats, they only fly 737s so they
| don't need to certify pilots or mechanics on any other types,
| you can only book with them and not with Expedia etc.
|
| It would not surprise me that their back-office operations
| are likewise economized and some things are just not done
| because "they can never happen."
| thomasjudge wrote:
| Just as a note: they are about to issue a $458M dividend.
| They plan to spend $4-4.5B in 2023 on planes. How much are
| they spending on system modernization?
| namdnay wrote:
| > How much are they spending on system modernization
|
| A fortune, they only just finished an 8-year migration to
| Amadeus
| LarryMullins wrote:
| > _Southwest is a "discount" airline._
|
| They're also the "friendly airline", they easily have the
| most personable and friendly staff. I don't know what they
| do different, but Southwest employees treat me human and
| all the rest generally treat me like human trash. It's got
| to be a company culture thing, maybe connected to Southwest
| not having a first-class section.
|
| Usually I fly with Southwest whenever possible without
| thinking twice about it, but this outage and the outage
| last year are forcing me to reconsider. Better to deal with
| rude people than to have my flight delayed..
| dkarl wrote:
| Yep, the other airlines are in the business of selling
| "class" and "status," and it's part of their product
| differentiation strategy to treat you according to how
| much you pay.
| AlotOfReading wrote:
| All airlines economize. An airline that doesn't is a
| bankrupt airline because typical industry margins on flight
| are razor thin.
|
| Southwest isn't a particularly budget airline compared to
| modern budget carriers like spirit and ryanair that haven't
| copied the open boarding policy. I suspect the opportunity
| to upsell seats / luggage and have distinct classes
| outweighs the turnaround time costs of assigned seating.
| mason55 wrote:
| > _It would not surprise me that their back-office
| operations are likewise economized and some things are just
| not done because "they can never happen."_
|
| Meh, it doesn't even have to be "never". It just has to be
| cost multiplied by frequency is less than the cost to
| prepare.
|
| If they lose $100m every five years due to a system
| failure, and it would cost $30m/year to plan for those
| failures, they it's just cheaper to let it happen.
|
| And I don't mean this in a judgmental, Fight Club-car
| recall speech kind of way. It's just business reality. At
| some point every business has to decide that the cost of
| planning for something is higher than the cost of letting
| it happen.
| e_y_ wrote:
| What's the value of the reputation risk of a major, very
| high profile failure?
|
| Sometimes businesses end up on the wrong side of that
| bet. They see only the costs but not the benefits of
| preparedness (by the time it fails, there will probably
| be a different CEO in charge) and make a bad call.
| mason55 wrote:
| Of course, no argument there. Ideally when you make that
| kind of decision you take reputational risk into account,
| as well as, like, is this an existential risk?
|
| The airline industry feels like one where each year it's
| a different carrier who has some catastrophic scheduling
| failure. Today, everyone says they're never flying
| Southwest again. But if you fly semi-regularly then it
| won't take very long before you don't have any airlines
| left to fly on.
|
| For people who weren't affected, I doubt very many are
| even going to remember this. Personally, I remember that
| this kind of thing has happened recently with other
| carriers but I couldn't even tell you who.
|
| And people who were affected can mostly be bought off if
| you need to. Some vouchers & hotel reimbursement and it's
| just the cost of doing business.
|
| Plus, the airline industry has proven over and over that
| people are willing to put up with a lot when you have the
| cheapest prices.
|
| It's different from an industry that's built on
| reputation and trust. Like, a password manager, the only
| real thing you're selling is your reputation. Losing
| trust is a real existential threat. Security costs need
| to be in the bucket of either "yes, we will do it" or
| "it's so expensive that if we do it then we don't have a
| business anyway, so we'll skip it and pray."
| hattmall wrote:
| Scheduling won't ruin an airlines reputation. Crashing
| the planes is what ruins an airline. Southwest has only
| ever had two passenger deaths and one of those was an
| attempted hijacker beaten to death by other passengers.
| khuey wrote:
| Eh, to a first approximation the FAA won't let you crash
| the planes. It's been 13 years since there was a fatal
| plane crash on a US passenger airline.
| burnte wrote:
| "I find "this isn't a drill you can run" to be _very_ wrong"
|
| As a IT-VP/CIO, the statement of "there's no way to test it"
| is not acceptable.
| dstroot wrote:
| Then you are senior enough to know what " then-CEO now-
| chairman Gary Kelly" really meant was "I haven't funded our
| technology team well enough to have resources to test a
| scenario like this".
| mason55 wrote:
| Or "we decided that the cost to plan for this is so high
| that it's not even worth testing. If it happens then
| we're fucked anyway and we'll just eat it."
| hattmall wrote:
| You can drill the initial failure, but not really the
| cascading events. In something as large as a global airline
| you are dependant on 1000s of third parties actions and the
| weather. No simulated drill is going to be sufficient or
| realistic. The only way to really mitigate or plan for
| something like this is multiple layers of segregation so
| that events in one area have less or no impact on others.
| Then you could drill total failure in various segments.
| [deleted]
| ninkendo wrote:
| Testing reveals the presence of bugs, never their absence.
| With hindsight you can always feel smugly superior in
| saying "you should have tested for this", but there's an
| infinitude of things you _might_ need to test, and if you
| haven't encountered a failure you didn't test for, you're
| probably just lucky.
| justinator wrote:
| I'm very certain Chaos Engineering is known in the airline
| industry
| ninkendo wrote:
| It's funny that you use unit tests as an example of it being
| possible to run drills for this kind of thing. Unit tests are
| by their definition not the kind of thing that simulates this
| kind of failure. Perhaps you have a false sense of security
| about what you've really been testing?
| 0x445442 wrote:
| So in my attempts to get my family home I've discovered that
| rental car company web sites are atrocious.
|
| The direct company sites (not Expedia for example) do not give
| you the ability to search for vehicles at multiple locations
| within a mileage radius. You are forced to enter a single
| location for pickup. This is also true for drop off.
|
| From what I can tell, rental car companies are not setup to
| easily service a one way rentals, which is what most people need
| right now. I've managed to reserve a car for pickup in KC and
| drop off in Phoenix for tomorrow. The quote for the two days,
| over $1200. This isn't really a web site issue but it gives some
| context.
|
| Here's the baffling bit. I make a reservation this morning at
| Enterprise in a KC suburb and get back a confirmation number,
| cool. Five minutes later I get a call back that they don't have
| any cars available. I ask him why, because I got the confirmation
| number. He says the website isn't accurate at the moment. WTF? So
| when a car is rented and driven off the lot the database doesn't
| update? Or worse, the local dealer data is not synced to
| corporate in real time?
|
| This is a cascading mess. They don't even have availability in
| Joplin, MO who's is 3 hours from KC.
| bmitc wrote:
| Doesn't it blow our minds that the U.S. is supposedly the most
| advanced country in the world, and yet our transportation
| systems are designed worse than seems imaginable.
|
| On rental cars, I know it's a meme, but my god it continually
| surprises me how it seems the agents have to rewrite the
| mainframe just to hand me keys for a pre-arranged reservation.
| bradleyankrom wrote:
| For whatever it's worth, I've used National extensively for
| years (we have a corporate contract) for business and personal,
| and have generally had positive experiences. The web site has
| been clear when inventory didn't exist, haven't run into the
| app-reality inconsistency problems.
| cowmoo728 wrote:
| Car rentals are a total consumer disaster and have always been
| this way. It keeps being like this because generally you end up
| with _a_ car and that 's all people need. Special shout out to
| Sixt that promised me a volvo xc90 over the phone and gave me a
| cadillac midsize crossover instead when I showed up. The
| employee at the manhattan location told me they've never even
| seen a Sixt XC90 at that location.
| BMFX00 wrote:
| Just to give you a possible cheat code. Despite them saying you
| can't when booking...
|
| I've had great luck booking an enterprise rental car from an
| airport (this is key), and on the booking saying returning to
| the same airport.
|
| After receiving the car, calling corporate and requesting a
| different city drop off. The only requirement being that it was
| dropped off at another airport.
|
| Paid $75 a day for an SUV, from SF and dropped off in Bishop,
| CA. Reno was the same cost, as well but Bishop made sense. Did
| this a week ago. Worst case there's an additional cost but is
| it going to be $1,000 more... no way.
| giantrobot wrote:
| Not that this helps but car rental places are like
| trailer/truck rentals. A particular location will often just be
| a franchise and own their own vehicles. They'll also handle
| vehicles from other locations/corporate but they have their own
| little fleet. They charge one-way rentals based on the
| likelihood of that vehicle returning to them in some time frame
| and/or them getting a replacement vehicle while the one-way is
| rented.
|
| More generally they schedule based on their return schedules.
| If they have a car scheduled to return at noon and you set your
| pickup time to one, they assume they'll have that car and be
| able to rent it to you. If I don't drop it off at noon they
| can't assume I'll drop it off at 12:01 so they need to let you
| know it won't be available (since they can't predict my
| actions).
|
| This is all compounded by agents on the lot renting out cars
| under the radar or booking agents doing some customer service
| override. They are incentivized to rent cars, not rent to
| specific people. For them it doesn't matter if _your_ car is
| unavailable, it got rented out which made them money.
|
| Good luck getting a car and making your drive.
| typest wrote:
| I was also in STL when this happened. Had to drive 4 hours away
| to Evansville, IN in order to get a rental car. Just showed up
| here and they told me I was lucky to reserve it last night,
| they're now turning away walk ups at the counter.
| hrunt wrote:
| > Here's the baffling bit. I make a reservation this morning at
| Enterprise in a KC suburb and get back a confirmation number,
| cool. Five minutes later I get a call back that they don't have
| any cars available. I ask him why, because I got the
| confirmation number. He says the website isn't accurate at the
| moment. WTF? So when a car is rented and driven off the lot the
| database doesn't update? Or worse, the local dealer data is not
| synced to corporate in real time?
|
| Same thing happened to me. My wife even called the night before
| and talked to someone (not at the airport, though) and
| confirmed. The guy at the pickup had to tell 6 people at 6am
| that they were only honoring reservations made at least three
| days prior, and all of them had confirmed numbers.
|
| Likewise, no cars are available within a two hour drive. It
| looks like the first few days of problems sucked up any
| available inventory in the system.
| chasd00 wrote:
| I never thought the movie Planes, Trains, and Automobiles was
| a documentary.
| hk1337 wrote:
| They know how to take the reservation, they just don't know how
| to _hold_ the reservation.
| dsimmons wrote:
| Great Seinfeld bit :)
| SoftTalker wrote:
| Cars are rented out with the assumption that prior renters will
| return the cars on time. If that doesn't happen, you will be
| told that they don't have a car for you.
|
| The reservation system probably overbooks to some extent
| because a certain percentage of reserved cars are never picked
| up.
|
| Many rental outlets are franchise operations. It may say
| Enterprise or Budget on the sign but it's Joe's Car Rental LLC
| running the place. They may be slow at updating rentals and
| returns in the system. It's not all one homogenous company.
| lotsofpulp wrote:
| Enterprise is usually not a franchise:
|
| https://www.enterprise.com/en/global-franchise-
| opportunities...
|
| > Please note that we do not franchise our brands in Canada,
| the United States, the United Kingdom, Ireland, Spain, France
| or Germany.
| [deleted]
| kylehotchkiss wrote:
| I've found for one way rentals, going between two larger
| airport locations is the best way to keep the price reasonable
| (and probably pay some of the difference for Ubers)
| leviathant wrote:
| Car rental inventory is absurd, and renting a box truck is even
| worse. I've learned over the years that a "reservation" is a
| best case scenario. - nothing's real until money changes hands.
| There's also a lot of weird nuance around renting at an airport
| and renting at an in-town location. And even when you've turned
| down all the needless upsells and paid for your car, you have
| to be careful about further sleight of hand!
|
| Earlier this year, I reserved a mid-size vehicle for a few
| days' drive for a business trip, and after they charged my
| card, they told me they're bringing around a compact vehicle. I
| probably should have let it go, the difference in price wasn't
| that much (and was a work expense) but old habits die hard.
| "That's not what I paid for - either you can refund me the
| difference, or you can bring me a mid-size sedan."
|
| They said if I could wait 15 minutes, they'd bump me up from a
| Hyundai Accent to a larger Toyota, but it wasn't until I sat in
| the car that I learned that even as a new car, it was as bare-
| bones as they make them. No CarPlay/Android Auto, the in-car
| GPS was disabled, no cruise control - I'm sure it would have
| had manual windows if that were an option. The Hyundai would
| have been a better ride, I shouldn't have fussed.
|
| But yeah, U-Haul or Ryder or whatever, it's the stone ages
| there. Years ago, I drove halfway across the state to pick up a
| sixteen foot truck I'd reserved, and when I got there - "Yeah
| we don't have a sixteen foot truck, I don't know why it let you
| reserve that." A smaller truck wasn't an option for what we
| were moving. I ended up having to call around and hit some
| local place up.
| hrunt wrote:
| > Car rental inventory is absurd, and renting a box truck is
| even worse. I've learned over the years that a "reservation"
| is a best case scenario. - nothing's real until money changes
| hands.
|
| As Seinfeld said, they're good at taking the reservation, but
| not good at holding it.
| jjulius wrote:
| Penske is the winner there (edit: in my experience, YMMV). We
| moved from CA to OR in spring 2021. Got a res through UHaul
| for a box truck, and got a call 24 hours later basically
| telling me that they were massively overbooked and wouldn't
| be able to confirm a truck for me _until the morning we were
| supposed to pack the truck up and leave_. There 's not a
| snowball's chance in hell that I'm gonna plan for a move with
| that level of uncertainty, lol.
|
| Called Penske, because I'd always had a good business
| relationship with them via my job. I was told, "Oh, yeah, we
| don't do that shit. If you book a truck that day, it's gonna
| be here, guaranteed."
|
| Sure enough, Penske had it. We packed up the truck, I got on
| the road, and about 45 minutes later I got a call from UHaul
| telling me that I could pick up an available truck, but it
| was three hours away.
|
| I laughed loudly and just hung up.
| pc86 wrote:
| I've probably rented box trucks a dozen times in my life
| for various personal and professional endeavors, including
| a stint of regular quarterly rentals for several years, and
| two cross-country moves. At this point Penske gets my money
| no matter what; I don't even look elsewhere.
| kylehotchkiss wrote:
| No CarPlay? Every Avis rental car I get seems to have it.
| Which brand?
| optymizer wrote:
| I rented cars in Europe, Carribean, Israel and the US. Some
| things I've learned:
|
| * In the US, Hertz with corporate discounts and status is
| usually the cheapest and most convenient option. Walk to the
| car, pick your car from the President's Circle, drive off.
| Absolutely the best experience I've had with rentals.
|
| * Without status, it's a coin toss: Avis or Hertz, with Avis
| generally having worse service across the board.
|
| * In Europe, Avis was the worst. In Israel I think it's a
| franchise, it was all manual entries, long waiting lines. In
| Portugal I had to wait in line for 1.5 hours to pick up the
| car because I wasn't Preferred. Their Preferred booth was
| empty the entire time. All I had to do was validate my
| drivers license, and they still refused to accept me at
| Preferred because they print the preferred customers in the
| morning. Almost missed my flight.
|
| * Europcar in Israel - cheaper but I wasted an hour in line
| to _return_ the car.
|
| * In Aruba (I think), Avis doesn't tell you they run shuttles
| to the airport and you have to wait 30+ min for the shuttle
| to come around and pick you up.
|
| * I rented 26' trucks from U-Haul twice. Both times went
| well, though the first truck was very bouncy and the second
| truck's engine light was on, but otherwise I got from A to B
| with no issues.
|
| But this all pales in comparison to this car rental company
| in Moldova (Eastern Europe) that took $100 to book an SUV a
| month in advance, and then when I arrived they said "we don't
| have it". The only car they had was $50/day more expensive
| and instead of apologizing and giving me that car, despite
| being more expensive, they told me that my only option was to
| pay more to get this car. They were fixing the A/C outside
| while I was trying to explain to them that it was a
| ridiculous demand. A bunch of crooks. I walked away. Sixt was
| a better experience there, and cheaper than Avis, though
| 4rent.md is definitely the best option with great customer
| service.
| barrenko wrote:
| There are some pretty cool threads on the subject of rental car
| companies here on HN.
| Anechoic wrote:
| _From what I can tell, rental car companies are not setup to
| easily service a one way rentals,_
|
| Which isn't surprising because at some point, someone needs to
| bring that car back to it's home base.
| trollied wrote:
| Indeed. Lots of them are franchises that own their own stock.
| lotsofpulp wrote:
| I thought none of the 3 US car rental brand holding
| companies franchised. But it seems that Avis Budget and
| Hertz do franchise. Enterprise does not.
|
| I wonder if that is why I seem to prefer Enterprise. Not
| that I would touch Hertz with a 10ft pole anyway.
| 0x445442 wrote:
| And Alamo appears to just be a front company for
| Enterprise.
| lotsofpulp wrote:
| That is why I wrote "3 US car rental brand holding
| companies". They are all part of either Enterprise, Avis
| Budget, or Hertz.
|
| Which is also why they share the same offices,
| countertops, parking lots, and I assume cars too when you
| go to the rental place in an airport.
| [deleted]
| bluesroo wrote:
| My observation of rental car companies is that they have
| figured out how to run on absolute skeleton crews.
|
| I rented one earlier this month online. When I showed up they
| handed me a confirmation and told me to pick a car and drive,
| key was in the cup holder. They had a manned exit barricade to
| confirm the car you chose, but that was it. I'm not even sure
| they cared if I picked a car that I didn't reserve, the exit
| gate person scanned the car and reservation, so it probably
| would have just updated the reservation on the spot. I don't
| think it's even possible to predict with certainty what your
| inventory is going to be 2-3 days out with this sytem.
|
| Just 2 people on the rental side, although I'm sure they have a
| cleaning/ turnover crew.
| drewg123 wrote:
| Try a moving truck rental, and rent a van. They're set up for
| one-way, and often have stock when car rentals are sold out.
|
| Some of my ex-wife's colleagues returned from a conference in
| Denver back to the east coast in a u-haul van after all air
| travel was shut down on 9/11
| thomasjudge wrote:
| Good grief. Did they buy some bean bag chairs or something?
| jeffrallen wrote:
| You get what you pay for. I suspect, weather permitting, NetJets
| is still flying.
|
| I fly very rarely. One of the reasons is that I cannot afford to
| fly on carriers who charge the correct fare for sufficient
| quality that actually reliably gets you to your destination.
|
| For destinations on the local rail network, I can afford it.
| RedShift1 wrote:
| Is this also one of the companies that completely outsourced
| their IT?
| water554 wrote:
| Data point delta airlines IT is almost entirely offshore
| qwertyuiop_ wrote:
| Yup sweep it under the rug. SW outsourced all of their
| development and support to WITCH. The VPs who get condos in
| Bahamas in return will not ever mention that this is the result
| of a failure of crew scheduling software because they signed
| off on the outsourcing.
| jrochkind1 wrote:
| > Other issues that have exacerbated the airline's struggle to
| accommodate the holiday rush include problems with "connecting
| flight crews to their schedules," Perry said. That issue has made
| it difficult for employees to access crew scheduling services and
| get reassignments.
|
| Wait, what issue? Those words didn't actually say anything.
|
| Is this implying that it was a software failure with the
| scheduling software?
| francisofascii wrote:
| Yeah, the word "connecting" has a double meaning. Do they mean
| crews of connecting flights, or associating crews with their
| proper schedule.
| code_runner wrote:
| Sounds like it's doing more than implying it. The scheduling
| software seems to be pretty hosed at the moment
| jrochkind1 wrote:
| To me, "problems connecting flight crews to their schedules"
| does not clearly say it's a software issue, but maybe implies
| it. But you have other information that it is?
|
| Any info on the details? Like, I'd think this is what
| scheduling software is for...
| acedTrex wrote:
| Appears their scheduling software is just woefully
| inadequate and requires significant human intervention and
| it just gave up.
|
| https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1t
| n...
| epylar wrote:
| Pilots literally having to call an 800 number to get their
| schedule, not enough capacity on the 800 number.
| sys32768 wrote:
| Our local airport closed yesterday due to icy runways. Planes
| would circle a while then refuel 100s of miles away, then return
| to try again.
|
| We don't often see freezing rain here. It creates a hard, bubbly
| layer of ice that cannot be scraped off with the usual tool.
| Kon-Peki wrote:
| Get out the curling rocks, eh!
| mensetmanusman wrote:
| A software bug can destroy a company these days.
| AlbertCory wrote:
| From personal experience: I used to preferentially use SWA
| because they were great. Over the years, it seems to have become
| that _every single flight_ is significantly delayed. Now I
| preferentially don 't use them.
| el_benhameen wrote:
| I love not being nickel and dimed by them, and the companion
| pass makes them significantly cheaper for family travel, but I
| absolutely avoid holiday travel with them, especially winter
| holidays.
| faangiq wrote:
| Let's see the C suite asset clawbacks.
| DoingSomeThings wrote:
| Incredible to see that 14 months ago there was an almost
| duplicate outage. Clearly the operations teams are running too
| lean and don't have enough slack to handle predictable
| weather/sickness events. Seems like something you'd plan for.
| Even at the 98%+ uptime range
|
| https://news.ycombinator.com/item?id=28823774
| hinkley wrote:
| Could be lean, could be a bus number problem. They flock
| together but are separate issues.
| chasd00 wrote:
| There's no slack at an airline, it's basically a real time
| system. Imagine a bottling plant, if one part of the line
| malfunctions the rest of the line just keeps throwing bottles
| at the malfunctioning unit until the stop button is hit.
| There's no way to build in artificial delay to those kinds of
| systems.
|
| Even if you could do that at an airline the margins are so
| razor thin. I bet there's a dollar value assigned to every
| second a plane is not in the air.
| bronson wrote:
| This is not entirely true. Lots of airlines have had events
| where they need to ground all takeoffs for a couple hours
| while they put their systems back together. People don't die
| when airline operations go down so artificial delays are used
| occasionally. Lots of air travelers have experienced them.
|
| Yes, there's a dollar value attached to everything and
| everything is being optimized. Southwest was among the
| pioneers. See the story of the ten minute turn.
| howinteresting wrote:
| I think the way to do it is how the EU does it, where slack
| is built in by law (large amounts of compensation for delayed
| flights). You pay more for each flight in return for
| predictability, and the race to the bottom is prevented.
|
| Though I'm not sure it would have helped in this case.
| ghaff wrote:
| And Aer Lingus had a meltdown just last September because
| of network issues.
|
| https://simpleflying.com/aer-lingus-data-center-meltdown-
| com...
| howinteresting wrote:
| While that is definitely unfortunate, it looks like the
| system is working correctly. Aer Lingus is on the hook
| for millions of euros thanks to EU law, and then it goes
| after the IT provider to be made whole.
| noirbot wrote:
| I guess let's wait to see how it works out in the US? The
| US government seems to be saying they're going to try to
| do something about this, so it doesn't seem like SWA is
| going to pay nothing here.
|
| Actually, it appears that Southwest is already saying
| they'll reimburse tickets for canceled flights and
| potentially pay out for necessary hotels and alternate
| arrangements: https://www.southwest.com/html/air/travel-
| disruption
| howinteresting wrote:
| EU compensation for cancelled flights is all of that plus
| a monetary sum:
| https://europa.eu/youreurope/citizens/travel/passenger-
| right...
|
| edit: though there is an extraordinary circumstances
| exception below. I'm not sure if this would qualify as
| that given that it's a combination of an adverse weather
| event and a technical system collapse.
| dendrite9 wrote:
| There are things called accumulation systems for lines that
| serve as a buffer to make the line more resilient and able to
| function if there is a malfunction somewhere in the line. It
| took me a bit of time to think of the name but I have seen
| them on large production lines down to small brewery
| bottling/canning systems.
|
| I had a partner who worked on airline optimization several
| years ago, as I recall there were standby aircraft in some
| places that could be deployed to fix problems like a plane
| needing unexpected maintenance. Or even needing a single
| label required to be flightworthy. That Southwest doesn't use
| hubs likely makes it more difficult to recover from this kind
| of disruption before breaking.
|
| https://www.kinexcappers.com/accumulation-table/
| https://www.nerconconveyors.com/Nercon/Documents/White-
| Paper...
| kibwen wrote:
| _> Even if you could do that at an airline the margins are so
| razor thin. I bet there 's a dollar value assigned to every
| second a plane is not in the air._
|
| You either pay the predictable, ongoing cost or you pay the
| massive, unpredictable cost like they are currently. There's
| no free lunch. That is, unless you're an executive looking to
| boost short-term profits by eliminating "redundancy" and then
| peacing out before the whole thing collapses from your short-
| sighted, greedy ineptitude.
| pkulak wrote:
| Is this really a massive cost, though? What does it cost to
| release some statements to the media and maybe reimburse
| some hotel receipts in 6 months? Especially balanced
| against the fuel savings of grounding half your fleet for 6
| days. It's absolutely cheaper to run without enough staff
| and then just deal with the fallout every couple years. No
| one will remember this. Even the people effected will pick
| the cheapest flight next time, not matter the airline.
| noirbot wrote:
| That does bring up an interesting question - are they
| actually saving on fuel? I would have assumed, given the
| relatively planned-out nature of most airline operations,
| that fuel would be something contracted out in advance,
| and that they may have already bought the fuel whether
| they use it or not.
| pc86 wrote:
| Yes and no, they certainly buy fuel in bulk and several
| weeks/months out, but they're still not flying for a
| week, so their yearly fuel cost is going to be less than
| projected.
| dehrmann wrote:
| People don't like to hear this now, but airlines chose the
| right thing to optimize for. Air travel is rarely _that_
| important to have these redundancies, and they vote with
| their wallets every time they buy economy tickets. Given
| that safe is a baseline, most people value price over
| reliability.
| ThunderSizzle wrote:
| In my experience, every airline will screw you if your
| not their frequent flyer. Fly once a year? No one is
| going to waive a fee or refund you anything. Might as
| well go for cheapest price.
|
| My wife's experience shows SWA as an exception in that.
| HDThoreaun wrote:
| Airlines are too competing to have extra labor for
| situations like this. Everyone will book on the other
| airline that works 99% of the time and you'll be out of
| business before a catastrophe hits. "Just add more slack"
| is not an option that also allows these businesses to be
| profitable.
| chasd00 wrote:
| years ago i interviewed for, and was given an offer, to work on
| this very system. glad i didn't take it. I feel bad for the devs
| trying to figure out what's going on.
| bronson wrote:
| It's an absolute nightmare of fixed-field records and XML
| (Southwest uses Amadeus for scheduling but Sabre is just as
| bad; and AFAIK operations at all airlines are a chaotic hodge
| podge). Southwest performs all operations in Central Time
| because nobody's figured out how to reliably add time zones to
| all systems all at once. This means they have something like a
| 34 hour operational day, and every flight must take off and
| land in the same operational day.
|
| It's a satisfying job for the right personality type!
| cratermoon wrote:
| In some recent work I've been doing for an airline, I've
| worked with 5 different and incompatible ways to represent a
| passenger, depending on what part of the system you're
| working in.
| encoderer wrote:
| My god. This is terrible but also riveting. I wish there was
| a website devoted to collecting stories of how these internal
| systems really work. Airlines. IRS. credit agencies. Etc.
| jeffrallen wrote:
| It's called "The Daily WTF".
| awillen wrote:
| I can't for the life of me understand how they're allowed to say
| this is due to weather (and thus not provide the same kind of
| compensation they'd have to if it were "Southwest's fault").
| Sure, weather is the precipitating (pun intended, sorry) cause,
| but given that other airlines are almost all at <10%
| cancellations, clearly Southwest's total lack of operational
| robustness and competence are the real issue. The CEO himself
| emailed Southwest staff and explained that one big problem is
| their antiquated computer systems - that is unequivocally their
| fault (and very personally the CEO's fault, since it's very much
| his job to direct the sort of large scale financial investment to
| do something like overhaul their system).
|
| I hope that DOT follows through on their earlier statement that
| they're going to investigate this. The CEO of Southwest has
| ruined more Christmases than the Grinch. Failures like this
| should trigger incredibly severe consequences (tickets refunded
| at 4x cost, $20/day paid to any passenger whose luggage is lost
| until they get it, lost slots in airports) that make it
| absolutely irrational to operate in the manner Southwest is.
| fairity wrote:
| > Failures like this should trigger incredibly severe
| consequences (tickets refunded at 4x cost, $20/day paid to any
| passenger whose luggage is lost until they get it, lost slots
| in airports) that make it absolutely irrational to operate in
| the manner Southwest is
|
| No need to implement the penalties you described if you're
| simply trying to align incentives. Their market cap has fallen
| by $1b, primarily due to this incident.
| awillen wrote:
| Their market cap today isn't what matters - if people forget
| about this and continue to fly Southwest (which, let's be
| honest, many will), their share price will recover when they
| announce earnings.
|
| The reality is they save a lot of money by keeping virtually
| no slack in their system and not doing things like scaling up
| customer service during the holidays. If they lose some money
| in flight cancellations, it's still rational for them to
| operate like this. Plus the reality is they probably won't
| actually lose much - most people whose flights were cancelled
| will still end up taking Southwest because it's too expensive
| (or not possible) to book last minute tickets on other
| airlines. That means they just push them into the very end of
| Dec/early Jan when they have open capacity anyway.
| AlexTWithBeard wrote:
| > people forget about this and continue to fly Southwest
|
| For me it means that most customers would happily accept
| such a disaster once in a lifetime in exchange for, say,
| 10% ticket price reduction.
| bronson wrote:
| I think that's right. Customers have decided that maybe
| getting charged a couple full fares once every ten years
| is better than spending 10% extra on every ticket.
|
| Which makes sense. The airlines have decided this as
| well.
| awillen wrote:
| While the capitalist in me agrees with you that this is
| one the market can sort out (and believe me, I'm going to
| pay more to not fly Southwest moving forward, at least
| for 2023), I do think that this is just a case where
| people are bad at making that calculation. I suspect most
| people would pay 10% more for even a relatively small
| reduction in likelihood of this kind of thing happening
| at Christmas, given the potentially huge costs if it does
| (missed time with family, being stuck in an airport for
| days with children, lost payments for hotels and other
| activities, etc.). People are generally quite poor at
| properly factoring in low-probability, high-impact events
| into their decisions, and I think that's happening here.
|
| Now whether the government should step in to protect
| people from that bias is entirely another question. I
| would argue yes, but I can very much see the other side
| that would say to simply let the market sort it out.
| fairity wrote:
| > if people forget about this and continue to fly Southwest
| (which, let's be honest, many will), their share price will
| recover when they announce earnings.
|
| Well, if you believe this, go buy their stock (LUV). The
| market disagrees.
| chasd00 wrote:
| I very well may buy some LUV depending on where they end
| up over the next few days. Remember the Equifax security
| breech that should have destroyed the reputation of that
| company in 2017?
|
| Here's some historical stock prices
|
| 9/1/2017 $134.64 - everything's fine
|
| 9/15/2017 $87.99 - scandal breaks and hits the news
|
| 9/17/2018 $130.55 - one year post scandal
|
| people forget
| awillen wrote:
| Yeah, I actually did really well after the Deepwater
| Horizon buying BP stock at the bottom, since the amount
| of value it lost in market cap exceeded the largest fine
| in history plus the cost of all the damage and lost oil
| by something like an order of magnitude.
|
| That said, while in theory I actually think LUV is a
| decent trade right now, broader market conditions will
| keep me from messing with it.
| supernova87a wrote:
| "Weather" covers a lot of sins. You will find that a lot of
| what gets called (and allowed to be called by the FAA/DOT)
| "weather" is, as you point out, not really weather, but
| operational choices.
|
| When a storm hits a city, yes, the first plane that had to wait
| to take off from the airport can be said to be delayed by
| weather. Every other plane waiting in line behind that plane,
| delayed, or because they weren't able to get to that city from
| other destinations because they run an overly tight schedule,
| is due to operational choices by the airline.
|
| Some airlines have hubs in places where snow is handled ok.
| Some don't. Others have them in places that have frequent
| thunderstorms. Some airlines operate in Hawaii and never have
| delays.
|
| JetBlue used to fly A321s across the country and in the winter,
| strong winds would force them to stop in Kansas to refuel.
| That's "weather" but also the airline's choices about how to
| operate.
|
| I don't think you'll ever find the FAA/DOT is going to "root
| cause" what weather means to airlines to be able to blame them
| for operational / strategic choices. It would be like the
| police writing up your car accident report and saying that the
| reason you had a fender bender was because you chose to live so
| far away from work.
|
| Thus, choose your airlines and roll your dice accordingly for
| when you want to get where you want to be.
| m-ee wrote:
| Another example that bit me, flying from east coast to west
| coast. Storm in the Midwest we can easily avoid but that
| requires changing the route. Time to change the route with
| ATC + slightly longer flight time made the pilot time out,
| flight is cancelled and it's the last of the night. AA
| refused to cover anyone's accommodations because the problem
| was "weather"
| awillen wrote:
| I generally agree, but on the other hand I think the
| situation here presents a clear case where you could draw the
| line - if your airline is canceling at a rate that's, let's
| say, double the national average, you no longer get to claim
| weather.
|
| It's difficult to punish the kind of individual cases you're
| describing, but from a game theory perspective that just
| means that in situations like today's, you bring down the
| hammer in incredibly punitive fashion in order to make a
| single systemwide failure like this so costly that it's a no-
| brainer to upgrade software and keep slack in the system
| (particularly at high-traffic, high-importance times like the
| holidays).
| [deleted]
| bodhiandphysics wrote:
| One thing to note is that airline crew scheduling is np-hard
| (trivially you can reduce Hamiltonian cycle to it). Ak
| interesting semi-empiricle fact about np-complete problems. Take
| a decision problem to determine if there is a schedule with k
| crews. Now there is some k' that is the minimum such k. For many
| Np problem problems the difficulty of the decision problem is
| related to the ratio of k to k'. (Not all decision problems are
| like this... those that are are on the complexity class APX. As a
| quick rule of thumb however this works) In terms of airline
| scheduling this means that if you more crews than you strictly
| need, your algorithm is probably going to successfully schedule
| them. When the number of crews drops below some critical value...
| all hell breaks loose. Thinking about this in complexity terms is
| useful I think. He issue isn't just bad software... with a small
| enough number of crews, all software is bad, unless P = NP
| maxerickson wrote:
| I wonder if the difficulty at this point is algorithmic, or if
| they even have the data to make "reasonable" decisions, never
| mind optimizing them.
|
| Like what do they do with the bags that are sitting on the
| plane for cancelled flights? Store them where they are? Figure
| out their origin and start queuing them to be sent there? Try
| to get information from the passenger about where to send it?
|
| And then with planes and crews, do they even have "potential"
| flights where they would be able to reliably seat a worthwhile
| number of passengers?
| SoftTalker wrote:
| > Like what do they do with the bags that are sitting on the
| plane for cancelled flights?
|
| Eventually you send them back to the passenger's home
| address. I've had lost luggage before that turned up on my
| front porch a week later, some courier service had dropped it
| off.
| chasd00 wrote:
| yeah it sounds like recovering from this requires standing up
| a brand new organization in real time. I hope someone is
| watching the trouble ticket system, imagine that going
| down...
| [deleted]
| bodhiandphysics wrote:
| I'm sure they know exactly where every plane and every flight
| crew is at every moment. My point about algorithmic
| complexity is that even with perfect knowledge, fixing this
| is a computationally difficult problem. The idea was to show
| how the properties of algorithms that perform combinatorial
| optimization actually matter!
| enneff wrote:
| > I'm sure they know exactly where every plane and every
| flight crew is at every moment.
|
| The story being circulated by Sw employees is that this is
| the failure. The system needs to be manually updated when
| flights are cancelled or changed, and in this situation
| with a cascading failure they have completely failed to
| keep that data accurate. SW employees have said they have
| spent 24 hours on hold trying to inform the company where
| they are.
|
| I don't think this has anything to do with algorithmic
| complexity.
| anon84873628 wrote:
| The statements from employees indicate that SWA does _not_
| know where flight crews are at every moment. When a flight
| is cancelled the downstream information is not updated
| accordingly. Crew have to call a phone line and speak to
| another human to update their information and assignments.
| With the large scale cancellations due to weather, that
| system was overloaded and fully collapsed. SWA had no
| choice but to cancel the majority of their flights while
| they sort out where the crews are at and how to schedule
| them next. The debacle is a direct result of lack of
| investment in modern software systems.
| chasd00 wrote:
| heh i wonder if they've asked everyone to use their
| phones to share their location with their immediate
| supervisor. Then someone setup a spreadsheet in
| sharepoint and the supervisors are going at it.
| someguydave wrote:
| Maybe there are many crews who are not able to fly because they
| hit the FAA hour limit? If so, then this cascade of failure was
| predictable.
| noirbot wrote:
| Predictable, assuming they knew which crews in which cities
| were close to their limit, which it seems like they don't now
| that the system is down, which is most of the problem.
| rossdavidh wrote:
| While this would be an excellent justification for not being
| able to schedule everything, it's not much of a justification
| for not being able to schedule 2/3 of your flights on a day
| with clear weather...
| hagmonk wrote:
| I don't get to exercise my dusty algorithms knowledge nearly
| enough to follow this nearly as well as I'd like. Are we
| talking about the bin packing problem and approximation ratios?
| So, is the intuition here that when the ratio of bins to
| objects goes up, the worst-case performance for the algorithm
| goes down into the toilet?
| raverbashing wrote:
| Yes, theoretically it is NP, if you approach it naively. In
| practice it is easier to optimize for sub-problems
|
| In practice, crews have their home base, their corresponding
| qualifications (though for WN they probably only have one or
| two _types_ )
|
| And you can bet you can start cancelling flights once you don't
| have enough crew. Problem is, your crew might have gone over
| their allowed time and them (and your plane) are in Smallville,
| OH and there are passengers in Chicago waiting to fly.
| tgv wrote:
| Also: you can work with sub-optimal solutions. Decent
| approximations exist for many classes of certain problems.
| Even running simulated annealing or genetic algorithm will
| get you close to the optimum. I'm sure such a large airliner
| has a staff with sufficient knowledge of optimization.
| bodhiandphysics wrote:
| I do simulated annealing for a living. Simulated annealing
| on graphs works great when a) the diameter of the graph is
| small (which for airlines corresponds to hubbie hub and
| spoke models) and b). when you can accept relatively
| approximate solutions. The issue it seems here is that
| southwest doesn't have enough pilots. Algorithmically, that
| means they can't accept approximate solutions!
| raverbashing wrote:
| > Algorithmically, that means they can't accept
| approximate solutions!
|
| Well, of course they can
|
| Not enough pilots make the general problem infeasible, so
| you have to make do what you have
|
| In practice this means a) flying the schedule as best as
| you can b) get people/crew/planes back to hubs c)
| prioritize based on displaced people/cost/other variables
| bodhiandphysics wrote:
| You have to be careful what you mean by approximate. In
| this context, it means a schedule that requires more than
| the optimal number of pilots. Another way of meaning
| approximate is canceling routes till you can find a
| schedule that fits the pilots you have. That's actually a
| harder problem, and is not CS means by an approximation
| algorithm. Obviously that's what SW has to do, but part
| of the reason why things are so disastrous is that this
| problem is quite difficult (in terms of algorithmic
| difficulty)
| bodhiandphysics wrote:
| "Optimizing for subproblems" itself has issues. For instance
| with a single hub (a complete hub and spoke model), the
| problem is algorithmically easy. But a) flight distance is
| much higher, and b) you run the risk of a single weather
| disaster taking out your entire network, and c) lots of other
| problems. My point was to suggest that its useful to
| understand what's going on in terms of algorithmic
| complexity. Southwest, because of decisions that they made I
| should add, is facing an extremely difficult algorithmic
| problem, and some of the issues of what's going on can be
| understood by thinking out those problems. I.E. your
| algorithms class actually matters
| ProjectArcturis wrote:
| I have always really wondered how airline scheduling software
| works. Despite being pretty good with algorithms, I just have _no
| idea_ how you 'd make a system that's robust to weather and
| mechanical delays.
| twawaaay wrote:
| I don't have experience with airline scheduling but I have
| experience with software in large financial corporations like
| banks. The situation with banks (at least the ones I worked
| with) is that there is a huge amount of software maintained
| mostly by mediocre to bad teams. These teams fail a lot, the
| software fails a lot, and yet everything seems to keep going.
|
| It is not about code or algorithm quality, it is about
| procedural side of things -- how the organisation is
| "programmed" to respond to failures. I use the word
| "programmed" in a very broad sense -- for me setting up a paper
| checklist and being able to rely on people to follow it is the
| same as programming.
|
| I suspect the main difference between airlines and banks is
| that banks can afford to throw money on the problem and just
| get things done regardless of how inefficiently.
|
| Airlines are in the much worse position -- they _were_ able to
| afford being inefficient and throwing money at the problem in
| the past but can 't do it anymore. They work with old, outdated
| software that wasn't built with efficiency in mind but now
| don't have funds to change it and are forced to maintain what
| they have. This may also be the answer to why sometimes they
| just don't have capacity to react to problem and let it cascade
| to bring everything to a halt.
| blevin wrote:
| Instead of framing this as saying they "don't have" the funds
| and were "forced" into inaction, another framing that the
| board of directors must consider is that current SWA
| leadership failed in their responsibility to recognize,
| prioritize, and manage the actual needs of their decades old
| business.
| twawaaay wrote:
| I am unable to speak for the management because I have no
| knowledge of their particular situation or extensive
| experience in the matter.
|
| I can only speak about the forces that act on development
| teams and what could most likely in my mind explain the
| current situation.
| htag wrote:
| One important component is slack. Every airline at every
| airport should have a certain number of crews and airplanes
| capable of providing service in place of a delayed flight.
| Running on maximum efficiency for airplanes and staff means
| unexpected delays will cause cascading failures. Weather can be
| forecasted, and additional crews can be routed to replace
| probable future cancelled flights. Temporary staff and
| increased hours can be utilized for peak demand seasons. We saw
| similar problems with manufacturing failures when the supply
| chain became unreliable because of a lack of slack. This type
| of slack can be seen as an inefficiency and costs money, so
| it's unsurprising to see budget airlines struggling.
|
| Another important component is disaster recovery. How quickly
| can the system recover from missed flights? What is the game
| plan for dealing with crews/airplanes that are out of place.
| How will they return to normal operations? Often times having a
| play book everyone is working from can lead to faster
| recoveries than dealing with each individual crisis as it
| happens, often with either too much micromanagement from
| leadership or too little coordination between departments. The
| play book generates a conciseness before the system is
| stressed.
| RC_ITR wrote:
| > Every airline at every airport should have a certain number
| of crews and airplanes capable of providing service in place
| of a delayed flight.
|
| Good luck finding pilots to be "on-call" to fly anywhere in
| the world (and most commonly to small US cities) on a moments
| notice, with a jump seat return flight as their way home
| (after a night in a small city hotel).
| apelapan wrote:
| You'll find that almost all airlines keep staff on call in
| various places and with various reporting times, because
| already at very small scale, you'll have some crew not
| making it to work for whatever reason all the time.
|
| Maintaining right sized and right placed operational
| buffers is an entire sub-category of within airline
| scheduling software/consultancy.
|
| Those buffers will never cover a major disaster of course.
| They should let you hit your on-time and cancellation
| targets at smallest possible cost, though.
| sokoloff wrote:
| > Every airline at every airport should have a certain number
| of crews and airplanes capable of providing service in place
| of a delayed flight.
|
| Airline pax are probably not willing to pay for spare standby
| aircraft and flight and cabin crews at every airport every
| airline operates from.
|
| Southwest's original low-cost carrier business innovation was
| to run an all-737 fleet and make business-wide efforts to
| optimize for fast ground-turns, in order to get more flights
| out of each aircraft.
| bronson wrote:
| Certainly not, that would destroy profit and
| competitiveness. Spare airplanes are mostly in for non-
| essential maintenance, and spare crews can be called up in
| an hour or two. That's good enough. Catastrophic outages
| every few years still cost less than building decent
| redundancy into all operations.
|
| This is partly because airlines are still externalizing a
| good portion of the cost onto their customers, who need to
| rebook at short-term pricing. I'd love to see legislation
| to address this loophole.
| linuxftw wrote:
| I don't pay for extra standby aircraft, but additional
| flight availability is why I pick one airline over another.
| If you choose to fly Spirit, and your flight is canceled or
| delayed for any reason, you might not make it to your
| destination for days. With a major carrier, you'll simply
| be rebooked on the next flight.
|
| Southwest used to be a budget-friendly airline with decent
| service. Now they're priced as much or more than the other
| major carriers with the added friction of having to book
| search flights only on their site.
| ryandrake wrote:
| Unpopular opinion but airline tickets are way too cheap for
| what they are doing. My last trip to Vegas, the Uber ride
| to the airport was more expensive than the airline ticket.
| My Uber money went toward 1. The driver's labor, 2. The car
| and gas, and 3. Uber's (mostly engineering) overhead.
| That's it. And it was like $150! My airline ticket pays for
| pilots with decades of training, dozens of trained
| professionals and support agents, baggage handling,
| security, airport operations, sometimes meal service and
| entertainment, not to mention the wizardry of launching me
| 30kft into the air so I can get to another state in an
| hour. All that for $99?
| jdminhbg wrote:
| > 1. The driver's labor, 2. The car and gas, and 3.
| Uber's (mostly engineering) overhead.
|
| 4. Taxes/fees set by the city and the airport
|
| > My airline ticket pays for ... All that for $99?
|
| Well, if you'd split the Uber with a hundred passengers
| or so, it would have been a lot less than $149.
| sokoloff wrote:
| Plus left at a time and to a destination that you didn't
| get to specify.
|
| Compare the Uber (personalized, on-demand) transport to a
| private airplane more than a bus-in-the-sky.
| noirbot wrote:
| For comparison, a Greyhound bus on the same route I was
| about to take a Southwest flight on was about $160 and
| took 36 hours with 2 transfers compared to $250 on
| Southwest with one transfer and 5 hours total travel
| time.
|
| The costs are fairly equivalent, oddly enough.
| eightysixfour wrote:
| I know a bit about this with airlines - there's a lot of
| thinking that goes in to making the system appropriately robust
| but just as much, if not more investment, in optimizing
| recovery for minimal damage as well. They have optimizers that
| figure out the minimal damage from a plane being taken out of
| schedule, and airport halting flights due to weather, or
| whatever the issue may be. What's cool is the math that goes
| into "minimal damage" with regards to passengers, crews, bags,
| etc.
|
| You can poke around at the website for SlickOR
| (https://www.slickor.com/) to get an idea of the surface level
| work that goes into this.
| ttcbj wrote:
| I worked on the medical resident scheduling problem for a
| while, and there is a giant body of work on all kinds of staff
| scheduling problems going back to the 1960s at least.
|
| The two classes of solutions that I considered where
| optimization solvers (see Gurobi Optimization for example), and
| meta-heuristics (see the book Metahueristics: From Design to
| Implementation). If I remember correctly, the people at Gurobi
| started at a previous company which was spun out of an airline,
| but I might be confused. All the algorithms in both classes of
| solutions are so nuanced that it can take years to begin to
| grasp how their strengths and weaknesses interact with your
| particular scheduling challenge, and how the way you formulate
| the problem interacts with the ability of the algorithm to
| solve it.
|
| All that said, the real problem for me was a human one: If you
| produce a viable schedule X, the organization involved will
| always want to alter the rules to stretch the available
| resources to cover more, and simultaneously all the schedule
| staff will want more flexibility and nuance in expressing their
| preferences. You, as the author of scheduling software, are
| caught between them. Neither side is ever happy with the
| result.
|
| I occasionally daydream about revisiting resident scheduling (I
| don't recommend it, the people who use your software leave
| every year, are not business oriented, and don't understand the
| complexity of the task until they've tried it on their own
| their first and only attempt). If I did, I would focus less on
| algorithms, and more on incentives to reconcile the tension
| between the organization, which wants to cover the most shifts
| with the fewest people at the cost of flexibility and
| preferences, and the staff, who want more flexibility and more
| preferences satisfied. I think that is the core problem at a
| business level.
| bombcar wrote:
| The easiest way to solve the tension is probably to add
| additional money into the mix - the hard to fill shifts get
| paid a bonus, etc. someone would figure out how to game it of
| course but you already have this somewhat when overnights pay
| more.
| linuxftw wrote:
| I was thinking similarly. Have people bid on shifts.
| thenewwazoo wrote:
| Pilots do precisely this already.
| pc86 wrote:
| Sort of. They bid based on seniority. So if you've been
| there forever you get the cushy flight that pays a ton.
| If you just joined to get the worst shift nobody wants
| (because it's the last one left).
| jasonpeacock wrote:
| The field of science (math) that studies this (and applies it)
| is called "Operations Research"[0] and it's about optimization
| & planning. About 30yrs ago they started applying it to airline
| scheduling, here's a few random papers I found:
|
| https://www.researchgate.net/publication/245236750_Airline_S...
|
| https://www.sciencedirect.com/science/article/abs/pii/030504...
|
| [0] https://en.wikipedia.org/wiki/Operations_research
| castratikron wrote:
| Sounds like they don't have much of an idea either.
| uptownfunk wrote:
| They need a new CTO, there is no reason any of this could not
| have been prevented by a solid engineering design. If we can have
| autonomous vehicles, rockets that can be reused, social media
| like tiktok, insta, snap, Twitter then no reason we can't have
| mission critical systems to support southwest (or any airline for
| that matter) solved by technology
| eightysixfour wrote:
| The comparisons you chose are terrible. We don't have
| widespread autonomous vehicles. Rockets are complicated but
| primarily constrained by knowable physics problems. Social
| media sites are entirely digital systems that do not need to
| deal with the complexity of the real world and, where they do,
| like networking connections, they're built on abstractions to
| manage that complexity.
|
| Logistics problems, especially the logistics of moving humans,
| are both complicated and complex. You are talking about systems
| which must interact with manual, human-in-the-loop processes
| every step of the way. The feedback loops between the digital
| systems and physical systems are often loose and very costly to
| tighten (regulations, union rules, passenger behaviors) and the
| underlying systems that all participants must interact with
| were some of the very first widespread digital systems.
|
| Today's problem was probably preventable, but don't
| underestimate how hard the problems become as soon as a system
| begins to depend on humans and the real world.
| bob1029 wrote:
| > there is no reason any of this could not have been prevented
| by a solid engineering design
|
| Not unless this new CTO's roadmap includes development of a
| time machine. Engineering design alone will not prevent this
| kind of outcome.
| chasd00 wrote:
| I've made this same comment in other replies but none of those
| examples present the kind of labor issues an airline has. If
| your essential workers don't show up / quit and operations
| grind to a halt then no amount of technology is going to fix
| that
| [deleted]
| htag wrote:
| Is this a problem with the weather, with the scheduling software,
| or with the recent shrink then expand of airline demand? Either
| way, I don't think I'll book a flight with SW for a long, long
| time.
| alistairSH wrote:
| Are there any airlines that haven't had similar problems over
| the last 5-10 years? Seems like they've all had issues. And if
| they haven't, it's just a matter of time.
| hrunt wrote:
| I am affected by this. We got halfway into our flight only to
| find our next leg was cancelled. SWA will not (cannot?) rebook
| anyone until the 31st. Our return flight was going to be Sunday,
| so we rebooked from our halfway stop back home on Sunday.
|
| Here are some crazy things I have encountered.
|
| Rental cars in our city are sold out. Same for cities within two
| hours drive. The websites will accept reservations, but when you
| show up, they tell you they have no cars. Because it's the
| holidays, busses and trains are booked and any flights on other
| airlines are crazy expensive ($1500 one way). When you are stuck
| in a city, you are probably truly stuck there.
|
| Your only hope of dealing with SWA is waiting in line at check-in
| or a gate. The phones don't work. Online chat doesn't work. The
| lines are long and slow.
|
| Luggage is hit or miss. If your bag was pulled off a plane, you
| might find it in baggage claim, but most bags are on a plane or
| on the tarmac. SWA told us it may be 30 days until we get our
| luggage. They won't pull bags for people, and the agents that we
| spoke with acknowledged and felt for people who may have had
| medicine in them.
|
| The workers are as befuddled as the passengers. They have been
| very nice and as helpful as they can be, but their phones haven't
| been working and their computer systems have been slow.
|
| On Twitter, someone posted a video of the announcement at Houston
| Hobby about no flights until the 31st and people keeping their
| receipts for hotels, etc. They said the same thing at our
| airport.
|
| People in the airport are so mad. It's unfortunate because it's
| not constructive. But tempers are flaring, and frustrated
| passengers who finally get to talk to an agent end up slowing
| things down because they spend a lot of time trying to hear
| something they're not going to hear.
|
| Ultimately, this is an operations failure. Companies talk a lot
| about accountability, but the typical way you hold people
| accountable is by replacing them with more capable people. It
| will be very interesting to see if any executives leave SWA over
| this. If not, I would say that no one was held accountable.
|
| To close, my family and I are fine. This is but a minor speed
| bump in life. No one is dying, and we will see how SWA takes care
| of the extra expenditures. Some people aren't so lucky. They have
| meds in bags, or finances that don't allow them to spend multiple
| nights in a hotel and get Uber trips for a few days. Hopefully,
| SWA takes care of them, too.
| slater- wrote:
| have you checked to make sure you're not just stuck in the plot
| of a Home Alone movie?
| imchillyb wrote:
| Every airline, pre-boarding, informs passengers to place any
| and all necessary medication in their carry on bag.
|
| This has been procedure for the last 40 years.
|
| There is no excuse that is feasable or plausible for
| 'forgetting' important medications in the checked baggage.
|
| This is a reading comprehension problem and not an airline
| issue.
| elAhmo wrote:
| I had an international flight every month on average in 2022,
| across Europe, and I was never explicitly informed about this
| by airline or any of the friends and coworkers.
| TomVDB wrote:
| I have a million miles on United alone, and many more on
| others as well: I've never once heard such a message either.
| HDThoreaun wrote:
| I've never heard this flying hundreds of times
| 3-cheese-sundae wrote:
| No excuse, none? Not even the harried passengers at the
| terminal suddenly being told they need to check their carry-
| ons due to lack of overhead bin space, right before they're
| about to board?
|
| You have never forgotten to pack something, or perhaps
| forgotten an important detail in a stressful situation?
| vincent___ wrote:
| At least it reduces the impact of human activity on global
| warming. We can always talk about optimization, database,
| companies... The responsibility of each individual is engaged.
| 5400 flights, privileges of rich countries.
| supernova87a wrote:
| As I understand it, Southwest is particularly badly affected by
| such issues because they run a lot of flights that are not in a
| hub-spoke model, but rather serial flights one non-hub city to
| the next and next (like eventually coming back in a loop). You
| can see this by going to Flightaware.com for example, and
| following back a flight's previous destinations. See for example
| https://flightaware.com/live/flight/SWA1092/history/20221223...
| and "track inbound plane" a couple times.
|
| They jump around the country, much less frequently going back to
| a hub as other airlines do. That means that the planes and crews
| have a relatively harder time recovering from system-wide
| disasters because they don't have as part of normal operations as
| much ability to centralize or pool resources and get
| people/planes reorganized. (everyone go back to base, consolidate
| passengers, crew, planes and redeploy them and sort things out in
| one place)
|
| Unfortunate, but that's their model. Good for some purposes, not
| so good for others. Maybe it's them being quirky and an active
| choice. I mean, up until a few years ago they did not fly to
| Hawaii because their scheduling system / people / processes did
| not want to have redeye flights.
| chasd00 wrote:
| > but rather serial flights one city to the next
|
| i would imagine that's especially vulnerable to disruption as
| any delay/issue is magnified throughout the rest of the
| flights.
| pookha wrote:
| They've operated without the hub-and-spoke model years but they
| haven't had to operate with over 8% of their staff leaving.
| They're understaffed. It was a major issue with Southwest all
| throughout 2022 and it got brought up on their earnings call
| with investors. They're a budget airline and they can't afford
| to take that kind of staffing hit.
| twblalock wrote:
| Other airlines have staffing issues too.
|
| The bottom line here is that the hub-and-spoke model is more
| resilient than the point-to-point model.
| miguelazo wrote:
| That is definitely not the case. The real issue is that
| even the staff that did show up didn't know where to go.
| There were employees lost for hours at Denver because the
| call-in scheduling system went down. Some employees hit
| their limits for work time before they could even obtain
| their assignments.
| spacemadness wrote:
| Not having staff didn't seem to stop them overselling what
| they can handle, however. They took a risk on stretching as
| far as they could go in an ideal environment and here we are.
| kulahan wrote:
| This is probably tied to how deeply ingrained the
| overselling habit is in the airline industry in general.
| They're legally protected when doing this, and it's why
| that doctor got dragged off that flight.
| Waterluvian wrote:
| I don't fly much but I noticed things like "the same flight
| number takes off at the same time each day and is always the
| same plane as a different flight number coming the other
| direction."
|
| It must really help all the employees with routine and
| consistency even if it's not optimal.
| dehrmann wrote:
| Most airlines have schedules that are consistent day-to-day.
| It's the efficiency vs. resiliency tradeoff that's
| interesting. I'd probably summarize it as "don't fly
| Southwest in the winter."
|
| That said, I flew Southwest from SJC to LAS for CES one year,
| connecting in SAN. Weather wasn't great, and they'd put you
| on the next available flight with an empty seat. They were
| even able to shuffle people without going up to the podium.
| Legacy carriers would have drug their feet, there'd be a
| line, and they'd charge for the privilege of changing
| flights.
| m463 wrote:
| probably helps since they don't have assigned seats
| (afaict)
| curiousllama wrote:
| > don't fly Southwest in the winter.
|
| I'd look at it the other way around: cancelling is so
| annoying for them that they're often the last ones to do it
| (barring catastrophic collapse, of course).
|
| When I was traveling weekly out of Chicago, I always made
| sure to bring my Southwest credit card, just in case.
| Southwest sucks, but it gets you home.
| chrisbolt wrote:
| United proactively rebooks you on a new flight if you will
| misconnect, gives you options in the app, and issues
| waivers that let you avoid weather by rebooking your own
| flight (often waiving fare differences as well). Haven't
| flown Delta or American as much, but at least United's tech
| is a bit more modern.
| makestuff wrote:
| Yeah delta will do this as well. The app will let you
| pick any flight that day for free if you do not like the
| one it auto rebooked you on. I usually just take the one
| it gives me, but if it tries to route me through DTW or
| MSP with bad weather in the winter I will try to find one
| that goes through ATL instead.
| iudqnolq wrote:
| Delta offers you a buggy website to rebook for no fee. I
| changed to a flight that was about $1,000 more expensive
| a few days ago. (note that the significant price change
| is an edge case caused by travelling on Christmas being
| particularly undesirable)
| DoingSomeThings wrote:
| Someone below linked this company as an example of
| resiliency tools used in airfare. Interestingly, there's
| a testimonial from United on one of the front page
| videos.
|
| Maybe not 1:1 for what you're describing, but it
| solution/reason, but does seem like a possible sign that
| they're investing in proactive tools.
|
| https://www.slickor.com/
| code_runner wrote:
| When I was delayed and flying delta they continually
| rescheduled me until I boarded one of the flights.
|
| It was super convenient even if I was fuming over the
| multiple hours delay.
| marcosdumay wrote:
| Pilots don't get into work on the morning, stay there for an
| entire cycle, and get back home in the night.
|
| The planes are scheduled that way, but the people won't stay
| for the entire plane's cycle.
| Bonooru wrote:
| Optimal isn't the right way to think about it. It's a
| tradeoff. Hub-and-spoke is usually better at getting you to
| your destination in less absolute time given the same number
| of total flights since you can have more frequent "shuttle
| flights" that travel to the hub, exchange goods/passengers,
| and shuttle back. Point-to-point on the other hand, is better
| for minimizing travel time since you go directly to the
| destination.
| maxerickson wrote:
| I wonder if this event will end up being a demonstration that
| they aren't sophisticated enough to use their operational
| model. I would think planning decisions would at least try to
| account for disruptions and recovery time.
|
| I see lots of people who are at least quite a bit less likely
| to use them in the future (and they are still in the middle of
| trying to fix it).
| loeg wrote:
| It also might be fine if they only have to deal with this
| kind of event once every few years but it lowers their costs
| substantially the rest of the time. I wouldn't love it as a
| customer, but who knows.
| spookthesunset wrote:
| > I see lots of people who are at least quite a bit less
| likely to use them in the future (and they are still in the
| middle of trying to fix it).
|
| Meh, everybody always says that. In six months when this is a
| distant memory... it will be business as usual.
| makestuff wrote:
| Yeah airline travelers are price conscious and it is a race
| to the bottom. If they offer some crazy sale or cheaper
| fares people will book it. Just look at Frontier/Spirit.
| They consistently get horrible reviews but people deal with
| it for a $50 flight.
| tshaddox wrote:
| People often try to say "airline travelers are price
| conscious" as if there are several options in the same
| price range and travelers will accept any reduction of
| quality or service to save a nickel (I'm not claiming
| you're suggesting this). But in my experience with US
| domestic flights the options are basically one "cheap"
| decently tolerable itinerary, a few slightly cheaper
| itineraries that are like twice as long in total
| duration, and then a couple of slightly better
| itineraries with better amenities that literally cost
| like twice as much or more.
|
| I just laugh at the upset attempts when you go to check
| in online: "get priority boarding and 2 inches of legroom
| for only an extra 50% on top of the ticket price." I
| really don't see much evidence that there was actually a
| race to the bottom. And I certainly won't blame consumer
| preferences when I don't see any options for _slightly_
| better service for _slightly_ more money.
| sokoloff wrote:
| That depends on your city pair.
|
| Boston to Las Vegas, Orlando, or San Francisco, I've got
| a wide variety of choices, 2-4 carriers flying more than
| that non-stops per day.
|
| Flying from Des Moines to Presque Isle, Maine, I have
| only a bunch of 2 and 3 stops on United.
| mauvehaus wrote:
| Boston is such a weird airport I'm not sure it's worth
| bringing up except as perhaps an exception that proves
| the rule.
|
| BOS has the "advantage" of serving a fairly large
| population while also not being big enough to be a real
| hub for anyone[0], while being simultaneously big enough
| to have service from nearly everyone.
|
| Unlike a lot of airports smaller or serving fewer people
| than BOS (and some of comparable size), you can get from
| BOS to a whole mess of hubs.
|
| A few select routes (BOS to SFO as noted) are incredibly
| well-served because of the volume of lucrative business
| travel between the two and the fact that a whole mess of
| airlines already serve both airports.
|
| [0] No, JetBlue doesn't count. Boston is as much a hub
| for them as CLE[1] was for Continental. I.e. a second
| class hub at best.
|
| [1] CLE by comparison only really serves Cleveland.
| Columbus, Dayton, Cinci, Indy, Pittsburgh and probably a
| few others from a similar radius BOS draws from all have
| decent(ish) airports. All of those have basically the
| same problem as CLE or are worse in some way. I've flown
| through or into and out of all of them.
| jlmorton wrote:
| For the ~75,000 travelers directly impacted, it will
| probably have some long-term impact in their purchasing
| decisions.
|
| But for the 329,925,000 other Americans, many of whom have
| a long history and belief in Southwest's reputation for
| customer service and fair policies? They will have
| forgotten by next week.
| hrunt wrote:
| Both can be true. Some people will swear them off. Some
| won't. Some SWA will win back with steep discounting.
|
| For me (I am affected), this is actually another in a
| series of recent events that are making me reconsider my
| preference for SWA. They are no longer a "cheap" airline,
| routinely more expensive than the other major carriers.
| Their planes are not nice anymore. I've flown on a few
| other airlines over the past few years and found their
| planes to be nicer with more features (like chargers and
| phone/tablet holders). And now this. The cancellations are
| one thing, but they totally botched the communication of
| it, and their practice of delaying flights throughout the
| day only to cancel half of them after several hours left
| people stranded.
|
| Will I stop using them? We'll see how they respond, but
| they may not be my first choice anymore.
| zippergz wrote:
| Just as with ISPs, for many people in the US, true airline
| choice is not a luxury they have. Depending on their origin
| and destination, there may be only one airline that flies it,
| or only one that flies without a ridiculous set of stops or
| layovers. Even if people want to switch airlines, unless they
| live near a major airport or have high flexibility on when
| and where to fly, it's not really practical.
| maxerickson wrote:
| Yeah, I have 2 reasonably drivable airports that are both
| served by Delta. It's even the case that I can mostly get a
| less expensive flight with a good itinerary (airport to hub
| to destination) or a more expensive flight with a bad
| itinerary (airport to hub to other hub to destination).
|
| Is Southwest the lone primary carrier for many of their
| airports?
| vincent___ wrote:
| At least it reduces the impact of human activity on global
| warming. We can always talk about optimization, database,
| companies... The responsibility of each individual is engaged.
| 4000 flights, privileges of rich countries.
| eiiot wrote:
| My family and I are stuck in Long Beach after our flight was
| cancelled. We called the airline and the next available flight is
| Saturday. No hotel compensation, no partnerships with other
| airlines for rebooking on another flight. What a nightmare.
| samename wrote:
| I just found out they have to provide some compensation or
| reimbursement, eventually:
| https://www.transportation.gov/airconsumer/airline-customer-...
|
| Save your receipts, and submit a reimbursement request.
| Escalate if needed.
| mdavidn wrote:
| That page describes Southwest's obligations after
| "controllable cancellations." When a cancellation is outside
| their control, Southwest only offers to "seek to arrange a
| discount" on accommodations. They list "weather" and "FAA-
| required crew duty limitations" as elements outside their
| control.
| 650REDHAIR wrote:
| This hasn't been weather related in days.
| hrunt wrote:
| SWA has been telling affected passengers to keep their
| receipts for reimbursement later, so hopefully once things
| clear up, they will go beyond the letter of their policy to
| help make things right with their customers. No guarantees,
| but also not many options otherwise for those of us
| affected.
|
| Their brand is going to need it, so whatever that cost is
| will probably be worth it for them.
| elijaht wrote:
| My datapoint: Had a flight this _Thursday_ from RIC- >ATL
| cancelled this morning. Interesting that they are cancelling
| flights this far out, where both cities are relatively unaffected
| by the storm beyond some cold temps.
|
| Was a blessing in disguise for me as I could find a similar
| flight on a different airline. Hope anyone else flying Southwest
| can find a way to their destination
| hrunt wrote:
| The agent this morning told me that they are using the next
| three days to move staff and planes where they need to be, and
| the result will be a lot of cancellations when the routes don't
| serve that. That's why nothing is getting rebooked until
| Saturday.
| asah wrote:
| tl;dr: distributed systems are hard.
| francisofascii wrote:
| or maybe you meant to say logistics with little slack is hard
| chasd00 wrote:
| if i remember my lore correctly it's not a distributed system.
| It was originally either a mainframe or like AS400 application.
| I want to say it was related to Saber somehow... maybe they
| leased it. When i interviewed there years ago they were porting
| it to some kind of Java stack or maybe they had already ported
| it and were building it out further. On a tangent, I have to
| admit, their interview was the best one i've ever had. Very
| competent people, well rounded process, it was actually more
| fun than stressful but super challenging too.
|
| / this was a lonng time ago, like 10+ years, it could be a very
| very different animal now
| bronson wrote:
| I don't think Southwest was ever on Sabre. Their scheduling
| used to be in-house, now on Amadeus. Computerized operations
| started with Braniff's software and have ... ah, grown since
| then. Name almost any technology and it's probably running
| somewhere in their stack.
| 11235813213455 wrote:
| This saves millions of tonnes of CO2 and pollution
| smm11 wrote:
| I don't understand this. I just flew from West to East Coast,
| then back, on another airline. Left on time from origin and
| transfer airport both ways, and arrived early on all flights. The
| "bomb cyclone" has been gone days, and SW passengers are still
| stuck, sleeping on floors, or have their bags hundreds of miles
| away.
| pc86 wrote:
| Southwest is just not a good airline.
| silisili wrote:
| In my 20 or so years of travel, Southwest has never been a
| good airline. But what more boggles my mind is that they've
| never been the cheapest, either(ignoring spirit and the
| like). Usually American or United was.
|
| Was that just where I lived (primarily in the east), or am I
| missing something?
| mdavidn wrote:
| I live on the west cost. When I compare United and
| Southwest, United is cheaper on face value but more
| expensive with bags. They hide some of total cost in fees.
| silisili wrote:
| That's a good point. I rarely travel domestically with
| checked luggage.
| bmitc wrote:
| There's no way Southwest hasn't been the cheapest. I just
| cannot see that being possible given that they don't nickel
| and dime you for everything.
|
| United is by far the worst airline imaginable. I actively
| will not fly with them after being stuck on an airplane
| with them for 19 hours straight due to their fuckup, only
| to have to spend a night in New Jersey at a terrible hotel
| because we then missed our connecting flight (obviously).
| In the end, all they offered was a measly certificate that
| required you to use it at United (and was the equivalent of
| a mere fraction of the total flight cost).
| silisili wrote:
| I haven't flown in 3 or 4 years, if that matters. Most of
| my flights were to California or DC, so perhaps that
| matters too.
|
| I know what you mean about fees. I just booked American
| for my wife last week, and it's horrid now. No free bags
| anymore, and you have to pay to even pick your economy
| seats. It was way more infuriating than the last time I'd
| booked.
| bmitc wrote:
| Yea, it could totally be region dependent, something I
| wasn't quite thinking of.
|
| We just did the same with American. It was like double
| the listed price by the time we finished with bags and
| fees.
|
| Almost all airlines have gotten worse due to them getting
| rid of a lot of nonstop flights.
| HDThoreaun wrote:
| It completely depends on location. Southwest is super
| cheap on some routes, less so for others.
| tpmoney wrote:
| My experience with SW has always been that they're better
| than everyone else at their pricepoint, and below their
| pricepoint the savings aren't worth the hassle. They've
| gotten worse over the years, but they're still consistently
| the roomiest flights I've been on, their employees have
| always been friendly and I never really dealt with any
| major delays with them.
|
| My experiences with Delta were that they were a 50/50
| between a nice experience or a "should have just bought a
| Jet Blue / Spirit ticket" experience. My experiences with
| United were that anything that wasn't a major route, you
| were likely going to fly in a tubo-prop coffin and come out
| with so many aches and pains any money you saved will be
| spent on advil and massages. They're also the only airline
| that's managed to lose my luggage twice.
| chrisbolt wrote:
| Everyone's experiences will depend greatly on the routes
| they're flying and their home airport(s). If you're
| flying on a regional for the legacy airlines, you will be
| on a smaller plane (though turbo-props are pretty rare
| these days). The worst is the CRJ200, aka "The Devil's
| Chariot."
|
| Since Southwest only flies 737s, you'll get a roomier
| ride than a regional.
| jmugan wrote:
| Out of Austin, Southwest is the only airline for many
| direct flights.
| vincent___ wrote:
| At least it reduces the impact of human activity on global
| warming. We can always talk about optimization, database,
| companies... The responsibility of each individual is engaged.
| 4500 flights, privileges of rich countries.
| avalys wrote:
| Astonishing. Based on reports from employees on Reddit (https://w
| ww.reddit.com/r/SouthwestAirlines/comments/zw32yt/p...), the
| actual story here is that Southwest has had a complete
| operational failure and is simply incapable of determining where
| their crews are, what flights they are eligible for, and
| associating them with flights on the schedule.
|
| It sounds like there is a semi-automated system which broke
| somehow, or more likely failed to support the load of cascading
| changes that resulted from weather disruptions, and they simply
| don't have the capacity or flexibility to deal with this other
| than by cancelling flights for three days until they can sort
| everything out manually.
|
| Reading between the lines a bit, one possible root cause is that
| their semi-automated system required crew members to update their
| status by phone when something goes wrong (perhaps with another
| human in the loop), and the sheer volume of disruption overloaded
| their phone system and resulted in the automated system becoming
| completely decoupled from the actual state of reality without
| sufficient bandwidth in the phone channel to get it back in sync
| in a reasonable amount of time.
| CharlesW wrote:
| According to that post they also have no idea where bags are:
| _" Checked bags are currently a disaster. Plan to not see your
| checked luggage for at least a month. In the interest of 100%
| transparency, some bags will be 30+ days lost in the system."_
| pwillia7 wrote:
| Unremoved reddit post:
|
| On behalf of all employees: WE ARE SORRY! I will give it to you
| straight- this meltdown was caused entirely by Southwest. It
| was triggered by the storm, but the failure to recover quickly
| is on Southwest 100%. If you are still hearing "weather" almost
| a week after the storm, it's not true. Couple main points: 1.
| Please be patient with us. We desperately want to do everything
| we can to get you where you're going. 2. This shitstorm is
| because the crew scheduling software went belly up and it
| almost all has to be unraveled over the phone with crew members
| calling scheduling. If we had better technology which
| eliminated the need for phone calls, this would have been fixed
| by now. 3. If you are able to find alternative transportation
| to your final destination- DO IT. Another airline, bus, train,
| Lyft, rental car, ANYTHING. Southwest WILL NOT be able to get
| you to your destination anytime in the next few days. 4. Like I
| said, it's gonna take at least a week to get back to normal
| operations for Southwest. If anyone has questions, I will try
| to answer them. I work ground ops at one of SWA's hubs. EDIT
| FOR FAQs----
|
| Checked bags are currently a disaster. Plan to not see your
| checked luggage for at least a month. In the interest of 100%
| transparency, some bags will be 30+ days lost in the system.
|
| Will my flight for X date go out? Next 3 days- plan on a
| cancellation. 4-7 days- likely to go as scheduled. 7+ days-
| should see operational recovery.
| hinkley wrote:
| Somewhere there's a retired guy sipping his morning coffee and
| saying, "Told ya."
| CyberDildonics wrote:
| There are people actively doing interviews saying that.
| achow wrote:
| The Reddit post is now deleted.
|
| Saved screenshot: https://i.imgur.com/bchgmm7.jpeg
| caycep wrote:
| Granted my usually routine flight in 3 days, from BUR to SJC
| got canceled. And no way to get thru and the usual handy dandy
| "full refund!" button has been replaced w just an email form
| with "we'll get back to you!" I sort of believe it
| [deleted]
| hintymad wrote:
| > the actual story here is that Southwest has had a complete
| operational failure and is simply incapable of determining
| where their crews are, what flights they are eligible for, and
| associating them with flights on the schedule.
|
| This is amazing. Southwest was famous for its operational
| efficiency and quality. Companies eagerly learned from them. I
| wonder what has changed.
| iamtheworstdev wrote:
| it's definitely epic. I know this is anecdotal but a friend
| of mine and pilot for Southwest had to pay for his own hotel
| room after he captained a flight a couple of days ago. SW
| apparently thought he was in another state (and it wasn't a
| neighboring one) even though he just flew the flight that
| they scheduled him on. Not long (hours) after that he was
| told to drive to the nearest Southwest physical location and
| check in. Rumor is that all Southwest pilots were told that a
| physical presence was required to verify location.
| wwweston wrote:
| Other Reddit discussion (with the occasional insider chiming
| in):
|
| https://www.reddit.com/r/bestof/comments/zwd1fq/u4sammich_ex...
|
| https://www.reddit.com/r/flying/comments/zw5lsl/southwest_pi...
| shmatt wrote:
| im kind of doubting that thread, not necessarily fraud but not
| every employee has the full picture
|
| * this memo[1] from the SWA VP thats circulating dated December
| 21st calling a staffing emergency saying all hell is about to
| break loose. I'm pretty sure that guy didn't know of the
| software crash before it happened
|
| * Every single piece of checked in luggage is going to take 30+
| days to be found? Weird none of the extensive press has
| mentioned people not getting their bag. Also its pretty much an
| airport thing and not an airline thing. Airports are in charge
| of deplaning your luggage and bringing it to you
|
| * They're running 40% of flights completely manually with 0
| scheduling software? That's extremely impressive
|
| [1]
| https://mobile.twitter.com/ruthschmidt/status/16074609858619...
| pcurve wrote:
| Wow... that memo... is filled with such contempt. It's
| shocking. I wouldn't want to work there.
| burkaman wrote:
| Yeah, and just to add some context for people who might not
| be familiar with US healthcare, what the VP is asking for
| is effectively impossible, so this is an order to work
| sick. The majority of Americans cannot just "go to the
| doctor" whenever they feel like it, especially 4 days
| before Christmas. The best most people could do in this
| situation is call a doctor's office and be told they can
| get an appointment in a couple months, or go to the
| emergency room if it's an emergency. So the choice here if
| you're sick is to waste emergency healthcare workers' time
| and probably pay extra for unnecessary emergency care, go
| to work sick, or get fired.
|
| This is obviously intentional, and in many parts of the US
| totally legal. I don't know specific Colorado laws but
| there is no federal law against this.
| hodgesrm wrote:
| On the flip side Southwest has stranded what look like tens
| of thousands of passengers. The memo seems to be focused on
| curing that problem. It's not clear what the alternative(s)
| would look like short of ditching all their passengers.
|
| It looks like this is not a fun week for anyone who works
| at Southwest.
| burkaman wrote:
| That memo was sent before the storm and before anyone was
| stranded. It was an attempt to prevent a future crisis.
| If your goal is to prevent a future crisis, alternatives
| to ramping up employee abuse 3 days before a big storm
| might be:
|
| 1. Hiring more people (more than 3 days in advance).
|
| 2. Paying people more so they don't quit.
|
| 3. Giving people paid sick time and not threatening their
| jobs when they take it so that they don't come to work
| sick and get a bunch of their colleagues sick.
|
| 4. Selling fewer tickets and running fewer flights if you
| don't have enough capacity to support the current
| workload.
| cratermoon wrote:
| That's a "beatings will continue until morale improves"
| message.
| paganel wrote:
| That's also an excellent ad for why unions are needed.
| tpmoney wrote:
| Aren't most airline employees in a union?
| burkaman wrote:
| > I'm pretty sure that guy didn't know of the software crash
| before it happened
|
| Why not? Some things aren't hard to predict, it's like a
| Ticketmaster employee saying "our site is probably going to
| go down when the Taylor Swift tour goes live". If you know
| your systems are deficient then you can predict they'll fail
| in a crisis, especially if that failure has happened before
| (as it has for Southwest).
| barkingcat wrote:
| luggage is totally an airline thing, not an airport thing
| sokoloff wrote:
| > [it's] pretty much an airport thing and not an airline
| thing. Airports are in charge of deplaning your luggage and
| bringing it to you
|
| That depends on the airline and airport. It's extremely
| common for the airline which operates out of a large part of
| a terminal to do their own ramp work.
|
| Southwest employs thousands of ramp agents to do their own
| ground ops and under-wing work at their bases.
| [deleted]
| avalys wrote:
| It's not really a software "crash", it's more like, they know
| their process can't handle more than a certain amount of
| disruption before falling apart. And so if you can see that
| due to weather, etc. a lot of disruption is likely to happen,
| you can predict there's going to be a problem, and also
| predict that you simply don't have the staffing and systems
| in place to do anything about it.
| phpisthebest wrote:
| There have been plently of reporting about the bag issue,
| including a Chicago local reporter posting a video of 100's
| or maybe even 1000's of bags in chicago most of southwest
| tags on them
|
| It appears there was a MASSIVE disconnect between baggage and
| people, bags are ending up all over the place even though the
| people never left.
| tiahura wrote:
| Sounds the the perfect setup for an AirTag commercial.
| ghaff wrote:
| The rare times I check a bag I throw an AirTag in.
|
| Although the airline app has always been accurate it's
| cheap insurance if a bag isn't properly scanned or
| something.
| airtag wrote:
| I've been flying with an airtag in my luggage in Europe
| since summer this year.
|
| It's been great. My luggage missed a connection in
| december, when I arrived at the destination I could
| instantly see that it still was at the other airport. So
| while the others were waiting for the bags, I could
| already file a complaint (at the nearly empty complaints
| queue). Then three hours later I could see bag movement
| to the tarmac and knew that my bag cought the next
| flight.
|
| Also, when in a city, it's a lot of fun to say: bring me
| back to my bag instead of looking up the hotel address.
|
| For anyone with an iphone & checked bags I can really
| recommend it.
| Dragonai wrote:
| You bagging the username "airtag" just to share this
| comment is so funny.
|
| I think this is a great idea though and I didn't even
| think about "bring me back to my bag" - that's awesome :)
| bombcar wrote:
| iPhone alert: there are ten thousand airtags near you
| giantrobot wrote:
| Well that at least narrows it down to _near_ you. /s
| karlkatzke wrote:
| > calling a staffing emergency saying all hell is about to
| break loose
|
| That memo is specifically about ground operations at DEN, and
| specifically because of the arctic weather conditions and a
| high number of sick calls and ramp agents that outright quit.
| You can earn more flipping burgers in Denver than you can as
| a SWA gate agent with five years of experience. Failing to
| pay staff is also a management failure.
|
| The scheduling software crashed due to the number of pilots
| and flight attendants that were out of position and the
| number of changes that were made to the schedule. I would
| imagine that there was an overflow in some situation -- i.e.
| "the number of changed schedules should never exceed 65535"
| that worked every year until this one. But this system was
| already known to be unstable, another Reddit comment said
| that "there are settings you don't change for fear the entire
| thing will crash." Which it has before in 2016. Not expecting
| that history will repeat itself and doing something about it
| is also a management failure.
|
| > Also its pretty much an airport thing and not an airline
| thing
|
| Absolutely incorrect. While the airport runs the automated
| conveyance system that gets the bag from where the rampers
| drop it to the baggage claim, the people that handle the
| baggage at every manual step in between are SWA employees. If
| there aren't enough of them, the bags don't make it on the
| plane on time.
|
| Notice that the people doing this work for Southwest Airlines
| are all wearing Southwest uniforms.
|
| Like most complicated failures, this was failure with
| multiple causes and contributing factors. The core of the
| problem seems to be that management was rent-seeking without
| making appropriate structural changes to keep up with system
| load.
| verall wrote:
| There's tons of people without bags, and additionally due to
| the total operational breakdown there is no one to ask about
| where your bag is and hear any idea about what's going to
| happen.
| kart23 wrote:
| https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi.
| ..
|
| i believe this story. the scheduling system is too fragile
| and can't automatically pick up disruptions to crew movement.
| they're still manually scheduling some crews, but I think
| their main problem is they don't have a way of knowing which
| crews are where and therefore can't schedule flights. and
| it's not every single piece that's going to take 30 days, but
| it seems pretty likely that some bags will be lost that long.
| treis wrote:
| It's not just that. There's complex rules around how
| frequently crews can work flights and how much time off
| they need. Delays can cause ripple effects as an hour delay
| can make an expected available crew unavailable. Then you
| have a mismatch of available crews and planes and scheduled
| flights. It's one big thorny ball of spaghetti that even
| the best designed computer system will struggle to
| unwravel.
| eightysixfour wrote:
| And to top it off, unions fight tooth and nail to make
| sure this type of information cannot be tracked, which
| keeps a lot of systems manual. I understand why the
| unions don't want the data to be captured, but it also
| makes it hard to optimize physical systems.
| raverbashing wrote:
| > the software for scheduling is woefully antiquated by at
| least 20 years. No app/internet options, all manual entry
| and it has settings that you DO NOT CHANGE for fear of
| crashing it.
|
| But hey it's cheaper if we don't change it right? /s
|
| (and I kinda can agree with this, like 10% because they
| might end contracting with some bigger IT company that
| doesn't care about shipping something that works)
| 0cf8612b2e1e wrote:
| Replacing an airline scheduling system sounds insanely
| complex. So many moving pieces, tens of millions of
| physical entities to track per day: crew, passengers,
| planes, spare parts, luggage, crew maximum allowed shift
| schedules, open gates, interacting with
| domestic+international terminals, seat assignments, and
| so on. Edge cases on edge cases on edge cases developed
| over years, probably in some god awful language running
| on a mainframe somewhere before "best practices" were a
| thing.
|
| Modernizing such a system would take enormous political +
| real capital. The new system would undoubtedly have many
| growing pains as formerly resolved problems were not
| covered in the new system. It's no surprise nobody would
| want to touch such a system.
| jchanimal wrote:
| One of the more tangible outcomes from my work on mobile
| sync is the United scheduling software. Seems like
| Southwest was too little too late on IT upgrades.
| https://www.couchbase.com/customers/united-airlines/
| apelapan wrote:
| How is Couchbase in general and the mobile sync in
| particular relevant to United not having this disaster
| right now?
|
| I've worked with airline scheduling software at almost
| the scale of United (about half the size, for the largest
| customer). That was using an Oracle database, which was a
| bit of a pain and a big expense, but worked fine as long
| as there was competent admins and competent devs. Would
| Couchbase be disaster-proof even when run by clowns?
|
| Not saying there aren't better and worse choices for
| databases in any given situation. Just saying that there
| are lots of them that work perfectly well in competent
| hands.
| jchanimal wrote:
| Architecture advantages over the phone based system
| described upthread are obvious. The reason United chose
| Couchbase is because it's designed for offline updates.
| So if a plane, phone, or airport is disconnected they can
| still do the data entry work and reconcile
| asynchronously.
| jcomis wrote:
| I flew out of Denver after several delays/cancels and not
| only was the baggage area full of bags, the outside gate area
| was absolutely flooded with bags COVERED in snow. Which means
| they have been there for days. I'm talking dozens and dozens
| of haulers loaded to the brim with bags covered in snow. The
| baggage area was wall to wall bags.
|
| Additionally I was told (by a crew member) Southwest refused
| to meet wage demands with ramp / fuel crews so they all quit.
| Apparently they sent them an email basically saying a
| recession is coming so be thankful you have a job...
| Meanwhile all other jobs pay more in the area. Anecdote I
| know, but we did have to wait 2.5 hours for fuel on my
| flight.
| chasd00 wrote:
| a people problem like you describe seems much much harder
| to solve in any reasonable time frame compared to a
| software issue.
| cududa wrote:
| Well I mean the article we're commenting on mentioned the
| baggage issue and has pictures of it
| pembrook wrote:
| Incidents like this go a long way toward teaching corporate
| America that software shouldn't be some forgotten "IT" vertical
| on your org chart that you staff with random contractors.
|
| Software is core to your business. If you don't invest in it,
| you're going to pay the price.
| lezojeda wrote:
| They are already pretty resentful at us because we have such
| high salaries compared to the rest of the population... I bet
| that nobody is willing to give any penny more to their IT
| departments
| bmitc wrote:
| People mistaking software as something that just _does
| something_ is the problem. Software is stored and executable
| knowledge, so if some component of that breaks down, you need
| to be able to pull it from somewhere else. Otherwise, you get
| situations like this.
| 0x445442 wrote:
| Yeah the storm angle is a lie.
| BitwiseFool wrote:
| This is hearsay, but I believe any cancellations due to
| weather are not subject to fines and fees from the FAA and
| the airline is not obligated to provide more generous
| compensation/accommodation for customers. So if there's any
| reasonable way to blame a failure on inclement weather
| airline companies will do so without hesitation.
| bumby wrote:
| > _cancellations due to weather are not subject to fines
| and fees from the FAA_
|
| I thought this too, but apparently there are no federal
| regulations other than reimbursement for the direct costs.
|
| "Airlines are not required to provide passengers with money
| or other compensation for costs that fall outside of the
| cancelled airline ticket and fees tied directly to the
| airline ticket (such as baggage fees, seat upgrades, etc.)
| when flights are cancelled." [1]
|
| "Each airline has its own policies about what it will do
| for customers on bumped or cancelled flights. There are no
| federal requirements." [2]
|
| However, there are rules around "bumping" passengers on a
| flight.
|
| "An airline is required to compensate you after
| involuntarily bumping you from an oversold flight in
| certain situations. However, there are many situations
| where you are not entitled to compensation." [3]
|
| [1] https://www.transportation.gov/individuals/aviation-
| consumer...
|
| [2] https://www.faa.gov/faq/what-are-policies-bumped-or-
| cancelle...
|
| [3] https://www.transportation.gov/individuals/aviation-
| consumer...
| 0x445442 wrote:
| You're wrong dude.
|
| https://www.reddit.com/r/SouthwestAirlines/comments/zw6upo/
| h...
| ipython wrote:
| How does that post in any way refute the parents point? I
| agree with the parent that management will try like hell
| to blame this on the weather- heck they already are. Just
| check out their press releases.
| 0x445442 wrote:
| Yeah I'm a bit confused how the comment even relates to
| mine. I took it to mean that my assertion that laying
| blame on the weather for this mess is bogus and a lie.
| But that assumption may be wrong.
| ThunderSizzle wrote:
| He's not wrong. Every airline will attempt to blame
| operational issues on non-finable causes, such as
| weather.
| tedunangst wrote:
| So what are the finable causes for flight cancellation?
| kube-system wrote:
| It is likely _both_. Scheduling is quite easy when it goes
| normally. Scheduling becomes a more difficult task when more
| exceptions happen. That reddit post is by is some boots-on-
| the-ground employee, not someone who understands the
| algorithmic details of SWAs somewhat unique routing
| challenges.
| EFreethought wrote:
| > Scheduling is quite easy when it goes normally.
| Scheduling becomes a more difficult task when more
| exceptions happen.
|
| I think some slack/spare capacity or redundancy somewhere
| would make a LOT of problems less likely to happen. Running
| lean only works out when nothing goes wrong. I know I
| should not be shocked by the extent of human stupidity, but
| why the "business" types do not get this is bizarre to me.
|
| And paying people more would help too.
| HDThoreaun wrote:
| The airline business is too competitive for labor slack.
| People put up with spirit's bullshit because it's $5
| cheaper. Any airline that incorporates slack will
| immediately find itself without customers. If they make
| it to a catastrophe it would work out, but it's extremely
| unlikely that they will be on business that long.
| tpmoney wrote:
| That memo from the VP someone linked earlier seems to
| indicate their slack / spare capacity has already been
| used.
| phpisthebest wrote:
| It seems they do their scheduling like they do their
| seating... it is all Free for All
| bombcar wrote:
| Should just text all pilots to bum-rush any plane they're
| equipped to fly. Why not add the two front seats into the
| free for all?
| kube-system wrote:
| You joke, but being able to swap any pilot to any plane
| is part of the reason their fleet is 100% 737s.
| vhiremath4 wrote:
| I received a notification that my flight had been cancelled from
| North Carolina to New York at 12am for a 12pm flight today. My
| "software incident" senses started tingling and I immediately
| redeemed credit and rented a car instead (currently on the road).
| So happy I did that.
| lancesells wrote:
| That's an incredible amount of people traveling. Even at 100
| people a flight that's 540,000. Half a million people.
| 0x445442 wrote:
| This is BS. The storm was last week.
| pimlottc wrote:
| Effects from the storm linger. Buffalo's airport shut down due
| to weather issues and remains closed through at least
| Wednesday.
| JustLurking2022 wrote:
| Seems they only linger for Southwest, as they made up 90% of
| all flight cancellations today.
| josephcsible wrote:
| There's also the issue that a storm, no matter how bad,
| shouldn't make them forget where they left their crew members
| and your bags.
| 0x445442 wrote:
| In the case of Southwest the scope of the issues is not
| because of the weather.
| [deleted]
___________________________________________________________________
(page generated 2022-12-27 23:00 UTC)