A Review of ``The Day the Phones Stopped'' by Leonard Lee At 14:25, 1990-01-15, exactly thirty-four years ago as of today, the AT&T national telephone network started failing to route roughly half of all calls correctly for a little over nine hours. The book under review describes this failure in some detail and, to my surprise and delight, this book mostly details many other failures caused by programming errors the author found to be worthy of inclusion. Amusingly and horrifyingly, this book tells of the ongoing software crisis, that is software growing beyond man's ability to tame it, in 1991. In some ways, it's funny to see how things have worsened. Poorly, this is yet another book which I'd to read on loan from my local library, as I couldn't find it any other way. I can easily read any old new garbage, but a nice book like this is hard to read. This book was written at a time when computer terms had to be explained to the reader, for computers weren't yet widespread and well-understood; newer books are poorer for the lack of such explanation. The first chapter, ``DANGEROUS MEDICINE'', starts with something more severe than telephone failure, by explaining the Therac-25, a controlled dosage radiation therapy machine well-known for killing or maiming several unfortunate patients. I've long known of this deadly machine, and its fatal flaw in hardware misconfiguration, but only learned the details in this chapter; where I'd imagined an array of mirrors or whatnot, I instead found a simple tungsten rod not in-place. The machine had two ways to deliver radiation: by a low-powered electron beam, or a high-powered X-ray striking the tungsten. The chapter elaborates on how the flaw was repeatedly uncovered in operation and explained away, for the technicians insisted the computer makes no mistakes; after each maiming, the units were taken in for analysis, to find no errors. The flaw was what is known as a race condition, due to the removed hardware mechanisms from previous models. Further, the emergency shutdown mechanism was too slow at three tenths of a second. After the flaw was known, the Therac-25 software was rewritten and a one- pulse shutdown mechanism, working at one three hundredth of a second, was installed onto all models. The chapter gives some room to other faulty medical devices too, such as pacemakers that don't work. The second chapter, ``THE UNFRIENDLY SKIES'', details airplanes and businesses operating airlines to some degree. The attention airplanes received in this book is outsized compared to all other topics covered and very conspicuously so. For some reason, this chapter is easily the longest in the book, at twice or thrice the length of most others, and more than four times larger than the smallest; two later chapters cover very similar or related topics even further; this chapter should've been split. It begins with an amusing remark that not all software errors work against the common man, using the example of airline rewards being improperly credited, but honoured nonetheless; follows is a case in which rewards weren't being properly credited, but this was also corrected. Following yet is a case of an automatic baggage handling system that couldn't properly track luggage when deployed; the book remarks that the software's testing environment was unrealistic, a common theme with later chapters. In July 1983, Air Canada Flight 143 ended abruptly after the plane exhausted its fuel supply in mid- flight; the fuel reporting subsystem could handle total failure of one of its two data channels, but not partial failure, and so failed to work; unit confusion later led to the plane being underfueled. When fuel ran out, the emergency battery systems allowed the fully-computerized cockpit to work, and an air turbine provided more power. The plane, without full power, provided a worse experience than older planes lacking computers entirely. That situation was worsened by newer radar systems failing to recognize the plane, because it no longer sent out coded signals with a transponder; fortunately, the particular nearby air control tower had an old system that would recognize anything. The flight ended without death with the plane landing as safely as it could on a racetrack around many viewers. Interestingly, shortly before starting the book I read this alternative report of the same incident: https://www.damninteresting.com/the-gimli-glider The chapter goes on to describe the Airbus A320, and taught me the meaning of fly-by-wire. On 1988- 06-26, an A320 crashed at a French air show, and was deemed pilot error. On 1990-02-14, Valentine's Day, an Indian flight crashed at an airport, and India was just as incompetent then as now, with the emergency response both pathetic and entirely unplanned, thwarted by things like a locked gate; this quote from the book properly captures the sorry state thereof, and the situation is no better today: ``One rescue truck tragicomically attempted to spray foam from the other side of the fence, but could barely reach any of the wreckage.'' The chapter contains warning about computers causing recklessness, such as with the French incident, by making pilots believe themselves to have more leeway than is present; I found some concerns to be similar to those for the current self-driving car nonsense. Another concern is planes programmed to disobey pilots when a maneuvre be deemed dangerous, but such things can be necessary in emergencies. Military failures are also noted, like a Swedish crash caused by delayed controls. On 1985-03-13, a Blackhawk helicopter crashed. In April 1987, a similar failure occurred. It was found to be caused by radio tower interference commanding the helicopter's fly-by-wire systems. The given solution was in the form of special maps which show strong sources of electromagnetic interference to be avoided. On 1983-04-31, Korean Airlines Flight 007 flew into protected airspace and was shot down by a Soviet fighter jet. The Inertial Navigation System (INS) tracks the position of a plane by movement deltas from an origin, and may have been slightly misconfigured. It's reportedly very easy to misconfigure and ignore any warnings. On 1987-07-08, an INS error was noted to have, nearly, caused a collision. The second chapter ends with notes on autopilot errors and grieving over how pilots were once taught about the entire plane and its failure modes, but no longer. It goes on to explain how they perhaps could be taught about the plane at a higher level, despite that a total overview is now impractical. The third chapter, ``THE BIGGER THEY ARE...'', finally explains that titular accident causing AT&T's telephone network failure that lasted nine hours. Unlike in the first chapter, I learned more about the technical nature of this failure from elsewhere, and following is a pleasant article about that: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse The chapter begins by describing AT&T's Network Operations Center (NOC) and how it had displayed the status of the 114 switching centers, so-called 4ESS units, across the United States; the display was updated every five minutes. At 14:25, 1990-01-15, the trouble began in New York and spread out from there. The chapter mostly details the affected businesses, such as telemarketing, and also contains the first mention of the Sabre airline reservation system. The New York Stock Exchange was designed in a way leaving it unaffected, and other long-distance telephone systems still worked without flaw. The flaw, introduced in December 1989, was triggered by a very rare set of circumstances. The self- healing network hadn't been designed to cope with all nodes failing by exactly the same way and time like this. AT&T employees after getting more specific information, so-called ``machine-discretes'', revealed the error to be in Signalling System 7 (common channel signalling) nodes, but not the older Signalling System 6. Signalling System 7 was out-of-band, faster, and capable of more features, but also more complex. AT&T maintained a model switching network in a laboratory for diagnosing failure of this kind, which is how the flaw was eventually found. That first patch prepared for the problem failed to work, and the second patch worked by preventing a signal from being sent; it took days for a final solution patch to be made. Ironically, news reporting on the event was difficult due to the communications issues for CBS News in New York. Similar telephone network issues happened later on, in Minneapolis and California. Oddly, I believe the C language wasn't mentioned at all in the book, despite being responsible for many of its failures, including this AT&T failure. One AT&T executive had this to say, and it may have been included to reinforce the false notion about software quality: ``Our goal is to achieve no errors certainly, but we're also not going to delude anyone by saying it'll ever be one hundred percent perfect.'' The chapter also discusses other minor failures, such as a flaw in a corporate mail system, to which one executive commented so, amusingly: ``How many computer errors are they going to get away with?'' The fourth chapter serves as a lamentation on software development more than covering any particular problems, although it also features examples of misguided space probes and stock exchange downtimes. Many times in this book is repeated a saying, a factoid, which I know to be false and which follows: ``In truth, software can never be made completely reliable and trustworthy.'' The book continually remarks on how no amount of testing will suffice, which is true, but also tells a persistent lie repeatedly, that all software must have flaws. The fourth chapter even begins with ``The bottom line is that all software contains errors.'' and later in the next paragraph ``It can't be done. Software can never be made completely error-free.'' and how false. Software can be tamed. The fourth chapter also features a cameo from Frederick Brooks, who tells of how software may be the most complex constructs by size ever created by man. Of perhaps basic interest is the mentioning of European Computer-Aided Software Engineering (CASE), but I chose to look no further into it for now. The fifth chapter, ``MONEY, MONEY, MONEY'', regards the woes of banking and other businesses, but is mostly concerned with banking. It covers how businesses waste money on systems that don't work, the improperly sent checks and other problems such systems cause, amongst other things. The Bank of New York had unbalanced books and was forced to take a loan from the federal reserve, with all assets as collateral, due to a programming error; something similar occurred with the Clearing House Automated Payment System (CHAPS) in the UK, with money sent improperly but returned voluntarily. Anyone who's read about banking software woes has read such stories before. This chapter also discusses problems with domain mismatch from programmers who know not about the very domain for which such programs are written, an example being a programmer writing banking software who knows none of the terms thereof. The chapter also contains some interesting paragraphs on throwaway software, so-called ``cardboard'' programs, that avoid some of the outlined issues by being disposable as requirements change or fall. I'd previously known not too much about the Sabre airline reservation system, and this chapter gives details about the secure and well-supplied compound housing it, alongside some amusing software woes causing failures, despite such extreme measures. Perhaps Sabre should've been given more attention. The sixth chapter, ``COMPUTERS AND UNCLE SAM'', is similar to the fifth, but about the government of the United States. It gives a very detailed description of the tax return procedure of the Internal Revenue Service (IRS) and its dysfunctions. Amusingly, a long text about the IRS wanting to upgrade its aging infrastructure could be written just as easily today. In November 1984 and anticipating a great amount of work, its unfinished and upgraded system was used, to ghastly results. This chapter details great amounts of redundant work required of the employees in order to cope with flaws in the software, and also things such as cases where disgruntled workers discarded tax returns in the trash to avoid dealing with them. This is yet another old work looking to the future with the hopes of an electronic filing system to avoid such horrible drudgery; unfortunately, such hopes have been dashed by parasitic businesses. The book is one so old that the senator's full name is given, Albert Gore. The seventh chapter, ``UP IN THE AIR'', is another aviation chapter, this time regarding air traffic control systems. It starts with the AUTOMATED RADAR TERMINAL SYSTEM (ARTS IIIA) and the story given about it in this chapter is one that sounds believable to the ignorant, too silly to the novice, and only all-too-believable to the wise. The machine tracking the flight plans would become overloaded, and crash for thirty seconds or so at a time, leaving the display useless. A patch was made to cope with a particularly large event in Dallas, but it left the display garbled and therefore unreliable; air traffic controllers were forced to use their memory, a pen, and paper of all things to track the flights, leading to near collisions even after new leaving flights were stopped. While the incident resulted in a new system, other places weren't so fortunate. This chapter includes a list of flight delays caused by computer problems. Interestingly, this chapter notes the anticipated increases for air traffic in the next millenium, and I wondered if they'd come true only to realize the attacks on 2001-09-11 probably put a damper on those increases, coincidentally. Also noted are developments of the FAA's National Airspace System (NAS) and, upon delays, its Interim Support Plan (ISP). Finally, the chapter gives information about a preventable plane crash involving one in bad visibility, which got in the way of another taking off, leading to deaths, and an emergency escape system failed also. The eighth chapter, ``THE GOD OF WAR'', covers the United States military and its software problems. It begins with examples of expensive and redundant military installations that have issues such as a radar system interfering with fly-by-wire controls. A third of the chapter is dedicated to the B-1B bomber project, then-new, and is an example of failures caused not just by software but hardware and general design problems with its goals; an example of a failure was something like the radio jamming ability jamming itself. The chapter includes an interesting paragraph about the future of war, with soldiers carrying handheld computers, wearing helmets with computer displays in the visors, and with computer identification chips in their teeth; as far as I know, only the final prediction has yet to come. I was pleased to see Ada mentioned in this chapter with the usual story of over three hundred languages being in-use and a replacement being sought, but, unfortunately, the book was written well before Ada 1995 made the language much more pleasant to use, and so the mention isn't very positive; instead, things like issues with the compilers available are noted, which made the software too slow to use in certain systems for which it was intended. Also listed is the ``Star Wars'' defense plan, which would be the most complicated piece of software ever written, like an even bigger version of a Patriot defense system. The latter pages of the chapter cover military accounting issues, including the 131 different and incompatible accounting programs in-use at the Air Force. While reading about it, I couldn't help but wonder how much of it were incompetence, and how much were cover for malice. The ninth chapter, ``A CASE OF MISTAKEN IDENTITY'', has a title which would've been better suited to the next chapter. It tells the harrowing story of the Aegis, named after Zeus's armor, installed on the USS Vincennes; the story slowly builds up over the chapter, starting with a ``fatal flaw'' later deemed ``a seemingly trivial design decision'' but seemingly forgotten as the story continues. As I was reading, dread hit me when I realized what had been happening, and that design flaw came back in full force to the fore of my mind. The chapter gives an extremely-detailed telling of the events on the ship, emphasized by an ominous foreshadowing with which some paragraphs begin, counting down the minutes until the disaster had hit. It was an excellent read which left me with the dissatisfaction of a bad situation where there was arguably no one to blame. A particularly good quote now follows: ``Incredibly, surrounded by $600 million worth of computers and electronics, Rogers has to order a crewman to start frantically thumbing through an ordinary paperback airline flight guide.'' This was a case in which the software worked correctly, but had a design that could've been improved to avoid misuse and misinterpretation. I found it very interesting to learn a tape containing a log of everything done was both available and analyzed. Crewman were found to have been panicking, such as with pressing the wrong buttons five and twenty-three times before they corrected their mistakes. The tenth chapter, ``OF CRIME AND PUNISHMENT'', details wrongful arrests caused by computer criminal records, and poor recordkeeping practices. It reads rather like a modern article decrying computers used in criminal identifications and arrests, but the state-of-the-practice was much worse then than nowadays. There were many different databases across the country tracking incompatible traits, poor logging on how they were accessed, and varying policy on how they could be cleared and updated. One police deputy spoke of how he ignores slight mismatches, for he expects the computers to be slightly wrong. The chapter goes on to explain the FBI's National Crime Information Center (NCIC) in detail, and how it stores a great amount of incorrect, outdated, and incomplete information on its subjects. A man demanded to have his entry updated after being held at gunpoint by police on several occasions due to a case of mistaken identity. All of this was worsened by such records having no photographs. The story of a retarded man who languished in prison for forty days, because his name wasn't entered correctly and thus couldn't be found by machine, was another example of such computer mismanagement. I thought it very interesting, how Janlori Goldman of the ACLU recommended a new governmental office to be established for data quality. Lastly, the passages in this chapter on the dangers of improper usage of all of this information for blackmail, employment checks, and whatnot feel quaint nowadays. The eleventh chapter, ``THE HUMAN ELEMENT'', is regarding human interface design and the like. This chapter correctly points out many differences between a computer and a human brain, something people still get wrong. Humans are overwhelmed by the sheer amount of data available, microscopic compared to the present, but machines can handle it with ease, albeit unintelligently. It notes many studies and research results that should be noted by interface designers, such as how the first suggestion a machine makes will be valued over others. Since machines are trusted to be correct, they must be if they're to be of use. The chapter laments that decision-making research has been made but not used. It also contains remarks from an Apple Computers employee, and notes on RSI and posture when using a computer. Repeated too is the idea that people must know the discipline for which they program, and programmers must also know how the programs will be used. Thus, this chapter is unlike many others. The twelfth and final chapter, ``NO MIRACLES'', contains a great chunk of information about remedies to the problems explained throughout the book, and a solemn warning that software is vital to nation and industry, which is already falling behind other countries on hardware. I was disappointed about some of the ignorance held in this chapter, as the author was apparently unaware of Edsger Dijkstra. At one point in the chapter, a Dr. William Wulf makes the following claim both ridiculous and false: ``knowing how to do it is still a deep intellectual problem. The proper foundation, even the appropriate mathematics, does not exist.'' I found it amusing that a programmer shortage was bemoaned even so long ago, the suggested solutions similar to the present day. Those solutions were using ``minorities'' in college programs, women in particular, to offset the supposed shortage. The chapter notes how programming has no education, no standards, and no practices, in general, and discusses the notion of professional certifications for software developers at some length, including how some programmers would rather move to other fields than accept such a state of affairs. AI research is also discussed a tad, in ways reflecting modern concerns about the unpredictability thereof. Those concerns of the US losing dominance in computing are well-justified in hindsight, and that situation has only worsened. The bills S.272 and H.R. 656 are the ``High-Performance Computing Act of 1991'' and mentioned in the final pages of the book, but not decided at the time of publishing; sponsored by Albert Gore, it ultimately passed, and I suppose improved the dire situation presented in the book, but I find it obvious that more needs to be done. Following is a list of topics mentioned in the final chapter that may make for interesting research: Japanese Daisys Japanese program to translate telephone calls in real-time Japanese Software Industrialized Generator and Maintenance Aids (SIGMA) European European Strategic Program for Research and Development in Information Technology (ESPRIT) European EUREKA DARPA prototyping The book was a wonderful read, albeit not without flaws, but my viewpoint is unconventional compared to most other programmers. I'm forced to wonder how he got so much detail for this book; it must've required months of research, poring over newspapers, interviews, and official reports; I'm impressed with it, and can easily and highly recommend others to read it; despite this, I'm still unaccustomed to reading such a book with such detail, and it forces me to question it to some degree, but I don't have particular issue believing what I read therein, despite issues I've noted throughout my review. The book ends with Wulf's lamentation, true: ``We are not going to find a quick and easy solution.'' .