by rba
ePluribus Media
In the age of instant communications, it is difficult to imagine a more frightening experience than to be isolated from the rest of the world, in the middle of one of the worst hurricanes ever to strike the United States. The entire communications network, the most important system in the matrix of response, was rendered functionally useless in less than 12 hours by Hurricane Katrina. No one had planned for that level of damage. No one.
Photo © NOAA.gov
The failure of that system rendered the entire panoply of responders deaf, dumb and blind. It is impossible to imagine the depths of rage, frustration, and depression that must have overwhelmed them. How could they tend to the stricken, find the lost and maintain some semblance of order in the middle of a hurricane?
The farther the sight, the nearer the rain.
Old sayings often have a grain of truth hidden somewhere inside. The grain of truth in the one above is that low pressure makes the air over the water clearer, bringing water and sky into sharp focus to the horizon. When the air becomes hyper-clear, experienced sailors know that a storm is coming and race to secure the ship.
Photo © NOAA.gov
On Saturday morning, August 27, meteorologists inside the National Hurricane Center confirmed that the hurricane was gaining strength, possibly to a Category 5, and would make a turn west-by-northwest and hit the Gulf Coast. Their projected time to the strike was as few as forty-eight hours, two hours less than the minimum time planners needed to secure the area in the forecast cone. The nearer the rain….
In New Orleans, Baton Rouge, Gulfport, Biloxi and points east, and in the areas deemed safe to the north, the nation’s lumbering bureaucracy prepared for the storm. Local, state and federal “resources” were moving into place, as they had done in Florida three days before. The proper forms, in the proper order, had been submitted, qualifying those in the projected path of the storm for disaster recovery aid. From parish to state, governor to president, to the massive Department of Homeland Security (DHS) — which by law assigned the task to the Federal Emergency Management Agency (FEMA), the fabric of disaster preparedness was being unfolded.
In government-speak, “assets” were being staged. Guided by the National Response Plan (NRP), the 40-odd agencies in the matrix of responsibility were meeting via conference calls every two to three hours starting on Saturday. People in locations from the Gulf Coast to Washington, D.C., sat at conference tables, talking to each other over a dedicated video teleconferencing system, itself a result of years of planning. Advanced communications among and between those agency heads and the people in the field shortened time to deliver services precisely where they were most needed. Put the best people for the job on a four-state picket line, start the long supply chains running, confirm implementation of the NRP Emergency Support Functions (ESF). Of the 15 critical needs in the midst of disaster, one of the most important is communications — ESF #2.
Anywhere/Anytime Network
We are in an age of ubiquitous global communication. We connect to the network using traditional land-line telephones, cellular and satellite wireless phones; we plug computers into that same network to connect to the Internet. In this the new century, it seems no place is safe from the sound of one-sided conversations. Our government agencies, law enforcement, emergency service personnel, and even the military use the same type of equipment as the public for general communications.
Amazing then that the construct of the network, the bricks and mortar of the system, hasn’t really changed that much in the past hundred years. Wires strung on poles still connect the voices of the world through a network of switches inside ugly block buildings in every city in America. We’ve added wireless technology, cellular and satellite, but those connect to the existing infrastructure. Signals sent and received over the air or through copper wire or fiber-optic cables are routed through complex switch gear to enable us to connect from point A to point B. A cellular phone call is routed through the air to the nearest tower, where the signal is sent down through the switch inside the concrete vault over the Public Switched Telephone Network (PSTN) to the recipient at the other end. Cellular or land-line, the signal is routed over fiber-optic cables and/or copper wires at some point along the transmission path. If the receiver is a wireless customer, the process works in reverse at the other end.
Wire-line systems are much more vulnerable to the weather than wireless. (How many times have you heard “the lines are down”?) But even the wireless infrastructure has its limitations. The system comprises towers topped with antennae and anchored in shacks at the base that house both the electrical and telecommunications components necessary to complete the call. In case of emergency, some have backup generators and uninterruptible power supplies (UPS). The generators, usually diesel- or natural gas-fired, will run from 24 to 48 hours, according to industry sources. In an abundance of caution, backup generators and UPS in flood-prone areas are usually located on platforms three to six feet above the floor. The enclosures themselves are generally made with concrete walls and floors and reinforced roofs and are impervious to most elements.
The Failure of the Comm Grid
“When stuff’s under water, electrical stuff doesn’t work,” he said. “Fundamentally, you are still dealing with the laws of physics.”
Monday morning brought the wrath and fury of the hurricane to the Gulf Coast. One of the cities most ill-prepared for the disaster was New Orleans, built below sea level and surrounded by levees. As the storm moved through, it dropped over eight inches of rain and brought winds in excess of 100 miles an hour. At 8:14 a.m. CDT, the flood wall along the Industrial Canal was breached; two other breaks later in the day added to the flooding.
By the time the storm had passed late Monday night, emergency personnel in the four-state swath of destruction, estimated at 90,000 square miles, realized that at all levels they had underestimated their needs by orders of magnitude. In addition to the more than 100-miles-per-hour winds that knocked out power to over 2 million people, the wireless network suffered the loss of 1151 cell sites. The flooding reached shelter structures in New Orleans, “drowning” the backup generators:
[Bell South] reported that 180 of its central office locations are currently running on generator due to a loss of commercial power in affected areas.
“This is not a normal hurricane restoration,” said Bill Smith, Chief Technology Officer. “For example, the water level in New Orleans is still rising and working conditions are extremely difficult. As the water recedes in Mississippi and Alabama, our normal restoration efforts are progressing.”
Long-distance switches were flooded and had to be powered down, and wireless sites with battery backup had no personnel or generators for recharging….
Customers in the Florida Panhandle have had difficulty placing and receiving calls, transmitting data and using the Internet because communications traffic is routed through New Orleans. Smith said BellSouth made some progress in Mississippi and Alabama, where flood levels were receding.
TechWeb, “Katrina Overwhelms Telecom Giants’ Contingency Plans” (Aug. 31, 2005)
Compounding the problem was the severed fiber-optic cable over the Lake Pontchartrain Causeway, the I-10 bridge that carried telephone, cellular and Internet traffic in and out of New Orleans and connected the city to the wider grid beyond. Nearly the entire communications system had been taken down. The approaching flood prompted the mayor and his staff and the police chief to relocate to the Hyatt Hotel, where, the Sept. 9 Wall Street Journal reported, they remained without communications capability for two full days.
The police department’s communications center flooded, leaving the police with only limited-range radios in the field. The few cell phones still working in the city were running low on batteries and could not be recharged because of the power outage. And backup generators, which initially kept some cell sites running, simply ran out of gas. Because of the volume of traffic generated both inside and outside the city, even those with cellular phones attempting to reach outside could not get through. With all the inbound calls from all over the country, that system quickly reached maximum capacity.
Even the National Guard was not safe from the flood:
The rescuers had to be rescued early into Hurricane Katrina relief operations as sudden flooding forced the Louisiana National Guard to airlift 150 troops out of its command center … to the Louisiana Superdome, where they re-established the command center….
The relocated center, operating on emergency generators, is relying on radio communications only, as the hurricane has rendered all land-line and cellular phones useless within the city.
Army Times, “Flooding forces relocation of New Orleans command center” (Aug. 30, 2005)
For all intents and purposes New Orleans had become a communications island.
Follow the Sound of My Voice
According to a memo from the Homeland Security Dept., the telecommunications “infrastructure in New Orleans, Biloxi, and Gulfport is considered to be total write-off.”
BusinessWeek, “After chaos, changes in calling?” (Sept. 2, 2005)
What would it feel like to be in the middle of a city still with 100,000 people, in the middle of a hurricane, in the middle of the night, and have no way to communicate with anyone beyond the immediate area? The single most important system, the one people used almost without thinking, was functionally useless: 9-1-1 was damaged and overwhelmed. They had to improvise:
Police in New Orleans, their main communications system knocked out, have been taking turns talking on a single radio channel with their walkie talkies. The Mississippi National Guard even resorted to ancient battlefield tactics, sending runners back and forth among commanders with information.
And a local sheriff, Sid Hebert of Iberia Parish, helped keep an ambulance company handling medical evacuations across southern Louisiana running by loaning it a portable command center.
“He personally drove it to (our headquarters). He got us back on the air,” said Richard Zuschlag, chief executive of Acadian Ambulance Service Inc.
The Associated Press, “Katrina Rescuers Improvise Communications” (Sept. 1, 2005)
The people who struggled to escape the rising waters looked to those law-enforcement officers for help in finding find shelter, medical care, food and water. Unable to connect with their respective commands, without knowing the situation in the rest of the city, they were forced to tell those able to walk that they were on their own. For four days the water rose, flooding more real estate, shrinking the city and pushing them back. When their cars ran out of gas or were under water, they commandeered others that were gassed and dry; and they used boats for rescuing those trapped in the floodwaters. New Orleans was in a state of organized chaos, with isolated islands of people attempting to do their job.
Only then, deep into the unfolding disaster, were people beginning to grasp the significance of not having immediate access to information, of not having a voice at the other end of the line. With the number of people trapped in homes, hospitals and on the street; the limited amount of communications between responders; and the lack of ability to implement a coordinated plan for rescue, it is amazing that so many were able to escape from New Orleans at all. The rules of society were held in place by a tenuous thread connecting all the citizens of the city, in uniform and out. But after four long days under the weight of pressure applied from within and without what was left of their hierarchy, the thread snapped.
Struggling to cope with widespread chaos in hurricane-ravaged New Orleans, Louisiana Gov. Kathleen Babineaux Blanco blasted telecommunications providers for the apparent collapse of the state’s wireless network.
“The communications network is completely gone,” Blanco said Thursday (Sept. 1)….
EE Times, “Louisiana governor blasts faulty wireless networks” (Sept. 2, 2005)
Communications were so impossible the Army Corps of Engineers was unable to inform the rest of the government for crucial hours that levees in New Orleans had been breached
“No one had access…. No one had communication…. Nobody knew where the people were,” recalled Secretary of Health and Human Services Mike Leavitt….
….National Guard officials in Louisiana and Mississippi had no contingency plan if they were disrupted; they had only one satellite phone for the entire Mississippi coast, because the others were in Iraq.
The Washington Post, “The Steady Buildup to a City’s Chaos” (Sept. 11, 2005)
Fault and blame for the various failures of govermental structures will be apportioned among those directly responsible for the planning, organization and logistics of disaster recovery. However that blame is apportioned, none should reach the street level. There, all available emergency responders went to work and did their jobs to the best of their ability. When facts are placed in the record, it may read that the “looter cop” with a laptop computer and television, an incident propagated throughout the media, was simply a man trying to restablish communications within his district.
When we asked what he would do if his network went down, a suburban police officer in Northern California had this reaction: His eyes flickered once, imagining the circumstance, then he looked straight back and said, “I don’t know.” There was panic in that flicker — fear of not being able to help someone in need.* (see note)
Two Steps Forward, One Step Back
We can lick gravity, but sometimes the paperwork is overwhelming.
Twenty-four hours a day, seven days a week, “five-nine uptime” (99.999%) was the rule for telecommunications services before Hurricance Katrina. That statistic should serve as a reminder of how difficult it is to design and build for that last thousandth increment of time. No system of any kind has ever been built — by any culture or nation in the world — that can withstand a true force of nature. In the recent past the United States has seen the island of Kauai decimated, thousands of square miles leveled by Mount St. Helens, and a wide area of Los Angeles severely damaged by an earthquake.
And in August 2005 a Category 4 hurricane brought in excess of 10 inches of rain and a storm surge in less than 24 hours, with high winds that knocked out power. The combination of lack of power and flooding made pumps useless, and the combination of high water and surge put enough pressure on the flood walls to cause two breaches, which caused further flooding that wiped out the backup generators and finally caused the entire comm network to fail. The final insult came when, at the storm’s height, flying debris pierced a radiator on top of the Entergy Centre office tower, causing most of the city’s emergency radio system to fail. But for a working communications network, those estimated 90,000 people stuck in New Orleans, and the thousands spread from Louisiana to Alabama, would have received help earlier.
The rage and frustration of the population is directed towards the failure of government’s ability to respond to a crisis. That rage should be tempered with the knowledge that those attempting to respond, within and without the damage zone, in too many cases were unable to talk to each other. Difficult at best to coordinate a volleyball team without a telephone, much less a damage zone of 90,000 square miles, with a population of over 2.5 million people requiring a response from 40 different agencies at three discrete levels of government.
Under the best of circumstances, the entire communications matrix would have had an added layer of satellite telephones. To augment or provide cover for the first responders alone would have required a minimum of 15,000 such phones, as well as a concomitant increase in the bandwidth capacity of the network. Yet even with that substantial investment, the balance of the network would still have remained down. No phone, cellular, or internet communications would have been available across the zone of destruction for the same four days.
The hardwired communications network collapsed as a result of a unique and specific chain of events that defines a common-mode failure. That type of problem does not have a fix.
* From a personal interview on September 11, 2005 with rba and a Northern California law enforcement officer.
ePluribus Media contributors, editors and fact checkers: JeninRI, DeFuning, Sue in KY, Cho, Standingup, Silence, Luaptifer
Photo Credits: NOAA
Come join us at the
ePluribus Media Community Discussion site.
Not finished reading yet. But I just have to say wow.
Thanks Susan. Nice to get praise from “home”.
So heartbreaking. Found pics of a cop carrying a gas can through the water to restart a hospital generator, the stories of medical staff trying to keep people alive at Charity and other hospitals, fire fighters struggling to put out fires, and regular folk grabbing boats.
In that first 48 hours from Monday through Tuesday night the lack of resources must just crushed all the responders.
In hindsight, it seems mad that the regional telecommunications hub was placed in the one city that was already known to have the bulls-eye for disaster painted on it.
So it would seem that lesson #1 is to relocate telecommunications hubs well inland from the coast. Multiple smaller hubs instead of one giant regional hub might also be advisable in areas prone to disaster.
Lesson #2 is back to the future: We managed to pull off the Normandy invasion using radio, and it’s high time to make radio the #1 backup system in emergencies. Any information on how HAM radio operators did in Katrina? I’d bet better than the telecommunications system did.
Lesson #3: Local officers in all response agencies must be trained and empowered to take the reins when communications go down. This should be anticipated, planned and trained for. I suspect that this is why the Coast Guard did so much better in Katrina than other agencies. This is how they operate.
Lesson #4: Runners carrying messages – this is more “back to the future” technology. In WWII, the army used motorcycles for couriers. I’d bet you’d have gung-ho young men (and women) lining up to learn how to ride dirt bikes to be emergency couriers (Many enlistees probably already know how to ride dirt bikes!). That sounds like even more fun than a day at the shooting range! 😉
Lesson #5: Local deployment of caches of supplies that local on the scene commanders can tap into. It’s shameful that emergency responders had to result to looting to get what they needed – good planning would have caches of supplies pre-deployed at locations accessible from all at-risk sites.
If a nobody like me can come up with common-sense ideas like this off the top of my head, I damned well expect to see this kind of thinking in any post-mortem done about Katrina, or there should be hell to pay.
There’s a reason so many Roman buildings and bridges are still standing 2000 years later – they were extremely overdesigned for the job they had to do. The Romans didn’t have the mathematical or theoretical tools to allow them to determine how light and flexible a structure they could get away with, but they did know from hard experience that from time to time buildings or bridges would fail. So they made them very, very strong. The designers of the cathedrals of Europe 1000 years ago learned the same lessons the hard way – those flying buttresses on Gothic cathedrals are there as insurance against collapse.
One of the great marvels of our age is our telecommunications system. We now have the mathematical and engineering tools to build it at the minimum cost using the fewest materials most efficiently. But there is something to be said for redundancy, for making systems more durable when threatened than they need to be, even if this is not the most economical or quickest way of doing business. If you don’t believe me, ask someone at NASA.
You’re right about the need for changes in system design, and that we have the technological ability to build a near fault-tolerant system. The question becomes are we willing to pay for it. That question came up and was answered by a few tech folk in the articles I read, and the answer was no. We still have wires on poles across the country, and building a water-proof, hurricane, earthquake-proof system would be prohibitively expensive.
Satellite phones alone in sufficient numbers would have significantly reduced the comm problem. But from what I’ve read the Mayor’s office only had four sat phones with dead batteries. That was then, and you’re right: this is now. All it takes is money and political will.
Awesome article!!
On dkos two weeks ago a participant in the planning for the response to a hurricane in New Orleans, the infamous “Hurricane Pam” exercises, put up some excellent posts regarding the attitudes during the sessions and the plans developed from those sessions.
I was suprised at the total breakdown incomm and the reliance on celluar phones. A hurricane is a windstorm, one would think it would be obvious cell comms would be severly compromised in such an environment. First from the wind, then from the lack of power. If nothing else, the blackout in the northeast in 2003 would have been a wake up call. Communications did not survive that event well, even without flooding.
There were also reports, unconfirmed and from less than reliable sources, of significant interference in the MF and HF radio bands. I tend to discount the intentional interference, but could it be atmospheric interference?
Again, a great look at massive systems breakdown.
thanks for the kudos. here’s one of the articles about the solar flares:
http://www.cellular-news.com/story/14013.php
So add one less common cause to the problem.
Cheers
Hurricane Pam Planning
Truly outstanding work.
Question: The United States faced the specter of nuclear war for several decades. No attempt was ever made to create a hardened back-up communications system for the authorities?
I read in the Washington Post that a European firm had offered to provide a turn-key cell phone system capable of supporting thousands of calls, which unfortunately waited on the tarmac for several days while the State Department dithered.
I don’t know about European system. Fairly sure the military’s comm system was hardened during the nuclear era (still is I imagine). Although such a system would have to be installed down to the City/County level. Which is where politics and policy crash into funding.
The problem here was the cascade of failures in rapid succession over a period of about 8-12 hours, incuding the police radio system almost totally wiped out by a shard of glass.
Also worth noting is the lack of national interoperability standards during a disaster. Good background article here from Government.com, Dec ’04.
Sorry it took so long to get back to you but here is the article.
I knew you all would come thru with a great article, as always. OUTSTANDING!!!!!!!!!!! Thanks for this ….
rba,
this is a fantastic diary, really good. Content aside, this is a very impressive collection of information with lots of links (i like that) and awesome photos of the hurricane. I’d give anything, almost ;o) to have half the writing talent that you do.
Thanks
Thank you. This one aspect of the disaster jumped out and didn’t seem to be covered anywhere.
Wow, excellent job! I hadn’t realized the widespread scope of the communications failures, although it was obviously an issue in NO.