by rba
ePluribus Media
In the age of instant communications, it is difficult to imagine a more frightening experience than to be isolated from the rest of the world, in the middle of one of the worst hurricanes ever to strike the United States. The entire communications network, the most important system in the matrix of response, was rendered functionally useless in less than 12 hours by Hurricane Katrina. No one had planned for that level of damage. No one.
Photo © NOAA.gov
The failure of that system rendered the entire panoply of responders deaf, dumb and blind. It is impossible to imagine the depths of rage, frustration, and depression that must have overwhelmed them. How could they tend to the stricken, find the lost and maintain some semblance of order in the middle of a hurricane?
The farther the sight, the nearer the rain.
Old sayings often have a grain of truth hidden somewhere inside. The grain of truth in the one above is that low pressure makes the air over the water clearer, bringing water and sky into sharp focus to the horizon. When the air becomes hyper-clear, experienced sailors know that a storm is coming and race to secure the ship.
Photo © NOAA.gov
On Saturday morning, August 27, meteorologists inside the National Hurricane Center confirmed that the hurricane was gaining strength, possibly to a Category 5, and would make a turn west-by-northwest and hit the Gulf Coast. Their projected time to the strike was as few as forty-eight hours, two hours less than the minimum time planners needed to secure the area in the forecast cone. The nearer the rain….
In New Orleans, Baton Rouge, Gulfport, Biloxi and points east, and in the areas deemed safe to the north, the nation’s lumbering bureaucracy prepared for the storm. Local, state and federal “resources” were moving into place, as they had done in Florida three days before. The proper forms, in the proper order, had been submitted, qualifying those in the projected path of the storm for disaster recovery aid. From parish to state, governor to president, to the massive Department of Homeland Security (DHS) — which by law assigned the task to the Federal Emergency Management Agency (FEMA), the fabric of disaster preparedness was being unfolded.
In government-speak, “assets” were being staged. Guided by the National Response Plan (NRP), the 40-odd agencies in the matrix of responsibility were meeting via conference calls every two to three hours starting on Saturday. People in locations from the Gulf Coast to Washington, D.C., sat at conference tables, talking to each other over a dedicated video teleconferencing system, itself a result of years of planning. Advanced communications among and between those agency heads and the people in the field shortened time to deliver services precisely where they were most needed. Put the best people for the job on a four-state picket line, start the long supply chains running, confirm implementation of the NRP Emergency Support Functions (ESF). Of the 15 critical needs in the midst of disaster, one of the most important is communications — ESF #2.
Anywhere/Anytime Network
We are in an age of ubiquitous global communication. We connect to the network using traditional land-line telephones, cellular and satellite wireless phones; we plug computers into that same network to connect to the Internet. In this the new century, it seems no place is safe from the sound of one-sided conversations. Our government agencies, law enforcement, emergency service personnel, and even the military use the same type of equipment as the public for general communications.
Amazing then that the construct of the network, the bricks and mortar of the system, hasn’t really changed that much in the past hundred years. Wires strung on poles still connect the voices of the world through a network of switches inside ugly block buildings in every city in America. We’ve added wireless technology, cellular and satellite, but those connect to the existing infrastructure. Signals sent and received over the air or through copper wire or fiber-optic cables are routed through complex switch gear to enable us to connect from point A to point B. A cellular phone call is routed through the air to the nearest tower, where the signal is sent down through the switch inside the concrete vault over the Public Switched Telephone Network (PSTN) to the recipient at the other end. Cellular or land-line, the signal is routed over fiber-optic cables and/or copper wires at some point along the transmission path. If the receiver is a wireless customer, the process works in reverse at the other end.
Wire-line systems are much more vulnerable to the weather than wireless. (How many times have you heard “the lines are down”?) But even the wireless infrastructure has its limitations. The system comprises towers topped with antennae and anchored in shacks at the base that house both the electrical and telecommunications components necessary to complete the call. In case of emergency, some have backup generators and uninterruptible power supplies (UPS). The generators, usually diesel- or natural gas-fired, will run from 24 to 48 hours, according to industry sources. In an abundance of caution, backup generators and UPS in flood-prone areas are usually located on platforms three to six feet above the floor. The enclosures themselves are generally made with concrete walls and floors and reinforced roofs and are impervious to most elements.
The Failure of the Comm Grid
“When stuff’s under water, electrical stuff doesn’t work,” he said. “Fundamentally, you are still dealing with the laws of physics.”
Monday morning brought the wrath and fury of the hurricane to the Gulf Coast. One of the cities most ill-prepared for the disaster was New Orleans, built below sea level and surrounded by levees. As the storm moved through, it dropped over eight inches of rain and brought winds in excess of 100 miles an hour. At 8:14 a.m. CDT, the flood wall along the Industrial Canal was breached; two other breaks later in the day added to the flooding.
By the time the storm had passed late Monday night, emergency personnel in the four-state swath of destruction, estimated at 90,000 square miles, realized that at all levels they had underestimated their needs by orders of magnitude. In addition to the more than 100-miles-per-hour winds that knocked out power to over 2 million people, the wireless network suffered the loss of 1151 cell sites. The flooding reached shelter structures in New Orleans, “drowning” the backup generators:
[Bell South] reported that 180 of its central office locations are currently running on generator due to a loss of commercial power in affected areas.
“This is not a normal hurricane restoration,” said Bill Smith, Chief Technology Officer. “For example, the water level in New Orleans is still rising and working conditions are extremely difficult. As the water recedes in Mississippi and Alabama, our normal restoration efforts are progressing.”
Long-distance switches were flooded and had to be powered down, and wireless sites with battery backup had no personnel or generators for recharging….
Customers in the Florida Panhandle have had difficulty placing and receiving calls, transmitting data and using the Internet because communications traffic is routed through New Orleans. Smith said BellSouth made some progress in Mississippi and Alabama, where flood levels were receding.
TechWeb, “Katrina Overwhelms Telecom Giants’ Contingency Plans” (Aug. 31, 2005)
Compounding the problem was the severed fiber-optic cable over the Lake Pontchartrain Causeway, the I-10 bridge that carried telephone, cellular and Internet traffic in and out of New Orleans and connected the city to the wider grid beyond. Nearly the entire communications system had been taken down. The approaching flood prompted the mayor and his staff and the police chief to relocate to the Hyatt Hotel, where, the Sept. 9 Wall Street Journal reported, they remained without communications capability for two full days.
The police department’s communications center flooded, leaving the police with only limited-range radios in the field. The few cell phones still working in the city were running low on batteries and could not be recharged because of the power outage. And backup generators, which initially kept some cell sites running, simply ran out of gas. Because of the volume of traffic generated both inside and outside the city, even those with cellular phones attempting to reach outside could not get through. With all the inbound calls from all over the country, that system quickly reached maximum capacity.
Even the National Guard was not safe from the flood:
The rescuers had to be rescued early into Hurricane Katrina relief operations as sudden flooding forced the Louisiana National Guard to airlift 150 troops out of its command center … to the Louisiana Superdome, where they re-established the command center….
The relocated center, operating on emergency generators, is relying on radio communications only, as the hurricane has rendered all land-line and cellular phones useless within the city.
Army Times, “Flooding forces relocation of New Orleans command center” (Aug. 30, 2005)
For all intents and purposes New Orleans had become a communications island.
Follow the Sound of My Voice
According to a memo from the Homeland Security Dept., the telecommunications “infrastructure in New Orleans, Biloxi, and Gulfport is considered to be total write-off.”
BusinessWeek, “After chaos, changes in calling?” (Sept. 2, 2005)
What would it feel like to be in the middle of a city still with 100,000 people, in the middle of a hurricane, in the middle of the night, and have no way to communicate with anyone beyond the immediate area? The single most important system, the one people used almost without thinking, was functionally useless: 9-1-1 was damaged and overwhelmed. They had to improvise:
Police in New Orleans, their main communications system knocked out, have been taking turns talking on a single radio channel with their walkie talkies. The Mississippi National Guard even resorted to ancient battlefield tactics, sending runners back and forth among commanders with information.
And a local sheriff, Sid Hebert of Iberia Parish, helped keep an ambulance company handling medical evacuations across southern Louisiana running by loaning it a portable command center.
“He personally drove it to (our headquarters). He got us back on the air,” said Richard Zuschlag, chief executive of Acadian Ambulance Service Inc.
The Associated Press, “Katrina Rescuers Improvise Communications” (Sept. 1, 2005)
The people who struggled to escape the rising waters looked to those law-enforcement officers for help in finding find shelter, medical care, food and water. Unable to connect with their respective commands, without knowing the situation in the rest of the city, they were forced to tell those able to walk that they were on their own. For four days the water rose, flooding more real estate, shrinking the city and pushing them back. When their cars ran out of gas or were under water, they commandeered others that were gassed and dry; and they used boats for rescuing those trapped in the floodwaters. New Orleans was in a state of organized chaos, with isolated islands of people attempting to do their job.
Only then, deep into the unfolding disaster, were people beginning to grasp the significance of not having immediate access to information, of not having a voice at the other end of the line. With the number of people trapped in homes, hospitals and on the street; the limited amount of communications between responders; and the lack of ability to implement a coordinated plan for rescue, it is amazing that so many were able to escape from New Orleans at all. The rules of society were held in place by a tenuous thread connecting all the citizens of the city, in uniform and out. But after four long days under the weight of pressure applied from within and without what was left of their hierarchy, the thread snapped.
Struggling to cope with widespread chaos in hurricane-ravaged New Orleans, Louisiana Gov. Kathleen Babineaux Blanco blasted telecommunications providers for the apparent collapse of the state’s wireless network.
“The communications network is completely gone,” Blanco said Thursday (Sept. 1)….
EE Times, “Louisiana governor blasts faulty wireless networks” (Sept. 2, 2005)
Communications were so impossible the Army Corps of Engineers was unable to inform the rest of the government for crucial hours that levees in New Orleans had been breached
“No one had access…. No one had communication…. Nobody knew where the people were,” recalled Secretary of Health and Human Services Mike Leavitt….
….National Guard officials in Louisiana and Mississippi had no contingency plan if they were disrupted; they had only one satellite phone for the entire Mississippi coast, because the others were in Iraq.
The Washington Post, “The Steady Buildup to a City’s Chaos” (Sept. 11, 2005)
Fault and blame for the various failures of govermental structures will be apportioned among those directly responsible for the planning, organization and logistics of disaster recovery. However that blame is apportioned, none should reach the street level. There, all available emergency responders went to work and did their jobs to the best of their ability. When facts are placed in the record, it may read that the “looter cop” with a laptop computer and television, an incident propagated throughout the media, was simply a man trying to restablish communications within his district.
When we asked what he would do if his network went down, a suburban police officer in Northern California had this reaction: His eyes flickered once, imagining the circumstance, then he looked straight back and said, “I don’t know.” There was panic in that flicker — fear of not being able to help someone in need.* (see note)
Two Steps Forward, One Step Back
We can lick gravity, but sometimes the paperwork is overwhelming.
Twenty-four hours a day, seven days a week, “five-nine uptime” (99.999%) was the rule for telecommunications services before Hurricance Katrina. That statistic should serve as a reminder of how difficult it is to design and build for that last thousandth increment of time. No system of any kind has ever been built — by any culture or nation in the world — that can withstand a true force of nature. In the recent past the United States has seen the island of Kauai decimated, thousands of square miles leveled by Mount St. Helens, and a wide area of Los Angeles severely damaged by an earthquake.
And in August 2005 a Category 4 hurricane brought in excess of 10 inches of rain and a storm surge in less than 24 hours, with high winds that knocked out power. The combination of lack of power and flooding made pumps useless, and the combination of high water and surge put enough pressure on the flood walls to cause two breaches, which caused further flooding that wiped out the backup generators and finally caused the entire comm network to fail. The final insult came when, at the storm’s height, flying debris pierced a radiator on top of the Entergy Centre office tower, causing most of the city’s emergency radio system to fail. But for a working communications network, those estimated 90,000 people stuck in New Orleans, and the thousands spread from Louisiana to Alabama, would have received help earlier.
The rage and frustration of the population is directed towards the failure of government’s ability to respond to a crisis. That rage should be tempered with the knowledge that those attempting to respond, within and without the damage zone, in too many cases were unable to talk to each other. Difficult at best to coordinate a volleyball team without a telephone, much less a damage zone of 90,000 square miles, with a population of over 2.5 million people requiring a response from 40 different agencies at three discrete levels of government.
Under the best of circumstances, the entire communications matrix would have had an added layer of satellite telephones. To augment or provide cover for the first responders alone would have required a minimum of 15,000 such phones, as well as a concomitant increase in the bandwidth capacity of the network. Yet even with that substantial investment, the balance of the network would still have remained down. No phone, cellular, or internet communications would have been available across the zone of destruction for the same four days.
The hardwired communications network collapsed as a result of a unique and specific chain of events that defines a common-mode failure. That type of problem does not have a fix.
* From a personal interview on September 11, 2005 with rba and a Northern California law enforcement officer.
ePluribus Media contributors, editors and fact checkers: JeninRI, DeFuning, Sue in KY, Cho, Standingup, Silence, Luaptifer
Photo Credits: NOAA
Come join us at the
ePluribus Media Community Discussion site.
