A detailed report on the Bush administration’s loss of millions of emails has been greeted as little more than a post mortem from several years ago. Even if it was only that, though, it would deserve much more coverage than it has received – and it has serious implications for the present and future as well.
For more on pruning back executive power see Pruning Shears.
No Associated Press content was harmed in the writing of this post
On Monday Citizens for Responsibility and Ethics in Washington (CREW) released a detailed report (pdf) on the Bush administration’s loss of millions of emails. They inherited a system called ARMS (Automatic Records Managements System) that met the requirements of the Presidential Records Act (PRA), but tried to move to a manual system. That alone seems like a willful violation of the PRA’s spirit. Anyone who manages a lot of files on a large hard drive or a big mailbox will sooner or later bump up against the concept of functionally lost: You know it’s in there somewhere but you either can’t find it or can’t get to it.
These are ordinary considerations for even relatively inexperienced IT professionals. So is the concept of testing, but the new system began being used without it. Standard practice is to set up a parallel environment, have a QA team try it out as a simple proof of concept, work out the kinks, have a handful of “real” users take it out for a spin, and manage the rollout in a way that no one starts using the production system until the test system has proved its reliability. This is elementary project management.
The post-ARMS system required manually backing up each user’s mailbox data, which Microsoft Outlook stores in what is called a PST file. The pitfalls of this are obvious: If Johnny backs up everyone’s PST files tonight they might go to one location, and if Jenny backs them up tomorrow another. One set of possibilities for each operator initiating the backup. That is just for one night’s backups, too.
Backups happen each night. As weeks and months go by workers leave for new jobs and new ones come on board. Pretty soon it is impossible to know where, say, Lewis Libby’s PST file from October 6, 2003 might be. It’s on ONE of these tapes, in ONE of these directories (functionally lost). At another point the administration stopped backing them up because of a legal ambiguity, so the files never got cleaned out and just kept getting larger. Eventually they were too big for the system to process and no one could read them. It’s in the file SOMEWHERE but it’s too big to restore and look through (functionally lost).
Managing the PST files turned into a circus. Trying to make sense of it allowed a menagerie of hapless contractors and ripoff artists to dive into the money pit and emerge with ever more complex and arcane non-solutions aimed only at further extending the Rube Goldberg contraption. The PST files were a kludge, and these opportunists were not trying to get onto a proper system but just offering kludge wrappers.
Familiar names in the defense industry like Booz Allen and Northrop Grumman got into the act, as did a whole host of IT vendors both small and large. By the time InfoReliance comes on the scene with its PST Inventory Verification and Investigation Tool it seemed like Washington contractors had figured out some modern version of hobo signs to let everyone know where a free lunch was available.
This is not just a reheated helping of the toxic stew of incompetence and criminality from the previous administration, either. If nothing else fidelity to the historical record ought to make finding and restoring as much data as possible a top priority. However, even for those in the “keep looking so resolutely forward we do not learn from the past” camp there are reasons to acknowledge it.
For example, Congress could extend the PRA to include standards for data and application migrations. Federal offices all the way through the White House should have automated record preservation systems in place, and should be enjoined from moving a new system into production until it has been satisfactorily demonstrated in a test environment.
Also, the Keystone Kops efforts to restore data would not have happened if proper disaster recovery procedures were in place. Twice a year (minimum) all high level IT departments should engage in a full offsite disaster recovery exercise. Assume a tornado went through the data center; how do you get everyone back up and running? Doing that would have identified the hazards of PST purgatory relatively early, and given them a chance to correct it.
Those are just two quick examples. The IT environment revealed in the CREW report was unworthy of a boiler room, much less the highest governmental offices in the country. Simply correcting the problems of the Bush administration is inadequate. New policies need to be drawn up and procedures implemented. The entire operation needs to be systematized and formalized. The alternative is to leave yet another part of the government shielded from sunlight.