NSA Scrapped an Effective, Legal Data Mining Program for an Ineffective, Illegal One

Siobhan Gorman writes in the Baltimore Sun that the NSA developed a telephone data sifting program in the late 1990s, called Thin Thread, that protected Americans' privacy and was highly successful at separating real threats from vast amounts of extraneous information.

But Thin Thread was never used, for political and bureaucratic reasons. After 9/11, another system -- favord by then-NSA director Michael Hayden -- was adopted. That system, called Trailblazer, had no safeguards against civil liberties abuses and was significantly less effective to boot.

The program the NSA rejected, called ThinThread, was developed to handle greater volumes of information, partly in expectation of threats surrounding the millennium celebrations. Sources say it bundled together four cutting-edge surveillance tools. ThinThread would have:

* Used more sophisticated methods of sorting through massive phone and e-mail data to identify suspect communications.

* Identified U.S. phone numbers and other communications data and encrypted them to ensure caller privacy.

* Employed an automated auditing system to monitor how analysts handled the information, in order to prevent misuse and improve efficiency.

* Analyzed the data to identify relationships between callers and chronicle their contacts. Only when evidence of a potential threat had been developed would analysts be able to request decryption of the records.
[...]
In what intelligence experts describe as rigorous testing of ThinThread in 1998, the project succeeded at each task with high marks. For example, its ability to sort through massive amounts of data to find threat-related communications far surpassed the existing system, sources said. It also was able to rapidly separate and encrypt U.S.-related communications to ensure privacy.

But the NSA, then headed by Air Force Gen. Michael V. Hayden, opted against both of those tools, as well as the feature that monitored potential abuse of the records. Only the data analysis facet of the program survived and became the basis for the warrantless surveillance program.

The decision, which one official attributed to "turf protection and empire building," has undermined the agency's ability to zero in on potential threats, sources say. In the wake of revelations about the agency's wide gathering of U.S. phone records, they add, ThinThread could have provided a simple solution to privacy concerns.

Digby wonders:

Is it possible that these people are actually working for Al Qaeda? It's almost impossible for anyone to fuck things up this consistently without consciously trying.

Or maybe these are people who simply enjoy having enormous power that that they can use to destroy lives or spare them, according to their whim.