Cross posted a larger version at Daily Kos
This week we have had the following developments in the Able Danger story:
- Curt Weldon confesses his chart is a “reconstruction” and not the true chart (2)
- Thousands of false positives: NYT’s Philip Shenon said that Able Danger came up with names not just of Atta [but] with a tremendous number of names of very decent American citizens.(3)
- Lt Cmdr. Shaffer comes out of the woodwork saying he was one of the military talking about Able Danger to the 9/11 commission (1)
- The 9/11 commission admits it was warned but not by Shaffer (link pending)
- Lt Cmdr. Shaffer admits “We didn’t [think] that Atta’s name was significant” at the time (1)
- Strange Retreat from Right-wing bloggers (John Podhoretz, Jim Geraghty) who now appear to be backpedaling on Able Danger (5)
- Lt Cmdr. Shaffer said Thursday that many of his allegations were based not on his memory but on the recollections of others. (4)
(1) Officer Says Pentagon Barred Sharing Pre-9/11 Qaeda Data With F.B.I. – New York Times
(2) The Washington Monthly
(3) Eric Umansky: Thousands of False Positives?
(4) Claim about 9/11 takes new twist / Officer admits he based allegations on what others told him
(5) The “Two Atta” Theory – It might explain the Able Danger scandal. By Mickey Kaus
Would a good rule of thumb be: “Don’t go near anything out of Curt Weldon’s mouth — not even with a ten-foot pole ?”
P.S. War & Piece is all over this story … she’s been tracking every morsel. Scroll through.
Not the first time Weldon caught in a lie
Weldon Rev Moon and Khadafi – Weldon’s office denied that he spoke at Moon’s coronation, but had to backpedal as a picture of him surfaced (6)
I kinda wish these assholes would have the decency to at least make up lies that are half way plausible ..who you gonna believe…me or your lying eyes might be the caption for the pix of Weldon at the shindig for Mr. Loonie-Moonie.
This is getting more strange, more fun, all time.
Thank you for these updates.
Approximately at the time Able/Danger was using “advanced data-mining techniques” the Defense Science and Technology Organization’s Land Operations/Simulation division of the Australian Defence Forces was using advanced Object Oriented Development programming techniques to write training simulators for helicopter pilots. In order to approach reality as close as possible the programmers added native fauna – kangaroos – to the system.
From PC/AI Magazine
“Being efficient programmers, they just re-appropriated some code originally used to model infantry detachment reactions under the same stimuli, changed the mapped icon from a soldier to a kangaroo, and increased the figures’ speed of movement. Eager to demonstrate their flying skills for some visiting American pilots, the hotshot Aussies “buzzed” the virtual kangaroos in low flight during a simulation. The kangaroos scattered, as predicted, and the visiting Americans nodded appreciatively . . . then did a double take as the kangaroos reappeared from behind a hill and launched a barrage of Stinger missiles at the hapless helicopter.”
The Moral: Just because “advanced data-mining techniques” were used doesn’t mean the results were reliable.
Moral II: If you happen to be flying a helicopter in Australia watch out for kangaroos. They be Armed & Dangerous.
Here’s my and the experts take on using it Booman Tribune ~ Using Data Mining to Find Terrorists – ‘Able Danger’ explained
one of the drawbacks of data mining: It produces thousands of false positives.
Sure enough, someone on the blogosphere / MSM brought that up on the context of Able Danger:
Eric Umansky: Thousands of False Positives on Able Danger ?
The NYT’s Philip Shenon, who has done some of the Able Danger reporting, was interviewed Friday on WNYC. There host Mike Pesca raised the false positives question. Here’s what Shenon said:
“I understand from others at the Pentagon that one of the problems here is that Able Danger came up with names not just of Atta and three others, it came up with a tremendous number of names of very decent American citizens.”..
Shenon’s comment matches with my personal experience: data mining brings up a lot of false positives.
Those limitations are not only the musings of yours truly, but have also been raised by data mining experts such as Herb Edelstein , an internationally recognized expert in data mining and a co-founder of The Data Warehousing Institute
Data Mining In Depth: TIAin’t
False positives. Given the difficulty of developing good signatures and the small number of terrorists relative to the population of the United States, there are likely to be an enormous number of innocent people identified as potential terrorists (false positives). The more you try to avoid false positives, the more likely you are to miss many true positives. Unlike a direct mail campaign where the cost of a false positive is only a few dollars at worst, the costs in identifying terrorists – in dollars, time and wasted opportunity – are staggering. Suppose we had a collection of algorithms that has a false positive rate of only 0.1 percent – extraordinarily good for a problem of this complexity. That would mean 220,000 false positives! ..Even if we concentrated only on non-citizens, we would still have more than 20,000 false positives to be vetted.
(I didn’t link to your earlier diary because, well, I didn’t think about it. I did link when I commented elsewhere on BooTrib.)
Me wonders if the “publicly available database” was the Brooklyn Telephone Directory. That would help explain the 20,000 (or so) “false positives.” And, that term bothers me. Most of these systems (note the CYA!) return “Maybe positives” or “Perhaps positives” or “Kinda-sorta in a strange recursive way positives (even though you thought you fixed that bug.)”
But, anyway, let’s do some arithmetic:
20,000 is the figure thrown about so:
20,000 times 4 hours, average, to investigate = 80,000 hours.
80,000 hours divided by 24 gives 3,333 days.
3,333 days divided by 365 is 9.13 years, working 24/7.
So Able/Danger (Charlie Fox) would still be investigating the 20,000 potential terrorists but – don’t worry! – they will have finished before the end of the decade.
Even only 1 hour, on average, is needed that still results in 2.28 years working ’round the clock. So it still wouldn’t have done a bit o’ good.
Let me do some more CYA.
The above “calculation” is an extremely simplified model of the problem. The ‘calculated’ time is obviously for one person slogging through the process. More people would quicken the process but then one would have to calculate time spent in administration, data sharing, report writing, and all the other ‘overhead’ of a multi-person endeavor adding 30%, or so, to the people/hours.
And this only deals with the 20,000 people Able/Danger found.
If we can say New York City resulted in 20,000 ‘hits’ then the other major population centers would also give approximately the same number. So the total data universe can be very roughly estimated (i.e., WAG) at 2,000,000 hits.
The 4 hour average is, again, a WAG.
Given the spurious numerical attribution this should not be taken to do anything but give an initmation of the scale and scope of the problem. Or…
YMMV. Here there be Dragons. Take with much salt. Do Not Pass Go nor collect $200.
20,000 hits on all US if we only look at foreigners. 200,000 if we include citizens (as London bombings tell us we should)
Now that’s a hilarious story!
dont mean to be shallow in my statement, but that is so funny!!!!!!!! refferint to the morals I & II
Shaffer said Thursday that his overall allegations were based on the recollections of himself and two others, Navy Capt. Scott Phillpott and a civilian employee of the former Land Information Warfare Center at Fort Belvoir in Virginia, whom he declined to identify. Phillpott could not be reached for comment.
Shaffer said that Able Danger, by analyzing publicly available databases, produced charts in “the late spring or summer of 2000” showing ties between suspected terrorists. Shaffer said that after the attacks, the civilian employee had shown him a chart allegedly from 2000 that purportedly identified Atta and three other hijackers.
Shaffer, who briefed the Senate Judiciary Committee on his allegations Thursday, said he recognized the charts from his work as a liaison between the Defense Intelligence Agency and Able Danger. But he said he was relying on the word of Phillpott and the civilian employee, who pointed to one of the charts and said, “We had them.”
Phillpott told the Sept. 11 panel in July 2004 that he recalled seeing Atta’s name briefly on an Able Danger chart in spring 2000, which was before Atta obtained a visa and entered the United States. The panel, noting a lack of supporting evidence, said Phillpott’s account “was not sufficiently reliable to warrant revision of the report or further investigation.”
Claim about 9/11 takes new twist / Officer admits he based allegations on what others told him
I mentioned this a long time ago but will repeat this story. I’m wondering what words, etc go into the data mining? I have signed at least a hundred petitions against bushco, have arab newspapers in my favorites that I read all the time, go to several other sites to do with Islam, have checked out the Koran at the library and some books on Islam….no big deal right.
Yet an odd thing happened about a year ago. I live in a small apartment complex(14 apt.)and one morning my doorbell rang. When I answered the door a middle aged man in slacks/tie was standing there. He told me he wanted to talk to the owner/manager which I replied that the we didn’t have a manager and the owner didn’t live here. He then said he was from the FBI(held up his badge)and asked me ‘who’ the owner was which from his earlier question I thought he already knew…I told him I could give him the owners name/phone number and asked what this was regarding. He said he couldn’t tell me and left.
He never did get a hold of the owner and he hadn’t talked to anyone else in the complex either..just an odd incident.
Move to Portland!
Just kidding.. You have done nothing wrong, right ? They may harass you a bit but without any proof of wrongdoing, they will probably leave you alone. After all 1.3 billion people read the Koran. They can’t arrest all of them!
I figure that the day they arrest Michael Moore and John Conyers is the day I have to start being afraid of saying what I think.
No my activism is strictly confined to the internet due to being partially disabled and not really able to do much public participating in anything.
When my sister came over that day she quite literally looked around the door and windows outside to see if there was anything that looked like a ‘bug’..ha ha
I’m not given to wild flights of fancy at all but as I said there was just something kinda odd about that little incident.
.
From my previous life as kossack – cached versions at Google
Accenture | Cheney | Halliburton | Ghost of Enron | SAIC & US Elections ◊ by creve coeur
MATRIX project has received grants and contracts in support from State of Florida, DoD and DHS. Speaking of data mining! In concern to privacy of data, Jeb answered: “We’ll hire extra state troopers to guard the property”.
Worthwhile to read through all info in relation to DoD & evote/Wolfowitz – Diebold/SAIC – Seisint/Accenture & Hank Asher – Navy Shipyards/Accenture
End of January 2005, I relaxed a bit with the NyBri/SusanG investigative journalism on Valerie Plame which evolved into the PropaGannon series of diaries.
~~~