Alkahest my heroes have always died at the end

It’s taken me a bit to write about Admiral Poindexter’s visit and the small group talk we had with him. Let me start by reminding folks that here’s a guy who was convicted of lying to congress. The conviction was later overturned on a technicality. He’s also very politically savvy. I once asked my father if he would ever pursue becoming a general in the army. He told me that he was hoping to make full colonel (he later retired as a lt. colonel), but that becoming a general required a literal act of congress and that you needed to become a politician. I would assume the same thing is the case with an admiral and doubly so in the case of Poindexter who managed to become the highest ranking geek in government. All of which is to say take my impressions with a grain of salt.

When I met Poindexter, he came across as a very kind, gentle and grandfatherly figure. He smokes a pipe and was more than willing to tell stories about his career. It seems that he started in the Navy in college, finishing up with a degree in engineering (w00t!). This was around the time the soviets sent up Sputnik. The first Russian satellite caused something of a panic in the US and, arguably, did more to encourage investment in science and engineering than any other event. The military’s response was to select 5 men from the army and 5 from the navy to pursue graduate degrees in science or engineering, anywhere in the country. Poindexter chose physics at Cal-tech. After discussing he trials getting into and then through grad school, he notes that he’s never taught physics, never been in a lab, never really used his degree, but it did give him a solid understanding of the scientific method.

After gradschool, he had several different positions and in each, he played the role of technology evangelist. One of the first to use computers in the Navy, set up the first video conferencing system among the nation security counsel offices, first to use email (on a mainframe!) in the whitehouse, etc. Like I said, the highest ranking geek in government.

Shortly after September 11, Poindexter was asked to head up the DARPA Office of Information Awareness (OIA) projects. In talking with him, I definitely have the sense of a man who loves his country and truly believes that terrorism is the greatest threat it has ever encountered. I disagree with him regarding the extent of the threat that terrorism presents, and so he and I may disagree on the appropriateness of the OIA, but unlike many politicians, I don’t think that he’s using the terrorism to advance other goals. I don’t believe that he’s hypocritical about his work.

So, what is his work? One of Poindexter’s chief complaints is that he (and TIA) were unfairly maligned in the media. If you recall, TIA was presented as a giant “Hoover” of a database. The government would collect information from a number of private sources and perform data mining on it in order to identify (potential) terrorists amongst us. Lots of us whom are concerned with security and privacy were worried about this. The privacy angle is disturbing enough, but from the security stand point, you are creating an attractive nuisance. The first hacker that comes along and can get through the governments security measures is going to have a huge amount of data. Consolidating databases also increases the likelihood that the businesses involved will use the information. For example, can you be denied insurance if you are overweight, but grocery records indicate you buy junk food?

Beyond the privacy and security concerns was the very real question of how this was going to work, i.e., would it really keep us safer? Traditional data mining techniques find statistically significant patterns in large data sets. Terrorists (one hopes) are not statistically significant – unless there are a lot more of them. This is actually one of Poindexter’s complaints – that his proposal should never be called data mining, data mining won’t work. He was working on a “data analysis” system.
In his presentation, Poindexter tells us that the media got it wrong. He never planned a single huge database. Instead, he planned to leave the data where it was and to build a distributed database on top. Each participating database would make use of a “privacy appliance.” The privacy appliance would be connected to a query system and would anonymize the data before sending it to the query system.

To detect terrorists, he would have a “Red Team.” This is the group that is intended to think like terrorists. Their job is to hatch plots and to determine what it would take to implement the plots. For example, blowing up a building might require large amounts of fertilizer and fuel oil. Purchasing these supplies would leave a footprint in “information space.” The Red Team would pass this step along to the analysts who would then query the system with this pattern to find anonymous individuals matching it. Of course, purchasing fuel oil and fertilizer would flag every small farmer in the country. So the Red Team would go back and look at step two, perhaps renting a large van. New query pattern, new search. Repeat until you either don’t find anyone, or until you are specific enough to get a legally authorized search warrant.

Poindexter also notes that this was a research and not an operational program. That the “total” in TIA was meant to encourage researchers to think broadly. Finally, that the reason the privacy part did not get off the ground sooner is that none of the researchers were interested in this aspect – they only received two privacy proposals.

Interesting idea. A few problems:

1. I’ve gone back through the documentation available at the time and I see nothing about either red teams, distributed databases or privacy appliances. The early architecture diagrams all seem to indicate a monolithic database.
2. It’s still not clear to me that this will work. The red teams will have to come up with millions of patterns and even then, you are not guaranteed to come up with everything.
3. Regarding research vs. operational. This is a lovely thought, but at the time, iirc, there were reports of TIA receiving real data. In fact, even as a research project, it would need real data in order to test.
4. Regarding the “total” in TIA – that was a pretty scary logo if that was the case.

So, it may be that this is a refinement of the original ideas. In which case, they seem like a good refinement. From the privacy and security standpoint, this seems to be better suited that the original ideas. However, I don’t think that Poindexter was being entirely forthcoming.

All in all, a very interesting data and a very interesting man.