Real Intelligence for National Security
Manny February 23rd, 2007
It has been an exciting and busy time since my last posting! For one, it was wonderful to visit London again. I was also in London right after the subway bombings a couple of summers ago. Then as now, I deeply admire the British stiff upper lip, no matter if facing WWII bombs from above or today’s terror within. I had been thinking of what to write next regarding national security as promised, but clearly, the topic of counter terrorism is a matter of international concern. The Brits are wonderful in showing how daily life goes on despite being on the front line of this new war. However, the world has continued to change in this new millennium, while we continue to rely on rather brutish technology.
Listen. I work for the CIA. I am not a spy. I just read books. We read everything that is published in the world, and we feed the plots, dirty tricks, [and] codes into a computer. And the computer checks against actual CIA plans and operations. I look for leaks. I look for new ideas. We read adventures and novels and journals. I…… Who would invent a job like that?
Three Days of the Condor, 1975
I love this quote! The job of an intelligence analyst was crazy even in 1975. Today, it is nearly impossible, and new technology is needed to help read an ever-growing amount of data. It is not just about reading books these days. The volume of the Web, including “deep Web” chat rooms, blogs, and more is completely overwhelming. In my last posting, I introduced “real intelligence” as memory-based reasoning. International security needs real intelligence to help read, comprehend, and remember everything so that analysts do not have to.
I recently read that Tom Fingar, the Deputy Director of National Intelligence for Analysis, declared “It isn’t intelligence until it has been processed through the brain of an analyst.” Until then, it is all just data. We are drowning in data and do not have enough analytical brains. Technology is required to help. But why do we still use the same old technology when we are fighting a new kind of war? It is a new millennium with a new kind of warfare, and yet we still use questionable 20th century technology like rule-based systems and data mining. Remember my first posting about Tukey and his doubts about statistical inference. The joking derision, “That is so 20th Century!”, can also be said (and often is said) against databases, rule-based systems, search engines, and data mining. Because I am a biologist and psychologists, even when data mining is based on “neural networks”, I believe this have little if anything to do with real neurons and real intelligence.
In a recent paper from the CATO Institute Jeff Jonas and Jim Harper highlighted that traditional data mining has failed to address the needs of the intelligence community. Statistical reasoning about population distributions and abstract generalities is irrelevant when looking for rare adverse events in a dynamic sea of normality. One of their arguments, to which I agree, is that data mining is very data “hungry” and requires enough historical examples for its notions of statistical power and significance.
I agree that the data mining methods have been marginally effective even for commercial problems, such as for predicting consumer behavior, and have largely failed when re-applied to the harder challenges of intelligence analysis. On the other hand, memory-based reasoning is a different type of data analysis, more akin to reasoning in real brains rather than reasoning by rules and statistics. The memory-based approach to data analysis does not share these problems, such as being data hungry. Real intelligence reasons by similarity to specific experiences. If I touch a glowing stove once, I get burned and learn not to touch it again. The use of more or less data is irrelevant to this kind of experience-based reasoning.
David Aha’s edition on “lazy learning” defines a class of machine learning methods within this new approach. They include memory-based, case-based, experience-based, and instance-based reasoning. This class is distinguished from “eager” learners such as traditional statistics and neural networks which try to fit data to a particular model. This fitting of data leads to all sort of problems, including complex parameterizations, dependency on the order of data arrival, and over-fitting the data to the model. For example, if you believe in model fitting, you must worry about over-fitting.
The many problems of fitting data to models are understood in statistical modeling, but the same it true for rule-based inferencing as just another kind of modeling. For example, one rule-based system for identity management (including alias detection) attempts to make decisions about aliases and name variants when given each new piece of identity data. Its rule-based reasoning leads to decisions about combining identities that are order dependent, and the consequences of these decisions are then impossible to untangle. These models become increasingly wrong and impossible to fix. There is an enduring hope and occasional claim that rule-based and statistical systems will become fully incremental and “sequence neutral” one day. This is a naive hope.
Suppose I gave you the mean of a distribution (of 10 numbers, let’s say, but I do not provide you with the size of the distribution). An average number is useful and can be used for decision making, such as deciding on whether something is below or above average. However, if I give you another number and ask you to update the average, you do not have sufficient information to do so. It is impossible to update even this simple statistic as new data arrives. And this is a very trivial case. The problem of incremental change becomes increasingly difficult with greater scale and complexity and will never be addressed by reductionistic statistical models, neural networks, and rules. Decision rules are too eager and confound what they know with what they have already decided to do about what they previously knew, which is wrong and even dangerous.
Memories on the other hand are perfectly incremental and order independent. They store association counts; counts are perfectly incremented with each and every observation. Any order of data arrival results in the same final count. In the case of alias detection, memories store all the co-incident information about each identity and can quickly recall similar entities when asked (using k-nearest neighbors, for example). The answers might change as new data arrives, but any decisions made about entity similarity do not confound the information about similarity itself. A memory updates its information about new data but does not include decisions about how to fit the new data according to some a priori model. Lazy learners are called “lazy” because they adhere to this principle of minimum commitment. Memories read, comprehend, and remember all the information without reductionistic fitting to a particular model. This memory of detail makes them much more universal, stable and accurate.
We need this new approach for international security because we need more brains. To again quote Tom Fingar, “We don’t have enough analytical brains to meet all of the challenges. We have to rely on technology.” So why don’t we add more intelligent brain-like technology? Brains learn and think much more quickly and fluidly than traditional technology. For example, our brains do not learn by the notoriously slow, parametric, data-hungry methods of data mining - even methods called “neural” networks. Instead, we learn by memory: comprehending new information by instant integration with what we already know. The best analytic minds do not fit new data into a single a priori model confounded with priori decisions. They read and remember and then quickly reasoning to new situations as the world unfolds. Memory-based representation and reasoning will help us think about the harder questions we face today. As we have all read in the news the past year or more, over-eager analytical reasoning is disastrous!
We are still clinging to technologies that have failed and will continue to fail the intelligence community. I am passionate about this belief and will have more to say in my next posting. I will include more examples of these failures and how our new approach is succeeding. As the saying goes, “The proof of the pudding is in the eating.”
I also look forward to speaking about this at Saffron’s ExtremeIntelligence Manifesto event on March 7 in Reston VA. See www.saffrontech.com/events.shtml to register. We are already collecting a nice crowd, but I hope to see even more of you. It is a time for change.
