Making money by data mining voters

James Verini of Vanity Fair writes about "Big Brother Inc.":

Knowing your business is big business for Aristotle Inc., whose Orwellian database of voter records has been an essential campaign tool for every president since Ronald Reagan. As the 2008 race heats up, the company’s shadowy founder, John Aristotle Phillips, unveils his most powerful personal-space invader yet.

I remember reading about Phillips during my days as an undergraduate at Boston University studying political science. Phillips won fame back then for drawing up plans to make a nuclear bomb while at Princeton and for nearly winning a race for Congress while in his early 20s.

The article notes that in 2003 he sued Kentucky for access to its voter list. The writer's point of view (or at least the editor's) is obvious from the title, "Big Brother Inc." and the use of words like "Orwellian" and "shadowy," but my thought throughout was: What's so wrong with making it easier for candidates to find like-minded voters? Isn't that the essence of democracy? For me, the article didn't make the case that this is a bad thing.

Text Analytics Wiki

... introduces itself:

This wiki aims to be a one-stop site for everything related to Text Analytics (also known as Text Mining or Information Extraction). It aims to go well beyond the limits of Wikipedia to provide links to people, organisations, the latest research and news.

I have it in my head that text analytics could be useful to a computer-oriented reporter such as myself searching for meaning in documents, but first someone will need to implant a microchip in my brain that will allow me to understand the algorithms involved.

Read/WriteWeb: "10 Semantic Apps to Watch"

The Read/WriteWeb defines a semantic application as an application that tries "to determine the meaning of text and other data, and then create connections for users." Here's its list of "10 Semantic Apps to Watch."

Evolution personal data mining software

Evolution is software that searches online sources, such as social networking sites, Wikipedia, Google Books and phone directories, for names, numbers, email addresses, phrases and Web domains, and graphically displays the links between them. I installed it after reading a Linux.com article that called it "a kick-ass application, just seething with power and potential":

Still don't grok it? Think of the NSA sifting through network traffic, looking for actionable intelligence. Or if that's too conspiracy-minded for your taste, think of trying to find something new and meaningful in the results of a Google search on Paris Hilton. Evolution is kind of like that, but more aggressive in finding results, and a lot more aggressive in trying to make sense of them.

I searched for just my name and it quickly found my telephone number, my email addresses, my wife, open record appeals I had written, this blog and my personal Web site, references to me on other sites, my latitude and longitude and home city, Google Book hits citing my name and more. You can then repeat searches (called "transforms") on entities it finds, exploring potential relationships. It also turned up a lot of information unrelated or only marginally related to me, and of course it's a lot easier to distinguish the relevant from the irrelevant if you're researching yourself. If you start cold on someone you don't know much about, you're going to have to do a lot more leg work to nail down what's meaningful and what isn't. Still, I was impressed. It runs on Windows, Macintosh and Linux. The Linux.com article quoted the creator, Roelof Temmingh, saying he's undecided about what to do with the Evolution, which is free, at least for now, and still in beta: "He said that he needs to make some money from Evolution or it will die," the article said. "He is considering everything from advertising to subscriptions, or selling the GUI and transforms, or selling only the GUI and making the transforms open source, and he is open to other suggestions."

UPDATE: You can no longer download Evolution from its Web site. Linux.com reports that the creator announced he had removed it "due to circumstances outside of my control. I am not sure how long this outage will last, but perhaps it will be permanent." The site now says that if you want to see what Evolution can do, contact evolution@paterva.com.

A "jaw-dropping" visual demonstration

I've seen a lot of presentations and it's rare that one generates spontaneous applause, as did this demonstration of Microsoft software called Photosynth:

Injecting science into the social

When I was a political science and economics major in the early 1980s, both fields typically constructed elaborate theories on sand -- whether it was Marxists spinning elaborate superstructures whose inner contradictions soon became only too apparent, or economists building economic models that always assumed away everything interesting and difficult about the world. It's heartening, therefore, to read that both fields are becoming more empirical, a trend driven by cheap computers and the vast quantities of data now being generated:

The quantitative social sciences are in the midst of a revolution in understanding the world and solving real problems. A dramatic increase in progress is now achievable because: 1) changes in technology enable us to collect and store unprecedented amounts of far more informative data about human populations and institutions; 2) new policies encourage the collection of data and its provision to researchers, including the computerization and automation of many government services, new data collection requirements, e.g., the No Child Left Behind Act, and the growing movement in science to make data publicly available; and 3) the development of novel methods of data collection and analysis that make it possible for scholars to extract information from new data, such as from the rise of social experiments that enable reliable causal inferences in major issues of public policy, natural language processing that enables scientists to extract information from millions of Web sites, newspapers, emails or other textual sources, and new informatics techniques that provide instant and reliably persistent, access to the world's data.

So says Gary King, the directory of the Institute for Quantitative Social Science at Harvard, in an interview for the Data Mining Review. Some of that data and a tiny fraction of the techniques he discusses may one day be adopted by journalists, who are the protozoa of the fact-based universe. More than a few gut-trusting editors, incidentally, would just guffaw at this quote from King: "The central conclusion of research in hundreds of fields and most of the hundreds of thousands of applications is the same: Whenever a sufficiently important fraction of information can be quantified, statistical analysis beats qualitative human judgment. There is just no contest."

Data mining middle school

A Louisiana middle schools supervisor uses SPSS data mining software to analyze disciplinary reports, InformationWeek reports:

A search for the word "uniform," for example, might show if teachers or schools are being overly strict in enforcing dress codes, while a search on "tardy" could identify habitually late students. Bowman can search the records of a student, school, or the entire district.

Bowman combines those findings with demographic data to look for additional patterns. "I'm trying to look beyond the obvious and find out if there are hidden factors in why things happen," he says. Are kids from low-income homes, for example, disciplined for dress code violations?

Government Data Mining Programs

Newsinitiative.org explains interactively how the government gathers data about you and how the information is used.

America's Hidden History of Racial Expulsions

Elliott Jaspin of Cox News Service used Census records dating back to the Civil War and years of research to document "America's Hidden History of Racial Expulsions":

Beginning in 1864 and continuing for approximately 60 years, whites across the United States conducted a series of racial expulsions, driving thousands of blacks from their homes to make communities lily-white.

In at least a dozen of the most extreme cases, blacks were purged from entire counties that remain almost exclusively white, according to the most recent census data.

Some of the places Jaspin highlights in "Leave or Die" are in Kentucky and Indiana:

Marshall County, Ky., where in 1908 vigilantes led by a local doctor posted notices telling blacks to leave. More than 100 armed and hooded men raided the town of Birmingham, picked about a dozen people at random and tortured them. An elderly black man and his two-year-old grandchild were killed. Nearly two-thirds of the blacks left. ...

Laurel and Whitley, neighboring counties in Kentucky, where in 1919 whites, believing that the arrival of a black railroad construction crew had spawned a crime wave, rounded up blacks at gunpoint, herded them to the train station and forced them to leave. ...

Washington County, Ind., where blacks were driven out between 1864 and 1867, apparently by whites alarmed that the Emancipation Proclamation could allow blacks to vote and become full citizens. Two black men who did not leave were killed. ...

Vermillion County, Ind., where in 1923 the then politically powerful Ku Klux Klan drove the expulsion of blacks from the mining town of Blanford after a white girl said she was assaulted by a black man.

Data mining database archive

The UCI Knowledge Discovery in Databases Archive is a repository of large, complex data sets for doing research in data mining.

Mathematician says data sweeps will be ineffective

John Allen Paulos, the author of A Mathematician Reads the Newspaper and other books, uses a little math to argue that massive government data mining sweeps will be an ineffective way to bag terrorists:

Even if an accurate profile of potential terrorists is drawn, the fact that such a vanishingly small percentage of us are terrorists means that the vast majority of the people investigated will be innocent.

Even if the probability that the purported terrorist profile is accurate were an astonishing 99 percent (if someone has terrorist ties, the profile will pick him or her out 99 percent of the time, and, for ease of computation, if someone does not have such ties, the profile will pick him or her out only 1 percent of the time), most of the hits would be false positives.

For illustration, let's further assume that one out of a million American residents has terrorist ties -- that's approximately 300 people -- and the profile will pick out 99 percent, or 297 of them. Great. But what of the approximately 300 million innocent Americans? The profile will also pick out 1 percent of them, "only" 3 million false positives, innocent people who will be caught up in a Kafkaesque dragnet.

Predicting company performance by data mining the Web

Reuters reports that a New York company, Majestic Research, is gathering freely available information on the Internet to predict the sales and financial performance of companies. " ... several dozen math Ph.D.s, statisticians and other quantitative analysts evaluate data spewed from computers using 'Web crawling' programs track sales and other information from tens of millions of Web pages or other on-line resources," says the article by Dane Hamilton. " ... The firm's methods differ from traditional Wall Street research, where analysts make forecasts based on conversations with company executives, advertisers, suppliers and mall visits to forecast company results and make recommendations."

Data mining using Google

A blogger explains how to do data mining with Google.

Visualizing the Bible

Exegesis graphically represents how often particular Bible passages are found on the Web, as measured by Google searches. But you must have Java installed and the site warns it takes a while to load and works better with a fast computer. If that interests you, you'd also be interested in Gospel Spectrum, which uses "data visualization and data-mining to deconstruct and then visually reconstruct the story of Jesus as presented in the Bible." (I learned of both of these courtesy of the information aesthetics weblog.)

About.com offers an introduction to data mining. Techniques from data mining will one day make their way into the computer-assisted reporting toolkit.

Data mining is how government, business and academia extract useful information from large databases but it's mostly virgin territory in the computer-assisted reporting world. Here's a Congressional Research Service report that gives an overview of the subject.

eWeek reports that a California company is using data mining to help retailers target shoppers who abuse store return policies:

Here’s how it’s supposed to work: A customer walks in and attempts to return a product. The clerk asks for identification and enters that into the system so that all of that customer’s purchases can be linked.
The identification information and the return data is automatically sent (either using dial-up, a broadband VPN or a direct T1 connection) to a database that The Return Exchange has set up exclusively for that particular retail chain. That database can be accessed—on The Return Exchange’s server—by anyone in that company’s IT department.
When that database (called Verify-1) sees what it considers to be a return abusive pattern, it will reject that return, in the same way that a POS would reject a stolen credit card. The clerk then would give the customer an 800 number to The Return Exchange, which would then investigate the case.