The Wikipedia as structured data

The Wikipedia is an enormous blob of text. Because it isn't well-structured, like a database, it isn't easy to analyze or cross-match with other data. That's where the DBpedia comes in. "DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia and to link other datasets on the Web to Wikipedia data."

The DBpedia dataset currently provides information about more than 2.18 million “things”, including at least 80,000 persons, 293,000 places, 62,000 music albums, 36,000 films. Altogether, the DBpedia dataset consists of 218 million pieces of information.

Open source computer-assisted reporting

The biggest barriers to computer-assisted reporting are time and desire -- not money. Chase Davis, Matthew Waite and Derek Willis, who are attending the NICAR conference in Houston this weekend, list the free tools data-minded journalists can use instead of commercial alternatives, some of them costing thousands of dollars. Note that they're sharing the list on Google Docs -- also free (but not open source, as is the other software).

Berkeley/Penn Urban & Environmental Modeler's Datakit

... is free to download and consists of electronic map files "describing many of the physical, administrative, transportation, demographic, economic, land use and land cover, and environmental characteristics of the 48 contiguous United States." These include:

• Municipal, county, metropolitan area, and state boundaries


• Census block and census tract boundaries and selected attributes, including net housing and population densities.


• Highway, railroad, and urban rail transit networks; and air and seaport locations


• Locations of major employment centers and employment data for 1994 and 2003.


• Measures of 30- and 45-minute job accessibility for every location in the U.S.


• Boundaries of all federal lands, including national parks and monuments, national forests, Bureau of Land Management lands, and military facilities.


• Elevation and slope data generated from the National Elevation Dataset (NED).


• Comprehensive land cover data for 1992 and 2001 from the U.S. Geological Survey, including agriculture, forest, pasture, urban, and wetland locations.


• Locations of all major water bodies including rivers, streams, lakes, and reservoirs.


• Locations of lacaustrine, palustrine, and riverine wetlands from the National Wetlands Inventory.
  

Infographic movies

Information Aesthetics now has its own YouTube channel for infographic movies.

Fixing publication bias

Ben Goldacre of Bad Science notes recent studies exposing publication bias:

The key issue is simple. In any situation, to make any kind of sensible decision about which treatment is best, a doctor must be able to take into account all of the available information. But drug companies have repeatedly been shown to bury unflattering data.

Sometimes they bury data which shows drugs to be actively harmful. This happened in the case of Vioxx and heart attacks, and SSRIs and suicidal thoughts. Such stories feel, intuitively, like cover ups. But there are also more subtle issues at stake, in the burying of results showing minimal efficacy, and these have only been revealed through the excellent investigative work of medical academics.

And says there are two quick fixes:

The first is obvious. Nobody should get ethical approval to perform a clinical trial unless there is a clear undertaking that the results will be published, in full, in a publicly available forum, and that the researchers will have full academic freedom to do so. Any company trying to silence academics should be named and shamed, and even attempting to do so should be a regulatory offence.

That’s the butch solution. But there is also a more elegant one, which is arguably even more important: a compulsory international trials register. Give every trial an ID number, so we can all see that a trial exists, they can’t go quietly missing in action, and we know when and where to look if they do.

Open Culture

... is "your guide to smart media":

Open Culture explores cultural and educational media (podcasts, videos, online courses, etc.) that’s freely available on the web, and that makes learning dynamic, productive, and fun. We sift through all the media, highlight the good and jettison the bad, and centralize it in one place. Trust us, you’ll find engaging content here that will keep you learning and sharp. And you will find it much more efficiently than if you spend your time searching with Google, Yahoo or iTunes.

Here's a nice list of "Free Online Courses from Great Universities", organized by subject, you can download as podcasts.

Add Data to Google Spreadsheets Using Forms

Not everyone is comfortable entering data in a spreadsheet, but anyone who has been online has filled out a form. Now you can gather data in Google Spreadsheets using forms. From the Google announcement earlier this month:

Google Spreadsheets has a new feature that lets you create a form to accept data. When you go to the Share tab, there's a new option to "invite people to fill out a form". The form is very simple and can be customized by changing the order of entries, their labels and the type of answers. It's also a nice way to get feedback people who wouldn't normally collaborate on a spreadsheet.

You can create forms from spreadsheets or using this URL. To keep track of your forms, add this gadget to iGoogle.

Saving the American Time Use Survey

A group is soliciting signatures to save the American Time Use Survey from being cut from the federal budget. They say the survey is "the most important new data initiative begun by the U.S government in at least 35 years":

The ATUS provides essential information on how Americans spend their time, including time spent caring for children, cleaning the house, working for pay, and caring for sick adults. Put simply, the ATUS is needed to expand our horizons beyond merely charting where dollars go, to charting where time goes too. Statistics on spending, jobs, incomes, and so on are undeniably important. But anyone who wants to understand the changing lives of American families, to monitor the well-being of the American population, to measure national output, productivity and other outcomes that are essential to sound economic policy-making, or to make informed social policy decisions also needs information on how our population spends its time.

Bill Moyers Journal: "Mr. Heath Goes to Washington"

Bill Moyers Journal profiles David Heath, a reporter for the Seattle Times who once worked at The Courier-Journal, and his year-long investigation of Congressional earmarks. The half-hour TV show chronicles how Heath and the Times, which has nine full-time investigative reporters and editors, built a database of Congressional earmarks and cross-matched them against campaign contributions to expose some outrageous examples of wasteful spending. You can read the series and look up earmarks in the database online. At one point, an intern for the Times who helped type data into the database, tells the TV show she learned that investigative reporting isn't glamorous, as you would think from tales of the exploits of Woodward and Bernstein, but "really tedious."

Open Government Data

Every journalist should read the Open Government Data Principles at the Open Government Data wiki. The principles where developed at a meeting of 30 open government advocates in October:

Government data shall be considered open if it is made public in a way that complies with the principles below:

1. Complete

All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations.

2. Primary

Data is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms.

3. Timely

Data is made available as quickly as necessary to preserve the value of the data.

4. Accessible

Data is available to the widest range of users for the widest range of purposes.

5. Machine processable

Data is reasonably structured to allow automated processing.

6. Non-discriminatory

Data is available to anyone, with no requirement of registration.

7. Non-proprietary

Data is available in a format over which no entity has exclusive control.

8. License-free

Data is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed.

Compliance must be reviewable.

And if that interests you, you should sign up for the Open Government mailing list. Its members include Carl Malamud, whose most recent project is putting all federal court documents online, and Aaron Swartz, who founded theinfo.org, a Web site devoted to dealing with large data sets. theinfo, mentioned here previously, also has a series of mailing lists, for getting, processing and viewing corpulent data.

Instances of Use of United States Armed Forces Abroad, 1798-2007

This Congressional Research Service report (PDF), courtesy of the Federation of American Scientists, summarizes instances in which the U.S. used its armed forces since 1798:

This report lists hundreds of instances in which the United States has used its armed forces abroad in situations of military conflict or potential conflict or for other than normal peacetime purposes. It was compiled in part from various older lists and is intended primarily to provide a rough survey of past U.S. military ventures abroad, without reference to the magnitude of the given instance noted. The listing often contains references, especially from 1980 forward, to continuing military deployments especially U.S. military participation in multinational operations associated with NATO or the United Nations. Most of these post-1980 instances are summaries based on Presidential reports to Congress related to the War Powers Resolution. A comprehensive commentary regarding any of the instances listed is not undertaken here.

Derby DataTrack and Many Eyes: 2008 Kentucky Derby contenders, trainers and sires

This weekend we released the latest version of Derby DataTrack, our database of potential contenders for the Kentucky Derby. I know we (read: I) can do a better job presenting this data, but I haven't yet figured out how. A while ago ManyEyes added a network visualization tool and a way of embedding their visualizations on any Web site, so I thought I'd give it a try:

While this is intriguing, this isn't the solution, so if you have any thoughts on how we can do better that don't involve mastering Flash or Processing in a week, drop me a note.

Vote for what you find interesting in the JFK assassination documents

You're on deadline and someone releases a highly newsworthy document with hundreds, maybe thousands of pages. Can you really know what's most important in it in time for publication? No, of course not. Maybe there's an executive summary you can read. Or maybe there are certain passages containing a significant name or event you can pick out while scanning it, so you focus on those. Maybe a source told you what to look for.

Regardless, newspapers should build into their online publishing systems more ways to solicit readers' help.

The Dallas Morning News put recently released JFK assassination documents on its site and invited the public to help it find "something interesting."

But like most newspapers, including ours, they aren't really set up to handle something like this. You'd like to be able to directly comment on the interesting portions of the documents, highlight them and share them with others -- immediately. Instead you're asked to submit a comment to a generic form that strangely asks you to "Vote":

 

It would be better to have something along the lines of what this site did with last year's immigration bill, where anyone could comment on specific passages, and the comments are next to the relevant passages, not submitted into the ether.

If there isn't such a tool freely available out there, there should be. And if you know of one, please tell me about it.

The New York Times and "Playgrounds for Data"

Cyberjournlaist.net pointed me to this article by Jared M. Spool at User Interface Engineering, who says the New York Times is "a leader in creating interactive modules to accompany their news stories, often yielding in an impressive and fun experience":

Every organization sits on a ton of data. Making that data useful is a constant challenge for designers.

By looking at what the NYTimes interactive team has done, often with very small time frames, we can see examples of what is possible. From their work, we can learn new ways of presenting complex information in fun and engaging ways.

Spool also wrote about "Playful Data: 3 Inspiring Interactive Web Sites":

It feels to us that we're just at the beginning of what will likely be a revolution in how we handle large data sets. The applications we're seeing now are just the tip of the iceberg. The real value will be when we see these types of playful data tools in almost every application we touch. For those of us who like to play with our data, we're about to have some real fun.

Alltop

Alltop, which was inspired by Popurls, is a site that aggregates "all the top" stories and blog posts from Web sites on subjects such as politics, design, gaming, sports, autos, photography and "egos." You hover your mouse over a link for a summary, allowing you to consume a lot of content quickly. You could do this yourself with an RSS reader, of course, but Alltop saves you the effort and will likely introduce you to sites you don't know about.

How to make your audio slideshows better

... as explained by Mastering Multimedia.

U.S. National Library of Medicine Drug Information Portal

Their database helps you quickly find detailed and presumably trustworthy information on more than 12,000 drugs.

Google Static Maps API

You can now generate a static Google Map just by constructing a URL. There's a wizard to make it even easier. These are image maps — not the zoomable, clickable Google Maps you typically see — which Google says are ideal for multimedia-heavy Web sites because they'll load fast.

The URL to make the map above, which shows the location of the CJ, looks like this:

http://maps.google.com/staticmap?center=38.2463,-85.7608&zoom=9&size=200x200&maptype=mobile\&markers=38.2463,-85.7608,greenc|&key=ABQIAAAAinb4lNGZMYVUNPnvHp7HRRT2yXp_ZAY8_ufC3CFXhHIE1NvwkxSUeiWoopzbMAgQgXOhmRYBQWWYaA

Just stick it in an image tag and you're done.

'Course, if you do this, you will get a map that looks, as a certain programmer/journalist points out, "so 2005."

BigThink

If you believe small, incremental ideas are the ones that truly move the world forward, can you still make use of Big Think?

This is a digital age, one in which a wealth of accessible information empowers you, the citizen-consumer. But where is the information coming from? How accurate and unprocessed is it, really? Ask yourself this: how empowered do you feel debating a television screen or a newspaper?

Our task is to move the discussion away from talking heads and talking points, and give it back to you. That is Big Think's mission. In practice, this means that our information is truly interactive. When you log onto our site, you can access hundreds of hours of direct, unfiltered interviews with today's leading thinkers, movers and shakers. You can search them by question or by topic, and, best of all, respond in kind. Upload a video in which you take on Senator Ted Kennedy's views on immigration; post a slideshow of your trip to China that supports David Dollar's assertion that pollution in China is a major threat; or answer with plain old fashioned text. You can respond to the interviewee, respond to a responder or heck, throw your own question or idea into the ring.

To answer my own question: Yes, you can.

Here are big thinkers addressing the question, "Can newspapers hobble on?"

"Links As News, Links As Reporting"

Scott Karp at Publishing 2.0  makes an excellent case for why linking to other sites is a fundamental part of news reporting:

Many news organizations and media companies have been — and some still are — wary of linking to other sites. Why would we send our readers away, the thinking goes, don’t we want them to stay here, on our site?

But that’s the funny thing about web — it’s so counterintuitive. The way it works is almost entirely the opposite of how media used to work.

The first comment on this post sums it up:

The funny thing is links as news and links as reporting are not really new things on the web. The print news industry are just late adopters.

AngryJournalist.com

... asks, Why are you angry today?

AngryJournalist.com is for the underpaid, overworked, frustrated, pissed off and ignored media professionals to publicly and anonymously vent their anger. Share your story. With any luck, you'll feel better.

The creator explains his motivations:

I created this site for several reasons. In private conversations with friends I sensed that there is a growing angst among the upcoming crop of journalists entering the field right now. Journalism-school graduates have the odds stacked against them.

More than likely, their education was inadequate — it’s rare that new media skills were taught or were de-emphasized — making the majority of them less competitive. The job market is terrible. More companies are having hiring freezes — or worse, layoffs — meaning fewer opportunities are available. It’s an instance where supply greatly outnumbers demand. And of what jobs are available, these entry-level jobs pay poorly. It’s even worse in broadcast media.

Couple this together with an industry that’s getting hammered by Wall Street. Stock prices are tumbling, circulation’s falling or remaining stagnant at best, advertising dollars in print aren’t being replaced by online revenues and the pressure increases from above to keep tightening belts and “do more with less.”

First, the obvious: There's nothing less valuable as journalism than a flood of anonymous comments from people you don't know.

Second, I worked more than two dozen jobs before becoming a journalist, and people bitched and moaned about their workplace at all of them. The plight of journalists isn't unique.

And finally, for goddsake, young fella, this is the golden age of journalism: Never before in the history of the world have you had access to as many sources of information, as many outlets for your writing or as large an audience as you have now. The Man can't stop you from publishing anything you want at any time, and it costs you next to nothing.

Get over it.

FelonSpy.com: "You need to know who your neighbors are. Especially if they're dangerous criminals"

Gary Swick sent me the link to "FelonSpy," which allows you to "Search for violent criminals in your neighborhood":

Safety starts with good information, even if it ends with you holding a loaded .44 caliber handgun. While FelonSpy.com can't help you get a gun, we can certainly help you figure out which direction to point it in.

...

Our patented Felon Search technology mines data from across the nation, from the web and otherwise, and combines it into a single, easy to use interface. Whether you're checking up on your own neighbors or trying to find out if that hotel you've been eyeing is in a safe place, we can help.

...

Simply type in the desired address, click enter and let your new knowledge be your peace of mind.

All it gives you is a name, age and conviction -- not even a conviction date. So did the 72-year-old man who lives a few blocks from commit his felony assault in his teens and live a clean life since then? Dunno, from this site.

The methodology is also dubious:

FelonSpy.com uses a broad array of databases, each completely legal when used individually. They are probably also legal when used together, but that answer hasn't yet come to us. Our servers compare city, county, state and federal criminal databases with telephone records to pinpoint the location of the person you need to know is living next door to you.

In many cases the public telephone directory has incomplete or insufficient data, so about once a month we get updated address and number records directly from a number of telephone companies and credit reporting agencies (at tremendous cost to us, I should add). Don't ask how we get most of our information, but we do it, and we get our top shelf info from sources who wish to remain anonymous, and we pay a lot of money to insure that we keep getting it.

Their about us page, above an ad for ammunition that resolves to Insulting.com, says "We are former law enforcement officers, information technologists land developers and community leaders, all of whom have given up our posts in pursuit of this noble, sometimes misunderstood quest to label the underbelly of society by their actions."

And this is who they say they track:

We track virtually everyone with a criminal record including sex offenders, ex-cons (felony and misdemeanor), and those guilty of some of the more serious traffic infractions. You have the right to know who your neighbors are. We hope to track persons accused of crimes but acquitted in the future, but at this time we do not have sufficient funding to expand our database that far.

Ask too many questions and who knows, we might be tracking you next.

Clearly, these are people we should take seriously.

UPDATE 2/20: A reader here at the CJ reports getting different results each time he searches on the same address. So do I, which I hadn't noticed before. Try it yourself and see.

He asks, "Is it legitimate?"

I had hoped the sarcasm in my original post would make my point of view clear, but I can't say definitively because I haven't taken the time -- nor do I want to take the time -- to investigate the people it says are criminals to see if they even exist. I will say definitively that you shouldn't waste your time with this site if you're seriously interested in crime in your neighborhood.

If you want a laugh, it may be worth your time. For example, the "Sponsored Links" on its home page include "Prison Dating" and "Free But Crappy Legal Assistance."

Snopes.com, meanwhile, says it is a hoax.

Planetizen: The Planning & Development Network

Planetizen "is a one-stop source for urban planning news, commentary, interviews, event coverage, book reviews, announcements, jobs, consultant listings, training, and more."

Planetizen is read by a diverse array of people interested in the built and natural environments, and their interaction. Planetizen's audience includes professional urban planners, developers, architects, policy makers, educators, economists, civic enthusiasts and others from across the United States and around the world.

Planetizen prides itself on covering a wide number of planning, design, and development issues, from transportation to global warming, architecture to infrastructure, housing and community development to historic preservation. We provide a forum for people across the political and ideological spectrum, ensuring a healthy debate on these and other important issues.

Stanford research tutorial

Stanford's Key to Information Literacy is an interactive tutorial on how to do research.

 
 

SchoolDataDirect.org

The sponsors of SchoolDataDirect include the Bill & Melinda Gates Foundation and the Council of Chief State School Officers. The site lets you browse and download a wealth of education data -- including test scores, finances and demographics down to individual schools -- and promises to always have the most current data available. You can also compare schools to each other -- not just on blunt measures like overall test scores, but also how well they perform with special needs kids, English language learners and kids with disabilities. Disappointing to me, however, is that it appears to prohibit newspapers from using the data en masse:

If you are not associated with an academic institution or nonprofit organization you may only reproduce, distribute, display, or transmit de minimus amounts of Education Data on an infrequent basis and only for noncommercial purposes.

State Voter Registration Verification Web sites

The Department of Defense's Federal Voting Assistance Program maintains a list of State Voter Registration Web sites. Here's the direct link to Kentucky's and to Indiana's. With Kentucky's, if you have someone's first and last name and a date of birth, it will return their home address, their party affiliation, whether they're eligible to vote in the primary, their precinct and the state and federal legislative districts for their elected representatives.

World Clock

Dan Nelson pointed me to the The Spirit of Now's World Clock, which gives real-time estimates of the population, species extinction, military expenditures, oil pumped, cars produced, abortions and other measures by the year, month, week, day or as of now. He says you should also check out the site's life expectancy calculator, if you dare.

Robert F. Kennedy Jr. writes for the official Google Blog on mountaintop removal

Kentucky is ground zero for this issue. Kennedy's post highlights a Web site that uses Google Maps and Google Earth to bring the issue home:

If the American people could see what I have seen from the air and ground during my many trips to the coalfields of Kentucky and West Virginia: leveled mountains, devastated communities, wrecked economies and ruined lives, there would be a revolution in this country. Thanks to Google Earth, you can now visit coal country without ever having to leave your home.

Every presidential candidate – and every American – ought to take a few seconds to visit an ingenious new website created by nonprofit organizations in Appalachia that lets you tour the obliterated landscapes of Appalachia. By entering your zip code into this amazing new website, you learn how you're personally connected to mountaintop removal. Americans from Maine to California can see these mountains and the communities that were sacrificed to power their home. This puts a human face on the issue by highlighting the stories of families living in the shadows of these mines.

Census Atlas of the United States

The Census Atlas of the United States is "the first comprehensive atlas of population and housing produced by the Census Bureau since the 1920s." It's available free online as a PDF or you can buy it for $165.

The Census Atlas is a large-format publication about 300 pages long and containing almost 800 maps. Data from decennial censuses prior to 2000 support nearly 150 maps and figures, providing context and an historical perspective for many of the topics presented.

A variety of topics are covered in the Census Atlas, ranging from language and ancestry characteristics to housing patterns and the geographic distribution of the population. A majority of the maps in the Census Atlas present data at the county level, but data also are sometimes mapped by state, census tract (for largest cities and metropolitan areas), and for selected American Indian reservations. The book is modern, colorful, and includes a variety of map styles and data symbolization techniques.

Google News and the dearth of local news

Google upgraded its news search this week so you can type in a city name or ZIP code to find news for where you live:

While we’re not the first news site to aggregate local news, we’re doing it a bit differently -- we're able to create a local section for any city, state or country in the world and include thousands of sources. We’re not simply looking at the byline or the source, but instead we analyze every word in every story to understand what location the news is about and where the source is located.

Google is years behind Topix, which is partly-owned by Gannett, The Courier-Journal's parent company. You've been able to search for local news by ZIP code at Topix since 2004. On its blog, Topix greeted the Google news by doing some math:

If you take the totality of news created from the mainstream press (newspapers, radio, television stations), and do the back of the napkin math for all the local stories generated on a daily basis, here's what you get

1,440 local daily papers X 6 stories a day = 8,640
2,303 news radio stations X 3 stories a day = 6,909
1,686 television stations X 4 stories a day = 6,744
-----------------------------------------------------------------------------
Total local stories per day 22,293
Number of populated US ZIP codes 32,500

Topix concluded that "Local news is not a search problem."

We started by trying to add more sources. We added government, weather and industry sources, and then we added 25,000 blogs to the mix to see what we'd get. And, pretty much, they still didn't provide the breadth of coverage around local news, especially around small towns.

...

The conclusion our users pointed us to, by sending us hundreds of stories a day through our feedback form, was that there wasn't enough coverage of most areas by the mainstream press or the blogsphere, and that the real opportunity was to become a place for people to publish commentary and stories -- because no one else was going to do it.

So, we launched the Topix forums, and two years and 25 million posts later, Topix has gone from being merely an aggregator of local news, to becoming the home of local voice on the web

I've been mostly disappointed with sites that purport to offer tailored, local news because I haven't found enough compelling content to keep me coming back. Google's latest offering needs work too. I typed in both my city name and ZIP code and got nothing, which seems more than a little ham-handed. And I have no clue why last night I typed google news in the search box and got nothing:

But it works fine today.

Regardless, if it's really true that we're all just narcissists who only care about ourselves, then maybe what we're really looking for is the future of news search as predicted by Google Blogoscoped.

Dispensational Charts

Don't let the fact that two books I read recently were God is Not Great and The God Delusion dissuade you from viewing these "Dispensational Charts" by Clarence Larkin first published more than 75 years ago. They illustrate topics such as The Heavens, The Failure of Man and The Underworld.

The books and charts by Clarence Larkin have been extremely helpful to Christians since they were first published over 75 years ago. They have passed into the public domain and we are making some of the charts available here as an aid to Bible study. Larkin's charts are well thought out and Scriptually sound. Some of the more detailed are books in themselves. They reveal Larkin's vast knowledge of the Scriptures and phenomenal grasp of prophecy. Practically all of the prophecy teachers today got their basic prophecy knowledge directly or indirectly from Larkin and C. I. Scolfield. Larkin's works, as well as Scolfield's, are definitive, works that will endure until Christ's return. No other book since their publishing over 75 years ago has much improved on them.

information aesthetics notes that "while appearing sparse at first" they "seem to pack a lot of information into a concise format."

TableTools for Firefox

TableTools is a Firefox extension that lets you sort, filter or copy HTML tables on the Web. For example, if you come across a Web table that isn't sorted alphabetically and doesn't give you the option to sort, you can right click on the table, choose the sort option from the TableTools menu, and it sorts the table in place, automagically, without leaving the Web page or having to cut and paste into a spreadsheet or other software. You can also filter Web tables : TableTools adds dropdown menus to the top of a table, allowing you to filter by terms in any of the columns. You also choose to copy the filtered tables as tab-delimited text or HTML to other programs. TableTools can't handle everything, though. The download site notes it has trouble with more complex tables, such as tables that are nested.