Can you trust someone's conclusions if you can't reproduce their work?

In theory the strength of science is that work done in its name is reproducible and verifiable, but what does it say about the theory when in fact that's not really true?

Journals and granting agencies are prodding scientists to make their data public. Once the data is public, other scientists can verify the conclusions. Or at least that’s how it’s supposed to work. In practice, it can be extremely difficult or impossible to reproduce someone else’s results. I’m not talking here about reproducing experiments, but simply reproducing the statistical analysis of experiments.

It’s understandable that many experiments are not practical to reproduce: the replicator needs the same resources as the original experimenter, and so expensive experiments are seldom reproduced. But in principle the analysis of an experiment’s data should be repeatable by anyone with a computer. And yet this is very often not possible.

[via Statistical Modeling, Causal Inference, and Social Science]

Can you believe reporters when they write, "studies have shown"?

Stats.org deconstructs the phrase "studies have shown" in reporting on infants fed breast milk versus formula:

... in the increasingly overly-simplified, context-free world of reporting on health, the phrase “studies have shown” is often a formula for telling the reader what the reporter assumes has actually been shown.

A Free Online Course in Science Journalism

... is offered by the World Federation of Science Journalists.

The authors and translators of this course are experienced journalists and trainers from all continents. They cover major practical and conceptual issues in science journalism, for example: how to find and research stories, exposing false claims, how to pitch to an editor, turning crisis reporting to advantage and so forth – topics that are relevant to beginners in journalism as well as more experienced reporters and editors in all regions of the world.

Scitopia.org

scitopia.org is a search engine devoted to science and technology. It was created by "15 leading science and technology societies":

Searching for a better way to help researchers quickly get to the quality content they need, these society publishers developed a gateway to the research most cited in scholarly work and patents. Scitopia.org searches the entire electronic libraries of the leading voices in major science and technology disciplines and provides relevant results, without the noise of other Internet search engines. More than three million documents, including peer-reviewed journal content and technical conference papers, spanning 150 years of science and technology can be searched through the site.

But to get the full text of the articles you find you'll have to be a member of one of the societies or pay per view.

Dilbert and the psychodynamics of science in the media

The Language Log used yesterday's Dilbert cartoon as a prompt for commentary on "The psychodynamics of science in the media":

It's true that things would be better if individual scientists were less willing to over-interpret or mis-interpret in order to make a splash; and if PR people were less eager to encourage and help them; and if individual journalists had the time and the ability to do some critical reading in the primary literature, instead of just decorating press releases with a few quick quotes from experts; and if media executives were not too focused on bean-counting to care one way or another about any of this. But focusing on individual failings ignores the fact that all of the people involved -- scientists and journalists and executives and rent-a-weasels -- are responding to the normal economic and psychological forces within their diverse subcultures, which interact badly in their areas of overlap.

On the whole, the whole system of science and engineering does a pretty good job of creating knowledge and technology. On the whole, the media do a pretty good job making information available to the public. Put them together, and the whole is noticeably less responsible than the parts.

This was the latest in a series of posts by Mark Liberman on ways the media can improve its reporting on science:

• IQ and birth order
• Two Simple Numbers
• Thou Shalt Not Report Odds Ratios
• Hand-waving in the Washington Post
• The "happiness gap" and the rhetoric of statistics

(And thanks to the Social Science Statistics Blog, which first brought my attention to this.)

Overcoming Bias blog

Overcoming Bias is a blog from the Future of Humanity Institute at Oxford University:

How can we better believe what is true? While it is of course useful to seek and study relevant information, our minds are full of natural tendencies to bias our beliefs via overconfidence, wishful thinking, and so on. Worse, our minds seem to have a natural tendency to convince us we that are aware of and have adequately corrected for such biases, when we have done no such thing.

In this forum we discuss whether and how we might avoid this fate, by spending a bit less effort on each specific topic, and a bit more effort on the general topic of how to be less biased. Here we discuss common patterns of bias and self-deception, statistical and other formal analysis tools, computational and data-gathering aids, and social institutions which may discourage bias and encourage its correction. Other topics may be discussed to the extent they exemplify important biases and correction issues.

Institutional Review Board Watch

... describes its reason for being this way:

Institutional review boards have been set up at nearly all research institutions in the US, to protect the welfare of human research participants.

Over the past decade, IRB's have grown greatly in power and range of authority. The home institutions have, however, largely abrogated their responsiblity to oversee and control the procedures followed by IRB's. As a consequence, the IRB's have increasingly harrassed researchers and slowed down important research, without protecting any human research participants.

The purpose of this site is to chronicle the abuses by IRB's.

I'd say one of the first rules of being a watchdog is to explain who you are, which this site doesn't do as far as I can see. Or at the very least, you should explain why you can't explain who you are.

I learned of this site from a posting on Statistical Modeling, Causal Inference, and Social Science, which agrees the process can "get a bit Kafka-esque."

Free science search tools

ResourceShelf gives a quick look at what's out there.

The three markers of scientific fraud

Dr. David Goodstein, the Vice Provost and Professor of Physics and Applied Physics at Caltech, explains that three motives or "risk factors" are always present in cases of scientific fraud. He says the perpetrators

• were under career pressure;
• knew, or thought they knew what the answer would turn out to be if they went to all the trouble of doing the work properly, and
• were working in a field where individual experiments are not expected to be precisely reproducible.

He also says scientific fraud is "almost always found in the biomedical sciences, never in fields like physics or astronomy or geology."

Google patent search

Google has unveiled an easy-to-use patent search. A search for "Louisville Kentucky" turned up three patents issued this year, including a "Method and apparatus for electronic collection, translation, grouping, and delivery of wage assignment information," "a Method and apparatus for bone fracture fixation," and a "Crankcase sealing apparatus."

SciTechResources.gov

SciTechResources.gov is "A catalog of government science and technology web sites" for the "the scientist, engineer, and science aware citizen."

Injecting science into the social

When I was a political science and economics major in the early 1980s, both fields typically constructed elaborate theories on sand -- whether it was Marxists spinning elaborate superstructures whose inner contradictions soon became only too apparent, or economists building economic models that always assumed away everything interesting and difficult about the world. It's heartening, therefore, to read that both fields are becoming more empirical, a trend driven by cheap computers and the vast quantities of data now being generated:

The quantitative social sciences are in the midst of a revolution in understanding the world and solving real problems. A dramatic increase in progress is now achievable because: 1) changes in technology enable us to collect and store unprecedented amounts of far more informative data about human populations and institutions; 2) new policies encourage the collection of data and its provision to researchers, including the computerization and automation of many government services, new data collection requirements, e.g., the No Child Left Behind Act, and the growing movement in science to make data publicly available; and 3) the development of novel methods of data collection and analysis that make it possible for scholars to extract information from new data, such as from the rise of social experiments that enable reliable causal inferences in major issues of public policy, natural language processing that enables scientists to extract information from millions of Web sites, newspapers, emails or other textual sources, and new informatics techniques that provide instant and reliably persistent, access to the world's data.

So says Gary King, the directory of the Institute for Quantitative Social Science at Harvard, in an interview for the Data Mining Review. Some of that data and a tiny fraction of the techniques he discusses may one day be adopted by journalists, who are the protozoa of the fact-based universe. More than a few gut-trusting editors, incidentally, would just guffaw at this quote from King: "The central conclusion of research in hundreds of fields and most of the hundreds of thousands of applications is the same: Whenever a sufficiently important fraction of information can be quantified, statistical analysis beats qualitative human judgment. There is just no contest."

The Science Inventory

The Science Inventory is a searchable database on what the EPA is up to scientifically. It provides "a snapshot of EPA science being conducted in its research laboratories and centers, program and regional offices, and through grants and other assistance agreements to universities and other institutions."

Physorg.com

Physorg.com writes about science, physics, technology, outer space, the Earth and more.

Good stuff from LLRX.com

• Criminal Justice Resources: Prisoners' Rights and Resources on the Web
• It’s Not Rocket Science: Making Sense of Scientific Evidence - "The purpose of this article is to review a search process using advanced search query features in Google, Yahoo, and other search tools to find publicly accessible Web-based information on toxic substances and the law and, more specifically, the reliability of scientific evidence about toxic substances."
• The Government Domain: Government Documents and the News
• FOIA Facts: Improving FOIA Operations

SciTechDaily

SciTechDaily links to articles from across the Web on science and technology.

Nature: Top five science blogs

Nature asks the people behind Technorati's five highest-ranked science blogs to explain the reasons for their success.

Knight Science Journalism Tracker

The Knight Science Journalism Tracker ("Peer Review Within Science Journalism") offers a daily roundup of important science news:

KSJ Tracker is a new service for science journalists, created and funded by the Knight Science Journalism Fellowship Program at the Massachusetts Institute of Technology. We believe that if science reporters and editors have convenient and timely access to the work of peers across the country, they can better evaluate and improve their own performance.

Our goal is to provide a broad sampling of the past day'’s science news and, where possible, of news releases or other news tips related to publication of science news in the general circulation news media, mainly of the U.S. Our goal is to have a new batch of posts up each day by 1 pm Eastern time.

We also give science journalists the opportunity to suggest stories and to comment constructively on one-another'’s work. The volume of science-related news makes a genuinely comprehensive site almost impossible, but we get what we can.

How to restore media credibility in science and health coverage

The suggestions come from a STATS contributor. They're an excerpt from a discussion hosted by the Financial Times on whether the old media should embrace the new.

ScienceBlogs

ScienceBlogs calls itself "a global, digital science salon." "ScienceBlogs is the web's largest conversation about science," the site says. "It features blogs from a wide array of scientific disciplines, with new voices coming on board regularly."