Why reading should not be believing

All comments (9)

• 
• • Ben Goldacre
  • The Guardian,
  • Saturday June 21, 2008

The best thing about this job is you have an excuse to read the Daily Mail every day: but sometimes, out of the corner of my eye, I worry that it might infect me. We are all biased by the information we expose ourselves to, through our friends, our reading, and our choices in life.

I think science coverage is pretty poor, and a lot of it is plainly wrong.

Gary Schwitzer used to be a journalist, but now he has turned to quantitative analyses of journalism, and this month he published an analysis of 500 health articles from mainstream media in the US. The results were dismal.

Only 35% of stories were rated satisfactory for whether the journalist had "discussed the study methodology and the quality of the evidence": because in the media, as you will have noticed, science is about absolute truth statements from arbitrary authority figures in white coats, rather than clear descriptions of studies and the reasons why people draw conclusions from them.

Only 28% adequately covered benefits, and only 33% adequately covered harms. Articles routinely failed to give any useful quantitative information in absolute terms, preferring unhelpful eye-catchers like "50% higher" instead.

Was this new? No. The same thing has been shown in Canada and Australia, and in the US almost a decade ago. Does it matter? Yes. Regardless of what they say in surveys about trusting doctors and priests, and despising hacks, in reality, people listen to journalists. This is not idle speculation.

A 2005 study in the Medical Journal of Australia looked at the impact of Kylie Minogue's breasts on mammogram bookings. They rose by 40% during the two-week publicity peak, and six weeks later they were still up by a third. The increase among previously unscreened women in the 40-69 year age group was 101%. These surges were unprecedented.

Am I cherry picking? A systematic review from the Cochrane Collaboration found five studies looking at the use of specific health interventions before and after media coverage of specific stories, and each found that favourable publicity was associated with greater use, and unfavourable with lower.

And it's not just the public: medical practice is influenced by the media too. Coverage of a flesh-eating bacteria outbreak led to a massive increase in group A streptococcus screening in one accident and emergency department (from 55 to 103 per 1,000 visits).

But even academics are influenced by media coverage: a seminal paper from the New England Journal of Medicine in 1991 said that if a study was covered by The New York Times it was significantly more likely to be cited by other academic papers. But for three months large parts of the NYT went on strike. The journalists wrote stories about academic research which never saw the light of day. The research saw no increase in citations. People read newspapers.

Despite everything we think we know, their contents seep in, we believe them to be true, and we act upon them.

· Please send your bad science to bad.science@guardian.co.uk