Quality of websites: kitemarking the west wind

Rating the quality of medical websites may be impossible

“There's a lot of harmful medical information on the internet. Something needs to be done.” The obvious solution is to provide users with some sort of quality rating, guaranteed by a trusted third party. 1 2 Bodies as diverse as the European Union and NHS Direct Online are attracted to this solution to their problems of protecting the public health. But implementing such a solution is likely to be difficult—if not impossible.

Experience to date has not been encouraging. Three years ago Jadad and Gagliardi counted 47 instruments for rating websites, none of them apparently tested for reliability or validity. They wondered “whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm.”3 After the appearance of a further 15 unvalidated instruments they stopped counting (A Jadad, personal communication).

Why is it so difficult to get beyond the good ideas phase? Take the quality criteria most frequently mentioned: accuracy, comprehensiveness, and balance. No omniscient detached observer exists who can simultaneously view an article through the eyes of a specialist researcher, doctor, patient, and member of the public, let alone take into account the different perspectives of orthodox and complementary medicine. Falling back on the hierarchy of quality of evidence—with randomised controlled trials at the top and descriptive case reports at the bottom—is superficially attractive but ultimately constraining. Why should a site comprised solely of patients' experiences of a condition or a treatment rate lower than one listing systematic reviews?

One option is to rate the process by which the content was produced rather than the content itself—a medical journal's website containing peer reviewed material would rate higher than a commercial site selling miracle cures for cancer. This is the strategy largely adopted by BIOME, the UK gateway that has rated 4500 sites with health or medical content in the past 5 years.4 Even if it completes its Sisyphean task (at least another 20 000 health and medical websites to go), will consumers appreciate that a site's process, not its content, has been certified? (This raises a more basic question: how much do we know about consumers' use of kitemarks and seals of approval on the internet?)

As a more manageable first step, organisations have published codes of best practice that are meant to help website producers raise their game.5-7 These codes have proliferated as public anxieties about the credibility of medical websites have driven down their owners' share prices. But the market's punishment of drkoop.com for, among other transgressions, mixing up information and advertising—knocking 96% off its share value8—suggests that the market may have its own highly effective solutions to this problem. Could the iron hand of the market ultimately be more successful than codes of practice, especially since examples have already occurred where sites have falsely claimed to be complying with codes? This conjures up the spectre of the need to ensure compliance by active policing, with all its costs. And what of websites that decide not to apply for, or not to publicise, their rating?

Organisations awarding kitemarks or confirming code compliance could face legal challenges. Consumers who are harmed by their reliance on an overvaluation of a site have grounds for damages; sites that are victims of undervaluation (which may affect traffic to the site and hence advertising or financing) have grounds for defamation or product disparagement.9

Before going further down this path it is worthwhile asking whether we need any quality control at all. It's easy to be captivated by the novelty of the internet and convinced that it changes everything it touches. But for other more familiar sources of information—newspapers, magazines, books, and radio and television programmes—we cope unassisted by kitemarks. Much of their content contains material that is wrong, incomplete, and unbalanced from the point of view of anybody except its originators. But governments of all but the world's most authoritarian countries have yet to regard this as a problem. Our shorthand way of dealing with the information overload that already exists is to develop loyalty to brands. We gravitate to products that reliably give us what we want. The pattern of use on the world wide web suggests that this is also happening in cyberspace.

By design, the internet has no centre and therefore resists attempts at central control. Initiatives that go with its grain have a chance of success; those that go against it usually fail. Worse than failing, however, is having unintended harmful effects. Without a programme of incident reporting—of good and bad events—how can we be sure that the internet is harming more people than it is helping? The onus should be on those who want to intervene to show that their actions will result in a net improvement in human health. Until they have done so, the message to trigger happy legislators should be: “Don't just do something. Stand there.”