Andrew Sullivan recently got a letter from an epidemiology doctoral student offering some rules-of-thumb for thinking about medical claims and treatments. The advice is so sensible, and so nicely representative of good critical thinking, that I post them here:
1. Ancient wisdom is the weakest possible form of plausibility. Given what we know about our extensive irrationality, it isn’t hard to imagine a therapy existing for thousands of years because it plays on our biases but doesn’t do anything.
2. “Scientific” does not apply to the act of cherry-picking desirable outcomes from among many, many contradictory examples. In the absence of large, diverse, double-blinded randomized trials, we have to look at the totality of evidence from different sources using different methods.
3. Recognize that our ability to evaluate evidence is colored by our level of personal investment. Wishin’ and hopin’ has never changed a p-value nor shifted a confidence interval. We could all take a page from Adam Smith’s Theory of Moral Sentiments, and try appealing to the “impartial observer” within and without. Don’t wait until you have cancer to engage in the health arena. Teach yourself some science and statistics and form opinions now, before you have a diagnosis.
4. There is no such thing as “alternative medicine”, only “alternatives to medicine”. Once something has been proven efficacious, it simply becomes medicine. Physicians are desperate for treatments, and researchers are desperate for high-impact studies; if a treatment was ripe for the picking, you can bet your ass there’d be a stampede to demonstrate its effectiveness.
5. Always beware of “individualized” therapies. These may be crucially important in years to come, but only in very circumscribed areas (personal genomics). Charlatans have always appealed to our vanity; remember that often the more personally tailored a treatment becomes, the farther it deviates from accepted standards of treatment.
5a. And walk the other way when you hear the claim that a treatment is too individualized to be evaluated by randomized trials. This is often the post hoc cry of those who had a hand in designing the trial in the first place.
6. Don’t put your faith in experts, but do conditionally trust them (trust, but verify). The more information we have, the harder it is to evaluate and synthesize it. There are more and more great resources whose sole purpose is to evaluate the evidence on complex topics; while these may not answer your questions, they are an essential starting point.
Oh, and it’s annual flu shot time (which experts, afterall, recommend). My wife and I, as well as our two children, got our flu shots this past week. We also got our pneumonia shots last year, a vaccine that experts also recommend. Pneumonia shots are good for ten years.