Monday, October 25, 2010

Should the public be told about the trouble with medical research?

Among the reaction to my Atlantic article there has of course been a certain amount of skepticism and criticism.  Some of the criticism has been perfectly fair and insightful, pointing out the ways in which I or Ioannidis (the physician-researcher I profile in the article who has documented and analyzed the high wrongness rate in medical research) might ourselves be biased, and might be spinning some aspects of the story.  I touch on this problem in the article, and have a whole chapter on it in the book, and I have no interest in denying that my own bias and sloppiness (as well as those of the people I interview and those of my editors) may skew things.

I generally welcome criticism, and usually don't respond to it--I figured I had my say--except as part of a formal response to comments submitted to the publication that ran the article. But I saw a blog post today that I think calls for a bit of a response.  The post, by the physician-researcher David Gorski (whom I actually briefly quote in the Atlantic article), echoes posts in other blogs from a few people in or close to the medical community in essentially suggesting that the problems with research that Ioannidis has uncovered and that I report on in the article aren't really big problems; rather, they're just an acceptable part of the nature of research.  I don't think so.  Bias and sloppiness may be routine and perhaps inevitable even among top researchers, but that doesn't mean we shouldn't be made aware of the extent of the problems and the toll it takes on the credibility of medical findings--and the vast majority of researchers seem to agree, to judge by most of what has been posted in response to the article.  Gorski's post also states that we should focus on the fact that (as I report in the article) 90 percent of large randomized controlled trials tend to hold up, and not pay much attention to the fact that other types of studies sink to levels as low 10 percent rightness and even lower.  This is to me a shocking argument, considering that large RCTs make up a tiny percentage of the studies that fill journals, make headlines and influence treatment and lifestyle decisions.

Finally, the post argues that everyone should put their energy not into looking into the problems with mainstream medical research, but rather into the poor or non-existent science behind alternative medicine--and that we should keep our mouths shut about whatever problems we do find with mainstream medicine because it only gives ammunition to the alternative medicine crowd.  Once again, I have to strongly disagree.  People who are drawn to alternative medicine have already proven themselves essentially either uninterested in or incapable of assessing scientific evidence, and when researchers like Gorski rant and rave about alternative medicine they're really just preaching to the choir, as is clear from the comments that appear under his post.  In fact, compare the nearly unanimous point of view in these comments to those that appear almost anywhere else in response to the Atlantic article--Gorski's fans clearly have their minds made up about people who criticize mainstream medical research, and they don't even actually have to read the criticism themselves to feel comfortable commenting dismissively and authoritatively on it.  Some of the comments, following Gorski's lead, include ad hominem attacks on me that are based entirely on sloppy and mostly incorrect assumptions about my background.

Unlike the well-known absence (with a few exceptions) of good science behind alternative medicine, the serious problems with mainstream medical research have largely been unknown outside of the medical community itself.  I believe the public has a right to hear about them, and to judge by the reaction the public seems quite interested in hearing about them.  I quote Ioannidis in the article as pointing out that if mainstream medical science tries to keep quiet about its problems and limitations, then it is doing what it accuses alternative medicine of doing--misleading the public.  I'm glad Gorski and his fans represent a small minority of the medical community in being unwilling to own up to and communicate these problems--see, for example, what the British Medical Journal had to say about my book--and in focusing instead on endlessly recycling the same old complaints about alternative medicine because it makes them look good in comparison.

And for the record: Along the way, Gorski belches out the sarcasm-and-bile-drenched claim that my earlier post on this blog relating to quack autism researcher Andrew Wakefield reflects my refusal to recognize that Wakefield was wrong.  That's just plain silly.  The point of the post was that the outing of a rare, gross fraud like Wakefield distracts from the more widespread, routine problems in medical research trustworthiness.  I think I was pretty clear about that, but judge for yourself.  I happen to be a little sensitive to the suggestion that I support quack autism research, because two members of my immediate family work with children with autism, relying solely on the treatment that is the mainstream standard of care, applied behavior analysis.  I know quite a bit about autism quackery, actually.  I just didn't think the intelligent readers of this blog needed me to rant and rave, Gorski-style, about an obvious and blatant charlatan who had long been making global headlines for his misdeeds. (And my Wakefield post linked to an article that described the misdeeds at length, though I just now changed the link to point to a similar article because the original is no longer online.) Considering that he passes himself off as the champion of objectivity, facts and reason, Gorski seems surprisingly comfortable distorting the facts to fit his nearly undisguised biases.  But maybe we should be grateful for undisguised biases--it's really the well-disguised ones we need to watch out for.

Saturday, October 16, 2010

Medical Wrongness in The Atlantic (and a few notes)

I have a big feature article in the new (November) issue of The Atlantic, which just came out. It's the magazine's annual look at "Brave Thinkers," and it leads off with my longish profile of Dr. John Ioannidis, who has spent most of his career studying and exposing the many problems with published medical research. My book Wrong opens with Ioannidis, and comes back to him a few times. But in the article I go much deeper into his work, his background, his personality and his thinking--I spent several days with him in Greece for the article--and say more about what his discoveries mean for medicine and doctors' ability to treat us effectively. It's not a pretty picture. Think of the article as an extremely long, important and (I'd like to think) colorful post for this blog.

By the way, sorry for the long gap since the last post, I got caught in an extended perfect storm of article deadlines, and travel for research and speaking engagements. (And it's not over yet.)

Meanwhile, on another semi-personal note that affects this blog, I've been transitioning to some new projects. I'll say more about these projects in the coming months, but for now let me just say I have a new special interest in obesity and weight loss, which I (and many others) think have become the biggest single health threat to the length and quality of our lives that is also potentially fixable. The blog's basic theme isn't really changing, but a lot of the posts will be about issues relating to obesity and weight loss. Note that I've parenthetically added the new special interest in obesity to the blog's title. I hope it goes without saying I'd love to get feedback from MSOMed readers on this partial change in focus.

Tuesday, August 24, 2010

Diet Secret Revealed by Scientific Study! (Gulp)

A new study getting some very prominent and excited press, including a big feature spot on the Today show this morning, has found that drinking two glasses of water before a meal is the secret to losing weight. The Today show and the articles have noted that the study was a randomized controlled trial, which as we all know is the gold standard of medical research. So get ready to guzzle the H2O, and finally lose those pounds!

Or maybe not. Diet studies are terrible, and for a number of reasons. They're of course subject to all the same problems that all medical studies suffer from: poor design, bad measurement, researcher and subject bias, patient selection problems, poor analysis, and more. Randomized controlled trials--in which, in the simplest case, subjects are randomly divided into two groups, one of which gets a special treatment and the other doesn't--can help with some of these problems, because in theory whatever mistakes are made or biases take hold in the study with one group, the same should apply to the other group, so any differences in the results with the two groups should be the real deal. Well, it doesn't always work out that way, even when an "RCT" is done right. But in this case, as with most diet studies that claim to be RCTs, it wasn't done even close to right. The really big advantage to RCTs, as I wrote in my last post, comes from their being "blind," meaning that both researchers and subjects ("double blind" is the technical term) don't know which subjects are in which group. That's what really has at least a shot at eliminating biases. But you can't blind diet studies, because, for example, you can't fool people into thinking they've just drunk two glasses of water when they haven't, or vice-versa. And that means that the researchers and subjects are free to be very biased toward the treatment under study. They all want the research to pan out, and they tend to find ways to make sure it does. It's also questionable whether non-blind randomized studies are truly randomized. The problem here is that once people in the study are subjected to the different treatments, you can start getting very different drop-out rates in the two groups, sometimes right up front. If you agreed to be in a study of some great new diet technique, and then were immediately told you're going to be put on a plain old diet without getting to even try the technique, wouldn't you be more tempted to just blow it off? After all, you've been on diets, they didn't work--why bother trying again with the same old thing? The researchers are probably hoping you'll fail on it so that their technique will prove effective, and they may communicate discouragement. It's a mess of a study set-up.

It gets worse. Diet studies rarely go on long enough to determine whether people actually keep the weight off, which as we all know is the real question. People tend to lose weight on almost any kind of diet, typically over a period of a few months--but then they gain it all back, and often more, over the next year or so. This water-drinking study as originally published some months ago went on for 12 weeks, a classic set-up for misleadingly concluding that the diet was effective. But here's where things get interesting. Though the original study only lasted 12 weeks, the newly released findings have the researchers claiming in public statements that the water-drinkers in the study on average kept the weight off for a year, and even lost a bit more. This is an extraordinary claim that sets off all sorts of alarm bells for me. Careful diet studies almost never report that sort of longer-term weight-loss success. And while the water-drinkers reportedly lost and kept off an average of 17 pounds after a year, the non-water-drinkers in the study lost and kept off an average of nine pounds after a year. Frankly, I'm more impressed by the non-water-drinking results--it's even more unusual for the non-gimmicked dieters in a study to keep weight off that long. In looking at the original 12-week study, I didn't see any specifics about what, if anything, the researchers did (or reasonably could have done, for that matter) to ensure that the subjects in the groups dieted in a way resembling the way people typically diet, that they didn't receive some special encouragement or coaching, that the two groups were treated identically by researchers, that the groups accurately reported what they were eating and drinking, that the water-drinking group really drank the right amount of water before eating and that the non-water-drinking group didn't drink a lot before or during meals. It also wasn't clear what special pains, if any, were taken to make sure that bias wasn't introduced by drop-outs, or by people who didn't comply with the instructions or accurately report data. (It has been shown again and again that inaccurate self-reporting, which is a problem in many health studies, is particularly egregious in diet studies.) Whatever problems crept into the 12-week study, I can't imagine it was anywhere but downhill in extending it to a year. The researchers certainly sound in the original report like they were trying to be careful, but there's really only so much researchers can do, and the result in most diet studies is that there are potential holes big enough to drive a car through. And that certainly seems like the case with this study. I can't help noting that the authors are releasing these newer findings before they have been peer-reviewed. Now, I've written at some length elsewhere that the peer review process isn't nearly the guarantee of quality that it is often taken to be by the public and by journalists. But non-peer-reviewed data is generally considered one step up from fairy tales in the research community, and I wonder if the claims of this study might change after peer review.

But you don't have to accept any of my reasoning about why this, or any other diet study, is likely to be fairly useless in telling you what's going to help you or anyone else lose weight. Here's a better reason to reject these studies: Scientific studies have found that anything is the secret to losing weight, including low carbs, high carbs, low fat, high fat, big breakfasts, small breakfasts, lots of veggies, lots of fruits, lots of nuts, snacking, not snacking, and on and on--and scientific studies have also found that every one of these gimmicks doesn't work, including drinking a lot of water. There's only one reasonable explanation for these conflicting findings: the scientific studies just aren't very reliable. They fail to shine any light on the problem. We don't have to agree on why that's the case, but it's hard to argue it isn't the case.

It's easy to explain why the water-drinking trick doesn't work in the real world. It's a type of diet I call a "sickening" diet. These diet gimmicks work by making you physically uncomfortable in some way, with the result that your appetite is dulled, as usually happens when you don't feel great. Any diet that forces you to eat or drink a large amount of some type of not-especially-appealing food or beverage will have this effect. Some people don't feel well even when eating low carb or low fat, which helps to explain why these diets may help many lose weight initially. The way the water throws your appetite off is obvious: it fills your stomach, and then when you add food on top of it you get an uncomfortably full sensation. None of these sickening diets are especially effective in the real world for what should be an obvious reason: people get sick of sickening themselves. They don't like it. They don't want to cram in another grapefruit, or celery stalk, or glass of olive oil, or chunk of meat. Go ahead, try the two glasses of water trick. Let me know how long you go before you find yourself staring at those two glasses of pre-dinner water and feeling, perhaps even with a touch of nausea, that the last thing you want to do is force them both down your gullet. Even worse, as the water or other sickening agent becomes more and more repulsive over time, your desire to go back to normal levels of eating (or normal varieties of food, if you've been restricting yourself in some way), will grow, especially as you lose weight, and you'll become hungrier and will experience intense cravings. No wonder people gain it all back and more.

Folks, people have been trying to lose weight by drinking a lot more water for at least four decades. I remember my mother forcing down eight glasses a day in the 1970s under the Stillman diet. She didn't keep the weight off, and neither did most people who have struggled with dieting and tried the water trick. If this silly gimmick worked, you'd have known it a long time ago.

Now having said that, let me say that drinking while dieting is a pretty good idea, as long as we're talking about modest, non-repulsive amounts of water or any low-calorie beverage. It's easy to mistake thirst for hunger, and it's certainly true that having at least a little something in your stomach can temporarily help take the edge your appetite. So sure, step up your water- or tea- or diet-coke-drinking, within reason. Just about every diet expert has been saying that forever, and it very well might help your diet efforts a bit. I'm not surprised the water drinkers in this new study lost and kept off more weight than the non-drinkers. I'm just highly skeptical of the claim that it makes a big difference, and especially of the claim that it can make the difference between your dieting successfully or not over the long term. The researchers (and especially the mass-media reports) seem to believe that it will, but I don't think that belief is anywhere close to being supported by the study. If you're tempted to believe it, then I sincerely wish you the best of luck in getting yourself to stuff down two full glasses of water before sitting down to every meal you take in for the rest of your life, and in restricting how much you eat at all of these meals because of it.

Sunday, August 15, 2010

Should You Bypass Obesity-Related Stomach Surgery?

The latest option for losing weight via surgically reducing stomach size: Having most of your stomach cut away and taken out by a procedure performed mostly through your mouth and down your throat. (A bit of the work is done through small slits in your abdomen.) What little is left of the stomach is then stitched up to form a mini-stomach, with the obvious result that it only takes a relatively small amount of food to fill it up. Other, more-established options for stomach surgery include the gastric bypass, a more extensive procedure in which in addition to the stomach being downsized it is also reconnected to a shorter span of intestine, cutting down on calorie absorption, and the lap band, in which much of your stomach is squeezed off by an adjustable band.

These procedures are often discussed as if they have been proven safe and effective in studies. But keep some points in mind when considering how "proven" a surgical technique is. There are (with a very few exceptions) no good, ethical ways to conduct randomized, blind trials of surgical procedures. It would be hard enough to do the randomization, in which patients are (to oversimplify a bit) randomly assigned to either a group that will receive the surgical procedure or a group that will not, because not many patients are willing to have surgical decisions made for them by the toss of a coin. But even if that issue were overcome, the real problem would be with blinding, which means patients don't know which group they're in. It's not easy to get people to not know whether they've had surgery or not--and even if you could pull it off it's even tougher to do it in a way that doesn't run afoul of ethical considerations. (It's been attempted, believe it or not, but hell was raised.) The result is that studies of newer surgical procedures are mostly observational studies, which means doctors just note how patients seem to be doing. (I'm not counting animal studies. Neither should you. No matter what researchers say, on average these studies don't translate well to humans.) Some procedures that have been widely performed for many years get "case control" (epidemiological) studies, which look back at groups of patients who have received the treatment and compare them to groups who have not. While you can't fully tell how trustworthy a study is just by noting what type of study it is--it's wise to consider all medical studies at least potentially somewhat untrustworthy--as a rough generalization randomized, blind trials tend to be more trustworthy than case-control studies, and observational studies are the bottom of the barrel, though to be sure there are lots of exceptions and qualifications to this pecking order.

In short, the fact that gastric bypass, and, to a lesser extent, lap band procedures, have been found in case-control studies and many observational studies to be fairly safe and effective should not be taken very seriously. These types of studies can be subject to intense biases on both the parts of doctors and patients--they generally want the procedures to succeed, and may distort their observations and do a poor job of recording all relevant data. Researchers also typically don't have access to all the data they need to fully analyze the situation. (For example, in the New England Journal of Medicine article I reference just above, the authors to their credit note that the study ignores the possibilities--likelihoods, I'd reckon, though the authors beg to differ--that people who had the surgery got much more or better medical attention afterward than other obese people get, and that they may have been healthier to begin with.) The procedures are a huge money-maker for surgeons, and the patients are often desperate to lose the weight and may be eager to convince themselves and others that they've done the right thing and that things are going to work out well. These biases, which can be and often are devastating to the reliability of medical studies, are exactly what randomized, blind studies attempt (if not always successfully) to eliminate. That's why many surgical procedures are widely accepted as safe and effective, only to eventually prove themselves not so safe and/or less effective than other, less invasive and less risky treatments--the list includes many types of heart-related operations, back-pain-related operations, brain surgeries, and on and on.

This newest stomach-reduction procedure has been performed on one person so far. And yet surgeons just about everywhere will soon be able to perform it on anyone they deem suitable, providing nothing obviously terrible happens immediately to this one patient as a result. True, these sorts of "laparoscopic" (performed through small incisions and/or through the mouth, using a tiny camera and light to allow the surgeon to see what's going on in there) procedures tend on average to be less risky than conventional, large-incision versions of the same surgery, but there can be reasons why they are sometimes less safe and/or less effective in some ways--for example, the restricted access and vision may result in a sloppier job of cutting or stitching, or in missing problems with bleeding.

And there are reasons to question whether the people who are getting these procedures really need them enough to justify the risks. (Though the twice-aforementioned NEJM study found that overall death rates went down for obese people who had gastric bypass surgery due to fewer deaths from heart disease, diabetes and cancer--findings that for a number of reasons I suspect may exaggerate the benefits of the procedure--around one percent died from the surgery itself shortly after the operation, and many had complications.) The people who sign up for these operations are typically obese people who have tried to lose the weight through diet and exercise but have failed, and whose health appears to be at some risk. But trying to compare the potential payoff from trying to diet and exercise to the likely benefits of surgery is a bit of a fixed contest. I've been studying the question of why people fail with diets and exercise, and it's clear that the success rates would likely be much, much higher if overweight people weren't getting such almost uniformly terrible advice about how to diet and exercise, and if their motivation to lose weight through dieting and exercise weren't on average so low. Why is their motivation often so low? For one thing, they keep hearing from many researchers and obesity experts that they'll probably fail with diet and exercise, and for another they've been led to believe that surgery provides a much easier and surer option that will provide all the same benefits of a successful diet and exercise program without any of the work--these procedures essentially are intensely marketed much the way cosmetic surgery is hawked, because they're similarly ridiculously profitable. (Another issue is whether the success rate for dieting and exercising is really anywhere near as low as we've been told--but that's a more complicated story.) Meanwhile, the success and benefits of surgical weight-loss keep getting pumped up through studies that are likely somewhat biased and otherwise flawed.

I don't doubt there are obese people who, when the whole picture is soberly assessed, might be making a reasonable decision to undergo one of the procedures. But I suspect it's a fraction of the number of people who are actually getting the surgeries. I was especially appalled to see a big and very well-researched and well-written cover story in the Atlantic earlier this year on the subject of obesity essentially advocate for, or at least seriously suggest the idea of, the government putting its anti-obesity money into handing out gastric bypass surgery for the obese, presumably instead of into research, education, good-nutrition programs, more-exercise initiatives and other anti-obesity public-health initiatives. The article, which for sure in many ways was terrific, more or less declared dieting and exercise to be a waste of time--a point of view which is pushed by many published medical studies relying on highly biased and flawed set-ups (as evidenced, for example, by the fact that findings from these studies widely and sometimes sharply conflict), but which flies in the face of the significant documented success rates that hundreds of real-world programs have achieved with obese people (and which has also been backed up by other published studies, for what it's worth).

I think this sort of give-up-on-dieting-and-exercise-and-hold-out-for-surgery(-or-a-pill) thinking is dangerous, and is in fact a contributing factor to the obesity epidemic. If you're thinking of getting some form of stomach-reduction surgery, I'd urge you to do some more research into the possible complications and into dieting and exercising options, and to discuss what you find with your doctor. And if your doctor is pushing you toward the surgery and especially if he seems biased toward it, try a second opinion from a doctor you have good reason to believe has an objective view about both diet-and-exercise programs and surgical alternatives. Many, many obese people have shed the weight and kept it off through dieting and exercise. Shouldn't you be absolutely sure you've given dieting and exercise your very best shot under the right program before you have one or even two of your major organs hacked up and extensively rejiggered?