Twenty tips for interpreting scientific claims

It's not often I share journal articles without comment here anymore, but I just wanted to share this with my readers.

Twenty tips for interpreting scientific claims  from Nature.

A Matter of Control

Bump.

---

Original Publication 12/10/2011

It seems to me that many people misinterpret -- at least in their minds -- the meaning of this word when it is used in the scientific context.   While most of those no doubt understand the concept, after hearing the term enough, it just seems it comes to mean something else to them after a while.  I submit as evidence, statements made by two popular bloggers.  

First up, J. Stanton at gnolls.org with How “Heart-Healthy Whole Grains” Make Us Fat.   Wait!!  Did you click that link already?  I forgot to caution you that the post you are about to read contains science, so you might want to proceed with caution.  < / sarcasm > .  Anyway, the study he discusses in that post is:  High Glycemic Index Foods, Overeating, and Obesity.  Stanton prefaces his discussion with the following:

Yes, I admit to a degree of hyperbole—but this study is so well instrumented and controlled, and its results so informative, that I believe it’s important for everyone to read it.

Read more »

Exaggerations?

direct image link

The following comes from a slide presented by (?) at the Low Carb Down Under tour.  You can click to enlarge, but here's what the slide says under the title "What changed in the 1900s?"

• Sat fat down 83%
• Veggie oil/margarine up 535%
• Sugar up 1150%

Now, no doubt there are changes in eating habits, and perhaps "down under" people changed their eating habits even more dramatically than we Americans have, or are purported to have.  I've blogged many times that something's "off" with any stats indicating that Americans actually consume a low fat diet, but I'm not going to address the fat claims above.  It's the sugar claim that just screams -- that can't be so?!

Read more »

Variability & Error (yeah ... and one more time Ebbeling et.al.)

One of the issues raised in recent days is quite important:  How does measurement error factor in?  How does variability in what's being measured factor in?  

Statistical analyses do not address these two things.   I'm not sure which podcast it was, there have been a few, with the dynamic Chris duo (Kresser with Masterjohn), but Chris made the comment {paraphrase big time} that the best way to change your LDL particle size favorably was to have a different test done.  I'd love a link gang, if this rings a bell for you.   LDL is not even measured directly in some (most?  all?) analyses of blood lipids.  So first, Chris addressed the error in the measurement.    And I think they discussed the normal day-to-day variability in the lipoprotein concentrations in that same installment as well.  Now I don't want to put numbers/words in Chris' mouth but I've heard similar statements from countless others -- the number 20 point fluctuations in LDL comes to mind.  

Read more »

More on the Ebbeling et.al. JAMA Study

This started out as a comment on my last post, but got a bit lengthy so I decided to make it a full post.  I also addressed some issues with this study in another post those only interested in this study might have missed based on the non-descript title.  Let me start by saying that I did a fair amount of updating on my last post after reading the supplemental materials, and as a result I have some new criticisms of this article, and most of my original concerns remain.  So we're all on the same page, here's the study, and here's the Supplemental PDF. 

Many are treating the results of this study as if it were conducted in a metabolic ward.  This is not true, it was a free-living study except for 3-day hospitalizations for analyses.  Now it wasn't your usual free-living study in many ways:  Participants were paid and had all food prepared for them for the duration of the study (added all up, a little over 7 months!).  Monday through Friday they ate one meal a day at the facility.  Daily diaries were filled out documenting any "cheats" or foods left uneaten.  They received counseling if they had difficulty complying with the diet.  It is unclear, but it seems they were weighed daily, at least during the various 4-week weight stablizing/stable legs.  Body composition by DEXA was only assessed before and after the weight loss phase.  TEE (by doubly labeled water method) was assessed over the last two weeks of the 4-week pre-weight loss and test maintenance diet legs, activity was assessed by accelerometer for 7 days (it is unclear which week, I'm guessing the 4th week) for each test maintenance diet.  REE was measured by indirect calorimetry.  Intake was ramped up during the first 4 days of the stabilizing phase and subjects were weighed daily and caloric intake adjusted accordingly.  Here's an important little piece of information in this regard from the supplemental materials:  "We allowed the duration of the run-in phase to vary among participants, to account for individual differences in the rate of weight loss."  And:  "The energy intake required for weight stabilization at the end of the run-in phase was established as the energy intake for the entire test phase, with no further adjustments regardless of any weight fluctuation with the test diets."

Read more »

A Friday JAMA-lama Ding Dong!

Well ... I wasn't really going to weigh in further past using that recent JAMA article (EDIT:  full text no longer free at JAMA, see link below) as an example of where statistics can lead us astray.  But ... as is probably expected, this study has kicked up some dust in the community.  I'm sure I'll miss a few, but here's the weighing in so far:

Stephan Guyenet

Anthony Colpo

Marion Nestle

Don Matesz

Fat Head

That's just the blog posts, not the tweets, FB postings or other social media buzz ... and I'm happy to edit in more, just drop the hint in comments.  And here's the full text of the study:  Effects of Dietary Composition on Energy Expenditure During Weight-Loss Maintenance

Read more »

A Modest Proposal for Peer Review Research

With the advent of, and inexpensive nature of online sharing of information, I propose that all peer-review research should include (anonymous) raw data for each subject.  At the very least, there should be scatter plots presented for the individual data points for the main outcomes.

I frequently teach statistics, and one of the first things we discuss in that class is sort of the "first purpose" of it all.  Because before we can analyze data, first we must summarize and present the data in such a way that the "consumer" can readily glean information.  In one classic stats text -- Triola -- this part is given the acronym CVDOT.  C = Center, V = Variation, D = Distribution, O = Outliers and T = Time.  So we go through the various ways we can convey the center of a data set, it's variability, distribution, etc.  In most of the studies we discuss here, data is presented as a mean value +/- either the standard deviation or standard error (C +/- V in the acronym).  And further statistical analysis compares these means between groups for statistically significant differences.   If I have 20 subjects in a study, I can provide you with a table of all results sorted by subject number assigned randomly.  This tells you very little.  If all I do is sort the data ascending or descending, you can now readily pick out the range and "center" of the data.  Perhaps if data is of a more rounded nature, you might be able to pick out the most frequent or common values.  Outliers will jump off the page.  

Read more »