North American Fruit Explorers mailing list at ibiblio

[Road's End Farm <organic87@frontiernet.net>]

On Mar 28, 2009, at 10:05 AM, William C. Garthright wrote:

> And yes, there's a big problem with negative results not 
> getting the publicity of positive results. But none of that is any big 
> surprise, is it? What else would you expect? And if multiple independent 
> researchers confirm the results, what difference does it make who funded 
> the original study?

Suppose 100 independent studies are done.

Suppose 10 of them show a benefit. Suppose 10 of them show harm. Suppose 80 of them show a wash (no overall benefits or harm), or results too confusing to get anything from.

If all 100 are published and equally accessible, then any reasonable person can tell that not enough is known about the subject to come to any conclusions yet (and can probably also tell that either at least some of the studies were poorly set up; and/or else not enough is yet known about the subject to design the studies properly). 

However, suppose only the studies that show a benefit actually get published? Then you have what looks like a clear conclusion that there is a benefit. Look, it was replicated 10 times!

If you're trying to produce cold fusion, things may be clearer. However, in the area of human nutrition, there are huge numbers of variables, many of them almost impossible to control for. Different people digest things differently. Different people have different genetics, different activity levels, different exposure to other environmental factors, different other items in their diets. The item being tested is also highly variable: different strains of blueberries, blueberries grown under different conditions (climate, particular weather, farming techniques), blueberries harvested at different degrees of ripeness, different weather, even possibly at different times of day; blueberries treated differently after harvest according to huge numbers of other possible variables -- all of these factors can affect the balance of the nutritional compounds in the specific blueberries used in the study.

This means that even skilled and non-biased scientists, with unlimited money and the best will in the world, are likely to get different results from different studies. As humans can't be treated like lab rats, portions of this problem are very hard to control for. As our knowledge of how different nutritional aspects interact with each other is, to put it mildly, incomplete, other aspects of the problem are also very hard to control for.

All of this doesn't mean that we can't find anything out. But it does mean that it's going to take considerable time to actually get good evidence from which we really understand the results. This problem, which is inherent to the situation, is made worse by the facts that a) some (not all!) of the people doing studies are biased and/or incompetent  b) some (not all!) of the people choosing what to publish are biased and/or incompetent c) there is a built in bias against spending money and time to publish studies that are inconclusive, even though the percentage of studies that are inconclusive is crucial information in itself d) most news outlets reporting on studies either understand none of this or don't care; their bias is having something to publish that looks interesting. "Nutritional science doesn't understand this yet, more work needs to be done" doesn't make much of a headline.

--Rivka
Finger Lakes NY; zone 5 mostly