Previous Next Index Thread

No Subject

 In article <>
   "Damien Broderick" writes:
 > Chris Lawson wrote:
 > > Meta-analysis, IMHO, doesn't count. [...] This is a bit like
 > > finding 3 inadequate samples of mince meat, and mixing all 3 in the mincer
 > > again.
 > Inadequate for what purposes?  You imply `tainted', but parapsychologists 
 > have tried (under the whip of their opponents, such as Hyman) to rid their 
 > data bases of contaminated data.  Are you really trying to tell us that 
 > adding 10 smallish samples together will not bring down the standard 
 > deviation, proportionately, to the point where an otherwise tenuous effect 
 > rises up over the noise level?  As you admit, pharmacologists use this 
 > procedure all the time.  It's not as compelling as levitating on to the White 
 > House lawn (and being shot out of the sky), but gimme a break here...  
 > Meta-analysis is acceptable in other fields.  Only an a priori conviction 
 > that psi is crap would make one *more* worried about its use in parapsych.
 I'm fairly sure I've seen negative views of meta-analysis outside the
 context of parapsychology. AFAIK, the problem is that meta-analysis can
 introduce biases.
 If one does enough experiments one will eventually get one with a result
 a few standard deviations ought. Combining this with the rarity of the
 publication of null results gives rise to a bias.
 For example, consider a system in which there is no correlation between
 a postulated cause and effect. Say 100 experiments are done, of which 80
 give a null result, 10 give a borderline positive correlation, and 10
 a borderline negative correlation. Say that of these 1/4 (20) of those
 giving a null result, 1/2 (5) of those giving a negative correlation, and
 all (1) of those giving a positive correlation, are published. In this
 circumstance meta-analysis clearly gives rise to a misleading conclusion.
 Another problem with meta-analysis is how to decide how to weigh the
 various data sets. Giving them equal weightings is wrong. If the
 experiments have no systematic errors then weighing them according the
 sizes and standard deviations of the data sets is appropriate. (Someone
 more statistically sophisticated then I am could provide you with the
 equations.) It is not obvious to me that it is always possible to
 produce objective weightings of the results of disparate parapsychological
 experiments; but, if the meta-analyst unconsciously gives greater weight
 to the positive results this skews the result of the meta-analysis.
 > Damien the skeptical sometime-parapsychologist
 Stewart Robert Hinsley             The adequate is the enemy of the good.