The MDS is intended to provide clinicians and policy makers with the most reliable summary of study results available when the results have been measured on different continuous or numerical assessment scales. Surprisingly, the method has not yet been the subject of a detailed examination of its own reliability. Previous searches were rare and focused on data extraction errors.2 4 5 In one study, authors found errors in 20 out of 34 Cochrane assessments, but because they did not give numerical data, it is not possible to assess how often they were important.4 In a previous study of 27 meta-analyses, of which 16 were co-screened assessments, 2 Could we not reproduce the smD result for at least one of the two studies we selected from each meta-analysis within our intersection of 0.1 in 10 of the meta-analyses. When we tried to replicate these 10 meta-analyses, including all the studies, we found that seven of them were false; One was then removed and a significant difference disappeared or appeared in two.2 This study complements the previous research by also highlighting the importance of different decisions in choosing results for meta-analysis. The results of our study are more general than for meta-analyses with MDS, as many of the reasons for disagreement are not related to the MDS method, but would also be important in analyzing data with the weighted mean difference method, which is the method of choice when outcome data were measured on the same scale. There is often a large amount of data in study reports, making it difficult to make the decision that should be used in a meta-analysis. In addition, data are often reported incompletely,2 3 which requires calculations or the inclusion of missing data, such as.B. the absence of standard deviations. Different observers may get different results, but previous studies of observer variation were not informative due to fewer observers, fewer trials, or less data.4 5 Here we report a detailed study of observer variation that examines the sources of differences of opinion when extracting data to calculate standardized mean differences.
. . .