Language bias

Not considering research findings in all languages for systematic review. Index of biases | Editing advice |  

Background
One can only be sure of the robustness of one's conclusion about the published data if one is sure that all the relevant data are included. A particular subcategory of publication bias is language bias. This occurs when literature searches are undertaken in one language, and misses out studies published in another language or languages. Whilst the vast majority of papers - 80% in one series of 600 trials - in the medical literature are published in English, a minority are not. Potentially, important findings (either positive or negative) might only be published in a non-English language, and if not included in a systematic review which only involved English language papers, this might skew the result.

Another, indirect, language bias may operate if non-English researchers choose to publish positive studies in English language journals, which will have a higher impact factor. Their negative studies will be less likely to be accepted in the more competitive English language journals, and thus submission to local language journals may prove more successful.

Example
Egger et al undertook a comparison of pairs of papers published in the same year by the same author in English and German journals. They found in 40 eligible such pairs that the only factor predicting publication in English language journals was a statistically significant result, with an odds ratio of 3.75 (95% CI 1·25–11·3). The authors concluded that 'English language bias may, therefore, be introduced in reviews and meta-analyses if they include only trials reported in English.'

Impact
In a review of 36 consecutive meta-analyses from 8 English language journals over 27 months, concluded that one of the 36 (selective decontamination of the digestive tract in intensive care units did not produce a significant change in mortality between treatment and control patients) would have arrived at a different conclusion (changing the OR from 0.70; 95% CI 0.45–1.09, to 0.67; 95% CI 0.47–0.95) if a paper written in German in a Swiss journal had been included in the analysis.

Methodological rigour
Moher and colleagues undertook a study analyzing the design, analysis and completeness of reporting of 229 trials, 96 of which were non-English. They found statistically significant differences between English and non-English studies, with less rigour in the latter: non-English ones were more likely to have adult participants, include two or more  interventions, and to compare two or more active treatments without an untreated control group. They were also less likely to report a clearly pre-specified primary outcome or any rationale for sample size estimation. Interestingly, the same group found in a later study no difference in overall quality (see diagram), but systematic reviews of trials in complementary and alternative medicine were of a higher quality than conventional medicine reviews. In this group, removal of non-English papers resulted in a reduced intervention effect.

Another study, comparing English and German papers, showed a greater likelihood of publication in English if there was a significant (i.e. p<0.05) difference between intervention and control arm, an odds ratio of 3.75 (95% CI 1·25–11·3).

Overall impact
In contrast, Jüni and colleagues found (in their analysis of 600 papers in 50 meta-analyses) a greater likelihood of papers with positive (i.e. p<0.05) findings being published in non-English papers (42%) than those in English (31%). However, they estimated that this measurable bias had less than 5% influence on the overall estimate of effect calculated in the meta-analyses (see diagram).

A more recent analysis of English and non-English papers included in meta-analyses found that whilst there were design and reporting differences, there was 'no evidence of a systematic bias from the use of language restrictions in systematic review-based meta-analyses in conventional medicine.' However, they added the caveat that within particular topics, it was possible that bias might occur (see example at the top of this section).

Preventive steps
Since one cannot know in advance if excluding non-English papers will result in a biased conclusion for any particular topic, ideally all studies relevant to that topic should be included in any systematic review of it. If, for any reason this is not possible, Egger has pointed out that undertaking a funnel plot may well give an indication of the presence of bias. However, as he points out, this technique is not completely reliable, especially if there is a limited number of small trials. Furthermore, as Terrin and colleagues found in an empirical test of the tool, even experienced researchers and systematic reviewers correctly interpret such plots (whether positively or negatively) on only about 50% of occasions.

Finally, Busse and colleagues have suggested a simple 'three-step rule' for identifying eligible articles for systematic reviews with large numbers of non-English articles: excluding languages with less than three articles; reviewing titles and abstracts for clear indications of eligibility, and noting the lack of a clearly reported statistical analysis unless the word “random” appears. This yielded accurate classification of 51 of 53 articles (sensitivity = 0.96; specificity = 0.70).

Cite as
Catalogue of Bias Collaboration. Byatt K. Language bias. In: Bias Wiki 2018. bias.wikia.com/wiki/language-bias/

Related biases
Publication bias