The following is excerpted and abbreviated from the Cochrane Handbook, Chapter 4.
Selecting Studies | Multiple Reports of the Same Studies | Selecting Studies | Selection Process | Selecting Excluded Studies | Software Support for Selecting Studies | Automating the Selection Process | Chapter 4 Information |
Two distinct processes are therefore required to determine which studies can be included in the review. One is to link together multiple reports of the same study; and the other is to use the information available in the various reports to determine which studies are eligible for inclusion.
As well as the studies that inform the systematic review, other studies will also be identified and these should be recorded or tagged as they are encountered, so that they can be listed in the relevant tables in the review:
records of ongoing trials for which results (either published or unpublished) are not (yet) available; and records of studies which seem to be eligible but for which data are incomplete or the publication related to the record could not be obtained.
Duplicate publication can introduce substantial biases if studies are inadvertently included more than once in a meta-analysis. Multiple reports of the same study should be collated, so that each study, rather than each report, is the unit of interest in the review.
MECIR Box 4.6.a Relevant expectations for conduct of intervention reviews
Collate multiple reports of the same study, so that each study, rather than each report, is the unit of interest in the review.
A typical process for selecting studies for inclusion in a review is as follows (the process should be detailed in the protocol for the review):
MECIR Box 4.6.b Relevant expectations for conduct of intervention reviews
Include studies in the review irrespective of whether measured outcome data are reported in a ‘usable’ way.
Decisions about which studies to include in a review are among the most influential decisions that are made in the review process and they involve judgement. Use (at least) two people working independently to determine whether each study meets the eligibility criteria. Ideally, screening of titles and abstracts to remove irrelevant reports should be done in duplicate by two people working independently.
MECIR Box 4.6.c Relevant expectations for conduct of intervention reviews
Duplicating the study selection process reduces both the risk of making mistakes and the possibility that selection is influenced by a single person’s biases.
It has been shown that using at least two authors may reduce the possibility that relevant reports will be discarded.
Experts in a particular area frequently have pre-formed opinions that can bias their assessment of both the relevance and validity of articles (Cooper and Ribble 1989, Oxman and Guyatt 1993).
Disagreements about whether a study should be included can generally be resolved by discussion. When the disagreement is due to a difference in interpretation, this may require arbitration by another person.
A single failed eligibility criterion is sufficient for a study to be excluded from a review. In practice, therefore, eligibility criteria for each study should be assessed in order of importance, so that the first ‘no’ response can be used as the primary reason for exclusion of the study...
For Cochrane Reviews the selection process must be documented in sufficient detail to be able to complete a flow diagram and a table of ‘Characteristics of excluded studies.’
MECIR Box 4.6.d Relevant expectations for conduct of intervention reviews
C41 : Documenting decisions about records identified (Mandatory)
A Cochrane Review includes a list of excluded studies called ‘Characteristics of excluded studies’, detailing the specific reason for exclusion for any studies...
By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies.
See the Tools & Resources Section of this LibGuide.
Research into automating the study selection process through machine learning and text mining has received considerable attention over recent years, resulting in the development of various tools and techniques for reviewers to consider...research suggests that adopting automation can reduce the need for manual screening by at least 30% and possibly more than 90%, although sometimes at the cost of up to a 5% reduction in sensitivity (O'Mara-Eves et al 2015).
Of particular practical use to Cochrane Review authors is a classifier (the ‘RCT Classifier’) that can identify reports of randomized trials based on titles and abstracts. The classifier is highly accurate because it is built on a large dataset of hundreds of thousands of records screened by Cochrane Crowd (http://crowd.cochrane.org/index.html), Cochrane’s citizen science platform, where contributors help to identify and describe health research (Marshall et al 2018).
Authors:
Carol Lefebvre, Julie Glanville, Simon Briscoe, Anne Littlewood, Chris Marshall, Maria-Inti Metzendorf, Anna Noel-Storr, Tamara Rader, Farhad Shokraneh, James Thomas, L. Susan Wieland; on behalf of the Cochrane Information Retrieval Methods Group.
Acknowledgements:
This chapter has been developed from sections of previous editions of the Cochrane Handbook co-authored since 1995 by Kay Dickersin, Julie Glanville, Kristen Larson, Carol Lefebvre and Eric Manheimer. Many of the sources listed in this chapter and the accompanying onlineTechnical Supplement have been brought to our attention by a variety of people over the years and weshould like to acknowledge this. We should like to acknowledge: Ruth Foxlee, (formerly) InformationSpecialist, Cochrane Editorial Unit; Miranda Cumpston, (formerly) Head of Learning & Support, Cochrane Central Executive; Colleen Finley, Product Manager, John Wiley and Sons, for checking sections relating to searching the Cochrane Library; the (UK) National Institute for Health and CareExcellence and the German Institute for Quality and Efficiency in Health Care (IQWiG) for support in identifying some of the references; the (US) Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program Scientific Resource Center Article Alert service; Tianjing Li, Co-Convenor, Comparing Multiple Interventions Methods Group, for text and references that formed the basis of there-draft ing of parts of Section 4.6 Selecting studies; Lesley Gillespie, Cochrane author and former Editor and Trials Search Co-ordinator of the Cochrane Bone, Joint and Muscle Trauma Group, for copy-editing an early draft ; the Cochrane Information Specialist Executive, the Cochrane Information Specialists’ Support Team, Cochrane Information Specialists and members of the Cochrane Information RetrievalMethods Group for comments on draft s; Su Golder, Co-Convenor, Adverse Effects Methods Group andSteve McDonald, Co-Director, Cochrane Australia for peer review.