Introduction | General Issues | Minimizing Bias | Sources to Search | Medline & Embase | Cochrane Central Register of Controlled Trials Other bibliographic databases | Ongoing Studies | Trials Register | Regulatory Agency | Other Resources | Structure of Search Strategy | Sensitivity v. Precision | Controlled Vocabulary | Restrictions | Search Filters | Peer Review | Alerts | Timing of Searches | When to Stop | Documenting the Search Process |
Key Points:
This chapter outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process, correctly document the search process and select studies from the search results.
Cochrane Reviews take a systematic and comprehensive approach to identifying studies that meet the eligibility criteria for the review. This chapter focuses on searching for randomized trials. Many of the search principles discussed, however, will also apply to other study designs. Considerations for searching for non-randomized studies are discussed in Chapter 24.
There is increasing evidence to support the involvement of an information specialist in the review to improve the quality of various aspects of the search process (Rethlefsen et al 2015, Meert et al 2016, Metzendorf 2016).
Cochrane Review Groups (CRGs)
Most CRGs offer support to authors in study identification from the early planning stage to the final write-up of the review, and the support available may include some or all of the following:
A search of MEDLINE alone is not considered adequate. Research evidence indicates that not all known published randomized trials are available in MEDLINE.
It is strongly recommended, however, that all search strategies should be peer reviewed, before being run, by a suitably qualified and experienced medical/healthcare librarian or information specialist (see Section 4.4.8).
For most of the sources listed in Section 4.3, the search process will retrieve individual reports of studies, so that multiple reports of the same study will need to be identified and associated with each other manually by the review authors.
MECIR Box 4.3.a Relevant expectations for conduct of intervention reviews
C19 : Planning the search (Mandatory)
Searches should be motivated directly by the eligibility criteria for the review, and it is important that all types of eligible studies are considered when planning the search.
Removing language restrictions in English language databases is not a good substitute for searching non-English language journals and databases.
C24 : Searching general bibliographic databases and CENTRAL (Mandatory)
Searches for studies should be as extensive as possible in order to reduce the risk of publication bias and to identify as much relevant evidence as possible.
The three bibliographic databases generally considered to be the most important sources to search for reports of trials are CENTRAL, MEDLINE (Halladay et al 2015, Sampson et al 2016) and Embase (Woods and Trewheellar 1998, Sampson et al 2003, Bai et al 2007). Wichor Bramer in his work notes that the optimal combination of sources is as follows, depending upon what is available:
Some bibliographic databases, such as MEDLINE and Embase, include abstracts for the majority of recent records. A key advantage of such databases is that they can be searched electronically both for words in the title or abstract and by using the standardized indexing terms, or controlled vocabulary, assigned to each record (see Section 4.3.1.2).
Cochrane has developed a database of reports of randomized trials called the Cochrane Central Register of Controlled Trials (CENTRAL), which is published within the Cochrane Library.
The Health InterNetwork Access to Research Initiative (HINARI) programme, set up by the World Health Organization (WHO) together with major publishers, provides access to a wide range of databases including the Cochrane Library for healthcare professionals in local, not-for-profit institutions in more than 115 countries, areas and territories.
The International Network for the Availability of Scientific Publications (INASP) also provides access to a wide range of databases (and journals) including the Cochrane Library.
Electronic Information for Libraries (EIFL) is a similar initiative based on library consortia to support affordable licensing of journals and other sources in more than 60 low-income and transition countries in central, eastern and south-east Europe, the former Soviet Union, Africa, the Middle East and South-east Asia.
See the Online Technical Supplement
PubMed (http://www.ncbi.nlm.nih.gov/pubmed/) provides access to a free version of MEDLINE that also includes up-to-date citations not yet indexed for MEDLINE (US National Library of Medicine 2018). Additionally, PubMed includes records from journals that are not indexed for MEDLINE and records considered ‘out-of-scope’ from journals that are partially indexed for MEDLINE (US National Library of Medicine no date).
MEDLINE is also available on subscription from a number of other database vendors, such as EBSCO, Ovid, ProQuest and STN. When searching MEDLINE via service providers or interfaces other than Ovid or PubMed, we recommend verification of the exact coverage of the database in relation to PubMed, where no explicit information on this is readily available.
Embase now includes all MEDLINE records, thus, technically, allowing both databases to be searched simultaneously. Further details on the implications of this for searching are available in the online Technical Supplement. There are more than 6 million records in Embase, from more than 2900 journals that are not indexed in MEDLINE (Elsevier 2016a). Embase includes articles from about 90 countries.
It is mandatory for Cochrane intervention reviews to include a search of Embase if access is available to the review team.
The online Technical Supplement provides guidance on how to search MEDLINE and Embase for reports of trials. The actual degree of reference overlap between MEDLINE and Embase varies widely according to the topic, but studies comparing searches of the two databases have generally concluded that a comprehensive search requires that both databases be searched (Lefebvre et al 2008)
Conversely, two recent studies examined different samples of Cochrane Reviews and identified the databases from which the included studies of these reviews originated (Halladay et al 2015, Hartling et al 2016). Halladay showed that the majority of included studies could be identified via PubMed (range 75% to 92%)
Many of the records in CENTRAL have been identified through systematic searches of MEDLINE and Embase (see online Technical Supplement). CENTRAL, however, also includes citations to reports of randomized trials that are not indexed in MEDLINE, Embase or other bibliographic databases; citations published in many languages; and citations that are available only in conference proceedings or other sources that are difficult to access. It also includes records from trials registers and trials results registers.
These additional records are, for the most part, identified by Cochrane Information Specialists, many of whom conduct comprehensive searches to populate CRG Specialized Registers, collecting records of trials eligible for Cochrane Reviews in their field.
...a search of the Specialized Register may be conducted instead of separately searching CENTRAL, MEDLINE and Embase for a specific review.
CENTRAL is available through the Cochrane Library.
Many countries and regions produce bibliographic databases that focus on the literature produced in those regions and which often include journals and other literature not indexed elsewhere. There are also subject-specific bibliographic databases, such as AMED (alternative therapies), CINAHL (nursing and allied health) and PsycINFO (psychology and psychiatry).
MECIR Box 4.3.b Relevant expectations for conduct of intervention reviews
C25 : Searching specialist bibliographic databases (Highly desirable)
regional databases (e.g.LILACS) should be considered...
It is important to identify ongoing studies, so that when a review is updated these can be assessed for possible inclusion.
Information about possibly relevant ongoing studies should be included in the review in the ‘Characteristics of ongoing studies’ table.
Even when studies are completed, some are never published.
Finding out about unpublished studies, and including their results in a systematic review when eligible and appropriate is important for minimizing bias.
Several studies and other articles addressing issues around identifying unpublished studies have been published (Easterbrook et al 1991, Weber et al 1998, Manheimer and Anderson 2002, MacLean et al 2003, Lee et al 2008, Chan 2012, Bero 2013, Schroll et al 2013, Chapman et al 2014, Kreis et al 2014, Scherer et al 2015, Hwang et al 2016, Lampert et al 2016).
There is no easy and reliable single way to obtain information about studies that have been completed but never published... [Sources] include trials registers and trials results registers (see Section 4.3.3), regulatory agency sources and clinical study reports (CSRs)...
Two earlier studies showed that a considerably higher proportion of CSRs prepared for regulatory approval of drugs provided complete information on study methods and results than did trials register records or journal publications (Wieseler et al 2012)...
those carrying out systematic reviews should continue to contact authors for missing data and that email contact was more successful than other methods (Young and Hopewell 2011).
Correspondence can be an important source of information about unpublished studies. It is highly desirable for authors of Cochrane Reviews of interventions to contact relevant individuals and organizations for information about unpublished or ongoing studies.
The RIAT (Restoring Invisible and Abandoned Trials) initiative (Doshi et al 2013) aims to address these problems by off ering a methodology that allows others to re-publish mis-reported and to publish unreported trials.
A recent study suggested that trials registers are an important source for identifying additional randomized trials (Baudard et al 2017).
Cochrane Reviews of interventions should search relevant trials registers and repositories of results (see MECIR Box 4.3.d). Although there are many other trials registers, ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform (ICTRP) portal (Pansieri et al 2017) are considered to be the most important for searching to identify studies for a systematic review.
it is not sufficient to search the ICTRP alone. Guidance for searching these and other trials registers is provided in the online Technical Supplement.
Agency for Healthcare Research and Quality (AHRQ) and the US Institute of Medicine (Institute of Medicine 2011) also advocate searching trials registers.
Potentially relevant regulatory agency sources include the EU Clinical Trials Register, Drugs@FDA andOpenTrialsFDA. Details of these are provided in the online Technical Supplement.
In late 2010, the European Medicines Agency (EMA) began releasing CSRs (on request) under their Policy 0043. In October 2016, they began to release CSRs under their Policy 0070.
Review authors should generally search sources such as dissertations and conference abstracts.
Review authors may also consider searching the internet, hand searching of journals and searching full texts of journals electronically where available. Theyshould examine previous reviews on the same topic and check reference lists of included studies and relevant systematic reviews.
MECIR Box 4.3.e Relevant expectations for conduct of intervention reviews
C28 : Searching for grey literature (Highly desirable)
Search relevant grey literature sources such as reports, dissertations, theses, databases and databases of conference abstracts. Searches for studies should be as extensive as possible in order to reduce the risk of publication bias and to identify as much relevant evidence as possible.
C29 : Searching within other reviews (Highly desirable)
C30 : Searching reference lists (Mandatory)
Many of the issues highlighted relate to both the subject aspects of the search (e.g. the PICO elements) and to the study method (e.g. randomized trials). For a search to be robust, both aspects require attention to be sure that relevant records are not missed.
Issues to consider in planning a search include:
The starting point for developing a search strategy is to consider the main concepts being examined in are view. This is often referred to as PICO – that is Patient (or Participant or Population or Problem), Intervention, Comparison and Outcomes.
Examples are provided in the appendices to the Cochrane Information Specialists’ Handbook (Littlewood et al 2017). It is usually unnecessary, however, and may even be undesirable, to search on every aspect of the review’s clinical question.
Therefore, in general databases, such as MEDLINE, a search strategy will typically have three sets of terms:
Typically, a broad set of search terms will be gathered for each concept, and combined with the OR Boolean operator to achieve sensitivity within concepts. The results for each concept are then combined using the AND Boolean operator, to ensure each concept is represented in the final search results.
Cochrane Reviews of public health interventions and of qualitative data may adopt very different search approaches to those described here.
The properties of searches are often quantified using ‘sensitivity’ (also called ‘recall’) and ‘precision’. Sensitivity is defined as the number of relevant reports identified divided by the total number of relevant reports in the resource. Precision is defined as the number of relevant reports identified divided by the total number of reports identified.
Increasing the comprehensiveness (or sensitivity) of a search will reduce its precision and will usually retrieve more non-relevant reports.
Article abstracts identified through a database search can usually be screened very quickly to ascertain potential relevance. At a conservatively estimated reading rate of one or two abstracts per minute, the results of a database search can be screened at the rate of 60–120 per hour (or approximately 500–1000 over an 8-hour period).
One is based on text words, that is terms occurring in the title, abstract or other relevant fields available in the database. The other is based on standardized subject terms assigned to the references by indexers.
MECIR Box 4.4.c Relevant expectations for conduct of intervention reviews
C33 : Developing search strategies for bibliographic databases (Mandatory)
Searches should capture as many studies as possible that meet the eligibility criteria, ensuring that relevant time periods and sources are covered and not restricted by language or publication status.
Evidence indicates that excluding non-English studies does not change the conclusions of most systematic reviews.
Inclusion of non-English studies may also increase the precision of the result and the generalizability and applicability of the findings.
If a Cochrane Review team requires help with translation of and/or data extraction from non-English language reports of studies, they should seek assistance todo so (this is a common task for which volunteer assistance can be sought via Cochrane’s TaskExchange.
Where it is not possible to extract the relevant information and data from non-English language reports, the review team should file the study in ‘Studies Awaiting Classification’ rather than ‘Excluded Studies’.
Search filters are search strategies that are designed to retrieve specific types of records, such as those of a particular methodological design.
Filters to identify randomized trials have been developed specifically for MEDLINE and Embase: see the online Technical Supplement for details.
The InterTASC Information Specialists’ Subgroup Search Filter Resource offers a collection of search filters (https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-systematicreviews), focusing predominantly on methodological search filters and providing critical appraisals of some of these filters.
MECIR Box 4.4.f Relevant expectations for conduct of intervention reviews
C34 : Using search filters (Highly desirable)
Use specially designed and tested search filters where appropriate including the Cochrane Highly SensitiveSearch Strategies for identifying randomized trials in MEDLINE. They should be assessed not only for the reliability of their development and reported performance, but also for their current accuracy, relevance and effectiveness.
It is strongly recommended that search strategies should be peer reviewed before the searches are run.Peer review of search strategies is increasingly recognized as a necessary step in designing and executing high-quality search strategies to identify studies for possible inclusion in systematic reviews.
An evidence-based checklist such as the PRESSEvidence-Based Checklist should be used to assess which elements are important in peer review of electronic search strategies.
Alerts, also called literature surveillance services, ‘push’ services or SDIs (selective dissemination of information), are an excellent method of staying up to date with the medical literature currently being published, as a supplement to designing and running specific searches for specific reviews.
The published review should be as up to date as possible. Searches for all the relevant databases should be rerun prior to publication, if the initial search date is more than 12 months (preferably six months) from the intended publication date.
Developing a search is often an iterative and exploratory process. It is oft en difficult to decide in a scientific or objective way when a search is complete and search strategy development can stop.
Stop if adding in a series of new terms to a database search strategy yields no new relevant records, or if precision falls below a particular cut-off.
Strategy is performing adequately. One simple test is to check whether the search is finding the publications that have been recommended as key publications or that have been included in other similar reviews. Also, an evidence-based checklist such as the PRESS Evidence-Based Checklist (McGowan et al 2016b) should be used to assess whether the search strategy is adequate.
Statistical techniques can be used to assess performance, such as capture-recapture (Spoor et al 1996) also known as capture-mark-recapture; (Kastner et al 2009).
Relative recall requires a range of searches to have been conducted so that the relevant studies have been built up by a set of sensitive searches.
If the search strategy found most of the studies that were available to be found in the database then it was likely to have been a sensitive strategy.
In evidence synthesis involving qualitative data, searching is oft en more organic and intertwined with the analysis such that the searching stops when new information ceases to be identified (Booth 2016).
Review authors should document the search process in enough detail to ensure that it can be reported correctly in the review (see MECIR Box 4.5.a). The searches of all the databases should be reproducible to the extent that this is possible.
MECIR Box 4.5.a Relevant expectations for conduct of intervention reviews
The search process (including the sources searched, when, by whom, and using which terms)needs to be documented in enough detail throughout the process to ensure that it can be reported correctly.
Medical/healthcare librarians and information specialists involved with the review should draft , or at least comment on, the search strategy sections of the review prior to publication.
The PRISMA-Search (PRISMA-S) Extension (Rethlefsen et al 2021), an extension to the PRISMA Statement, addressing the reporting of search strategies in systematic reviews, should go some way to addressing this, as should the major revision of PRISMA itself, which is due to report in 2021.
For Cochrane Reviews, the bibliographic database search strategies should be copied and pasted into an appendix exactly as run and in full, together with the search set numbers and the total number of records retrieved by each search strategy. The search strategies should not be re-typed, because this can introduce errors.
Many database interfaces have facilities for search strategies to be saved online or to be emailed; an off line copy in text format should also be saved.
Documenting the searching of sources other than databases, including the search terms used, is also required if searches are to be reproducible. Details about contacting experts or manufacturers, searching reference lists, scanning websites, and decisions about search iterations can be kept internally for future updates or external requests and can be reproduced as an appendix in the final document. Since the purpose of search documentation is to support transparency...
The searcher should record any correspondence on key decisions and report a summary of this correspondence alongside the search strategy. The narrative describes the major decisions that shaped the strategy and can give a peer reviewer an insight into the rationale for the search approach.
It is particularly important to save locally or file print copies of any information found on the internet, such as information about ongoing and/or unpublished trials, as this information may no longer be accessible at the time the review is written. Local copies should be stored in a structured way to allow retrieval when needed. There are also web-based tools which archive webpage content for future reference, such as WebCite (http://www.webcitation.org/).
Decisions should be documented for all records identified by the search. Details of the flow of studies from the number(s) of references identified in the search to the number of studies included in the review will need to be reported in the final review, ideally using a flow diagram such as that proposed by PRISMA.
A table of ‘Characteristics of excluded studies’ will also need to be presented.
The flow diagram records initially the total number of records retrieved from various sources, then the total number of studies to which these records relate. Lists of included and excluded studies must be based on studies rather than records.
Lefebvre C, Glanville J, Briscoe S, Featherstone R, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Paynter R, Rader T, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.