Systematic reviews

  • What is a systematic review and how do I perform one?

What is a systematic review?

Within some fields there is an increasing tendency towards literature searches being required to be carried out as systematic reviews. There are specific requirements for a systematic review, and it is often a very time-consuming task. We feel the following definition outlines the scope of a systematic review::

A systematic review is a structured and pre-planned synthesis of original studies involving predefined research question, inclusion criteria, search methods, selection procedure, quality assessment, data extraction, and data analysis. No original study should deliberately be excluded without explanation, and the results from each study should justify the conclusion."
LUND, H., JUHL, C. & CHRISTENSEN, R. 2016. Systematic reviews and research waste. The Lancet, 387, 123-124.

Depending on time and resources, you should consider whether there is another type of review that might be better suited to your research project.

See also AU Library's pages on systematic literature search.

Need help?

 If you have any questions or concerns, feel free to contact the liason librarian associated with your field, who will be happy to assist you with your questions.

  Alternatively, you are always welcome to contact your local service location.  

The requirements of a systematic review

A systematic review is characterized by the following requirements being met:

  • A predefined search protocol that specifies the purpose, key search terms, methods and inclusion and exclusion criteria for the systematic review.
  • Precise documentation of the individual steps and decisions made along the way, so that the reader can follow and assess the reviewer’s methods.
  • The literature search should, in principle, identify everything that has been written about the research question (including grey literature) in order to avoid bias. · The use of established standards in assessing the quality of the studies included and the reporting of the results (e.g. PRISMA).

Types of reviews

There are several different types of review, of which the systematic review is just one. Read short descriptions of various review types and find references to relevant literature on the subject.

 

Research literature reviews

  • Overall term for reviews that make use of the systematic and transparent approach used in systematic reviews. Meaning that a thorough search of the academic literature within a specific field in a structured and systematic manner is made, but that the established standards for search protocols and reporting are not necessarily employed. The aim is often to identify what has been written on a certain research issue and is similar to the process termed a scoping review.

 

Systematic reviews

  • The literature search should, in principle, identify everything that has been written about a research issue (including grey literature,) in order to avoid bias in the background data/literature used. There are predefined protocols and standards for how you should perform literature searches, outlining how you should assess the quality of the literature found (critical appraisal) and how your report should look (e.g. PRISMA).

 

Scoping reviews

  • Often referred to as a preliminary version of a systematic review, as you create a map of the existing literature within a field - how much is written, how is the subject discussed, and where are there holes or needs for further work within the field. A scoping review, in other words, provides the context for a given research area. It typically covers a slightly broader area than the subject itself, and includes various types of study.

 

Meta-analyses

  • Often related to a systematic review. Here you use statistical methods to aggregate quantitative data from at least two and ideally several studies on a given research issue. The studies must be functionally comparable empirical studies.

 

Traditional narrative reviews

  • Here the basis for the literature review is often a subjectively selected body of literature that an expert in the field then assesses. The aim of a traditional narrative review is to present “the current state of knowledge” about a specific topic or subject area, often with a desire to develop a new perspective or interpretation via the review.


Read more about reviews

Protocols

  • PROSPERO (International prospective register of systematic reviews)
    Database for registering protocols for systematic reviews, so that your review work can be made visible even in the early stages of the process. The intention is to avoid duplication and to make more transparent any change of focus that occurred during the review (in order to avoid publication bias). Contains a checklist for the content of the protocol.
  • PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols)
    Checklist to develop and report on the protocol.

Focus on the research question

PICO

PICO is a model which helps you to formulate a focused question and create the basis for a literature search. The focused question helps to structure, define and delimit the literature search. The focused question can be grounded in the PICO model, which describes the 4 aspects of clinical questions:

P = the patient or problem being addressed
I = the intervention or exposure being considered
C = the comparison intervention or exposure where relevant
O = the clinical outcomes of interest  

An example of a research questions could be: "Mentalization based therapy is an effective form of treatment for women with borderline personality disorder?"  

If you used PICO to structure the question it might look like this:

 

In addition to PICO there are a number of other tools, such as:

  • PIRO (Population, Index test, Reference standard and Outcomes)
  • SPICE (Setting (context) – Perspective– Intervention – Comparison – Evaluation)
  • SPIDER (Sample ‐ Phenomenon of Interest – Design (of study) – Evaluation ‐ Research type)

Software til systematic reviews

  • Covidence 
    Web-based tool that will help you through the process of sorting your references, attaching PDF-files, and keeping track of which references you have chosen to exclude and include with note attachments. It is possible to export the results to RevMan and MAGIC, as well as print a flowchart with statistics. You can invite colleagues to assess the references. Covidence is used by Cochrane and the Danish Health Authority. Covidence is free of charge for one review with two participants, after which payment is necessary.
  • EPPI reviewer
    Software which can be used for all types of review; literature reviews, systematic reviews, meta-analyses, narrative reviews, etc. The system is developed by the EPPI-Centre at the Social Science Research Unit in the Department of Social Science, UCL Institute of Education, University College London. Payment is required for use. AU does not have access.
  • DistillerSR
    Software for managing the process of making a systematic review. The program is Canadian and developed together with McMaster. Payment is required for use. AU does not have access.
  • Rayyan - a web and mobile app for systematic reviews
    The program can keep track of references at the title/abstract level, but it can not handle full-text. It is developed in Qatar and is free of charge.
  • SR ToolBox
      A web-based catalogue of tools to support the process of making systematic reviews across multiple domains. It is developed by Christopher Marshall from the York Health Economics Consortium at the University of York and is freely accessible.    

Assessing the quality of studies

Critical appraisal refers to the systematic analysis and assessment of research articles with the aim of assessing their reliability and quality. There are various checklists depending on the kind of study design employed by the article being assessed.

Particularly for Health

  • Systematic reviews
    AMSTAR (A Measurement Tool to Assess Systematic Reviews).
    ROBIS (Risk of bias in systematic review)
  • Guidelines
    AGREE II (The Appraisal of Guidelines for Research and Evaluation).
  • Randomized trials 
    RCT (risk of bias assessment tool, Cochrane Collaboration).
  • ROBINS-I tool (Risk Of Bias In Non-randomized Studies - of Interventions).
  • Diagnostic primary studies 
    QUADAS-2 (A quality assessment tool for diagnostic accuracy studies).
    GRADE (Grading of Recommendations Assessment, Development and Evaluation)  
    An internationally used systematic and transparent system for assessing the quality of evidence and providing recommendations.

 

Particularly for BSS

  • AMSTAR (A Measurement Tool to Assess Systematic Reviews)
    AMSTAR can be used to assess the quality of the methodology employed in systematic reviews, and as a guide to performing a systematic review.
  • CASP qualitative studies (Critical Appraisal Skills Programme)
    10 questions to be considered when assessing qualitative research.
  • CASP systematic reviews (Critical Appraisal Skills Programme)
    10 questions to be considered when assessing a systematic review.
  • GRADE (Grading of Recommendations Assessment, Development and Evaluation) 
    An internationally used system for assessing the quality of evidence in academic literature (primarily within the health sciences). The evidence is assessed according to the outcome measures across studies, and the evaluation criteria are the risk of bias, inconsistencies, inaccuracies, etc.
  • SURE qualitative studies (Specialist Unit for Review Evidence)
    10 questions to be considered when assessing qualitative research.
  • SURE systematic review (Specialist Unit for Review Evidence)
    11 questions to consider when assessing a systematic review.

Reporting standards

Reporting standards have been developed with the aim of ensuring complete, accurate and transparent reporting of research studies.

  • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses)
    Tool consisting of a checklist and a flowchart including the minimum requirements for the reporting of the systematic review or meta-analyses. 
  • Equator network (Enhancing the QUAlity and Transparency Of health Research)
    An international network that works to improve the quality of medical research literature by promoting the transparent and accurate reporting of studies.

Relevant resources for systematic reviews

  • August
    Group at Aarhus University, which works with systematic reviews within animal science.  
  • ClinicalTrials.gov 
    Database where you can register and search for protocols for clinical studies. The intention is to counter publication bias and raise awareness of current studies. Registration is voluntary (although it is mandatory for some types of study in the USA).
  • Grey literature
    You can find useful links to databases of grey literature at the Karolinska Institute’s university library.
  • PRESS Evidence Based Checklist (Peer Review of Electronic Search Strategies) 
    The report presents an evidence-based checklist that can be used so that you yourself, or a colleague, can assesses your search strategy. The checklist can be found on page 39.   
  • SYRCLE
    Centre at the Radboud University Medical Center, which conducts, and teaches how to perform, systematic reviews for animal studies.

Literature on systematic reviews

Particularly for Health 

 

Particularly for BSS

If you only have time to tackle one single text, we recommend the article by Papaioannou et al., 2010.


Systematic literature searches

Tackle your search process in a structured manner and organise your search in advance. You can minimise the risk of reproducing already existing research and have a greater chance of avoiding bias.

Alerts

Read more about how to get alerts on articles, new editions of journals, as well as new results for previously performed searches.

Piktogram - referencehåndtering

Reference management

AU Library provides a number of Reference mangement tools to emplyees and students at Aarhus University.