Aarhus University Seal

Systematic reviews

  • What is a systematic review and how do I perform one?

What is a systematic review?

Within some fields there is an increasing tendency towards literature searches being required to be carried out as systematic reviews. There are specific requirements for a systematic review, and it is often a very time-consuming task. The following definition outlines the scope of a systematic review:

A systematic review is a structured and pre-planned synthesis of original studies involving predefined research question, inclusion criteria, search methods, selection procedure, quality assessment, data extraction, and data analysis. No original study should deliberately be excluded without explanation, and the results from each study should justify the conclusion."
LUND, H., JUHL, C. & CHRISTENSEN, R. 2016. Systematic reviews and research waste. The Lancet, 387, 123-124.

Depending on time and resources, you should consider whether there is another type of review that might be better suited to your research project.

See also AU Library's pages on systematic literature search.

Need help?

If you have any questions or concerns, feel free to contact the liason librarian associated with your field, who will be happy to assist you with your questions.

Alternatively, you are always welcome to contact your local library.  

Need access to Covidence?

Health - contact: 

Nat-Tech - contact: 

Arts - contact:

BSS - contact:

Read more about Covidence and other types of software for systematic reviews.

The requirements of a systematic review

A systematic review is characterized by the following requirements being met:

  • A predefined search protocol that specifies the purpose, key search terms, methods and inclusion and exclusion criteria for the systematic review.
  • Precise documentation of the individual steps and decisions made along the way, so that the reader can follow and assess the reviewer’s methods.
  • The literature search should, in principle, identify everything that has been written about the research question (including grey literature) in order to avoid bias. · The use of established standards in assessing the quality of the studies included and the reporting of the results (e.g. PRISMA).
  • It often requires two people or a team to perform a systematic review, especially when screening articles and extracting data.  

Types of reviews

There are several different types of review, of which the systematic review is just one. Read short descriptions of various review types and find references to relevant literature on the subject.

Research literature reviews

  • Overall term for reviews that make use of the systematic and transparent approach used in systematic reviews. Meaning that a thorough search of the academic literature within a specific field in a structured and systematic manner is made, but that the established standards for search protocols and reporting are not necessarily employed. The aim is often to identify what has been written on a certain research issue and is similar to the process termed a scoping review.

Systematic reviews

  • The literature search should, in principle, identify everything that has been written about a research issue (including grey literature,) in order to avoid bias in the background data/literature used. There are predefined protocols and standards for how you should perform literature searches, outlining how you should assess the quality of the literature found (critical appraisal) and how your report should look (e.g. PRISMA).

Scoping reviews

  • Often referred to as a preliminary version of a systematic review, as you create a map of the existing literature within a field - how much is written, how is the subject discussed, and where are there holes or needs for further work within the field. A scoping review, in other words, provides the context for a given research area. It typically covers a slightly broader area than the subject itself, and includes various types of study.

Meta-analyses

  • Often related to a systematic review. Here you use statistical methods to aggregate quantitative data from at least two and ideally several studies on a given research issue. The studies must be functionally comparable empirical studies.

Traditional narrative reviews

  • Here the basis for the literature review is often a subjectively selected body of literature that an expert in the field then assesses. The aim of a traditional narrative review is to present “the current state of knowledge” about a specific topic or subject area, often with a desire to develop a new perspective or interpretation via the review.


Read more about reviews

Protocols

  • CAMPBELL COLLABORATION 
    Contact "Coordinating groups" within the different fields.
  • OPEN SCIENCES FRAMEWORK REGISTRY 
    OSF is a free, open-source collaboration platform for researchers. Here you can share publications, data and ongoing research projects.  
  • PROSPERO (International prospective register of systematic reviews)
    Database for registering protocols for systematic reviews, so that your review work can be made visible even in the early stages of the process. The intention is to avoid duplication and to make more transparent any change of focus that occurred during the review (in order to avoid publication bias). Contains a checklist for the content of the protocol.
  • PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols)
    Checklist to develop and report on the protocol.

Focus on the research question

PICO

PICO is a model which helps you to formulate a focused question and create the basis for a literature search. The focused question helps to structure, define and delimit the literature search. The focused question can be grounded in the PICO model, which describes the 4 aspects of clinical questions:

P = the patient or problem being addressed
I = the intervention or exposure being considered
C = the comparison intervention or exposure where relevant
O = the clinical outcomes of interest  

An example of a research questions could be: "Mentalization based therapy is an effective form of treatment for women with borderline personality disorder?"  

If you used PICO to structure the question it might look like this:

 

In addition to PICO there are a number of other tools, such as:

  • PIRO (Population, Index test, Reference standard and Outcomes)
  • SPICE (Setting (context) – Perspective– Intervention – Comparison – Evaluation)
  • SPIDER (Sample ‐ Phenomenon of Interest – Design (of study) – Evaluation ‐ Research type)

Software for systematic reviews

  • Covidence 
    Web-based tool that will help you through the process of sorting your references, attaching PDF-files, and keeping track of which references you have chosen to exclude and include with note attachments.
    You can invite colleagues to assess the references. 
    Employees and students at Aarhus University as well as employees at Aarhus University Hospital can create a profile.
    To get access, you must contact your library.  
    Others have the option of a free trial, but with a limit of 500 references. 
  • EPPI reviewer
    Software which can be used for all types of review; literature reviews, systematic reviews, meta-analyses, narrative reviews, etc. The system is developed by the EPPI-Centre at the Social Science Research Unit in the Department of Social Science, UCL Institute of Education, University College London. Payment is required for use. AU does not have access.
  • DistillerSR
    Software for managing the process of making a systematic review. The program is Canadian and developed together with McMaster. Payment is required for use. AU does not have access.
  • Rayyan - a web and mobile app for systematic reviews
    The program can keep track of references at the title/abstract level, but it can not handle full-text. It is developed in Qatar and is free of charge.
  • SR ToolBox
      A web-based catalogue of tools to support the process of making systematic reviews across multiple domains. It is developed by Christopher Marshall from the York Health Economics Consortium at the University of York and is freely accessible.    

Assessing the quality of studies

Critical appraisal refers to the systematic analysis and assessment of research articles with the aim of assessing their reliability and quality. There are various checklists depending on the kind of study design employed by the article being assessed. The list below is not complete, but for inspiration:

Reporting standards

Reporting standards have been developed with the aim of ensuring complete, accurate and transparent reporting of research studies.

  • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses)
    Tool consisting of a checklist and a flowchart including the minimum requirements for the reporting of the systematic review or meta-analyses. 
  • Equator network (Enhancing the QUAlity and Transparency Of health Research)
    An international network that works to improve the quality of medical research literature by promoting the transparent and accurate reporting of studies.

Relevant resources for systematic reviews

  • August
    Group at Aarhus University, which works with systematic reviews within animal science.  
  • ClinicalTrials.gov 
    Database where you can register and search for protocols for clinical studies. The intention is to counter publication bias and raise awareness of current studies. Registration is voluntary (although it is mandatory for some types of study in the USA).
  • Grey literature
    You can find useful links to databases of grey literature at the Karolinska Institute’s university library.
  • KUB Systematic Review Service
    Copenhagen University Library has created a libguide for systematic reviews.
  • PRESS Evidence Based Checklist (Peer Review of Electronic Search Strategies) 
    The report presents an evidence-based checklist that can be used so that you yourself, or a colleague, can assesses your search strategy. The checklist can be found on page 39.   
  • SYRCLE
    Centre at the Radboud University Medical Center, which conducts, and teaches how to perform, systematic reviews for animal studies.

Literature on systematic reviews

Particularly for Arts

  • Hart, C. (2018). Doing a literature review: Releasing the social science research imagination (2nd ed.). London: Sage Publications. 
    This book is a powerful contribution, partly because it is targeted Master's thesis students and PhD students. The book presents methods and techniques for the development of literature reviews and goes into depth with the various steps in the process and the skills required.  
  • Greenhalgh, T. , Thorne, S.E. & Malterud, K. (2018). Time to challenge the spurious hierarchy of systematic over narrative reviews? European Journal of Clinical Investigation, 48(6), e12931. doi: 10.1111/eci.12931.
    In the article, the authors define differences and similarities between "systematic reviews" and "narrative reviews, and discuss the possibilities and limitations of the different types of research summaries.
  • Grønning, A., Drotner, K. & Rasmussen, L. T. (2018). Et diskussionspapir om litteratursøgning på Humaniora. Institut for Kulturvidenskaber Syddansk Universitet, U6.
    This paper describes why the need increases in the field of humanistic research for knowledge, skills and competences in literature searches, and it indicates how humanistic researchers and students produce and rate literature summaries.
  • Maclure, M. (2005). "Clarity bordering on stupidity": where’s the quality in systematic review?Journal of Education Policy 20(4), 393-416. doi: 10.1080/02680930500131801.
    Here a critical perspective on systematic reviews is presented as a source of scientific knowledge in educational research. The author proposes to consider whether in-depth reading of few sources can contribute more to a research project than superficial screening of many.     

Particularly for BSS

If you only have time to tackle one single text, we recommend the article by Papaioannou et al., 2010.

  • Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: Consideration of a range of search techniquesHealth Information and Libraries Journal, 27(2), 114. doi:10.1111/j.1471-1842.2009.00863.x.
    The article describes the complexity involved in performing a systematic review within the social sciences. The authors demonstrate, via a case study about e-learning, how they have used several different approaches and search methods to identify the relevant literature.
  • Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell Pub.
    The book is a thorough exploration of a seven-point guide to making systematic reviews within the social sciences. Each step in the process is described and supplemented with examples, checklists and guidelines.

Particularly for Health


Systematic literature searches

Tackle your search process in a structured manner and organise your search in advance. You can minimise the risk of reproducing already existing research and have a greater chance of avoiding bias.

Alerts

Read more about how to get alerts on articles, new editions of journals, as well as new results for previously performed searches.

Reference management

AU Library supports the reference management tool EndNote, which all employees and students at Aarhus University have access to for free.