Some useful tips on how to conduct a systematic review

This is a very old article I wrote in 2013, but given the recent events in my academic career, I am posting it publicly to help anyone who sets out to perform such a task.

There are countless manuals on how to do a good Systematic Review, so here I will provide a collection of ideas that I have strictly copied from the authors of the references and include what I did to maintain my sanity during the process.

…

One of the main facts today is that we live in the information age where the volume of data and information generated increases almost exponentially every year.

This fact has a huge impact when we talk about academic research, specifically for researchers who wish to understand what is being written even amidst this myriad of information being generated every passing day.

This post is specially dedicated to:

1. People who are at the moment of defining their research project for a PhD or Master’s degree.
2. People who are writing a scientific article but would like to know what is being discussed in the literature.
3. People who are doing corporate research to determine practical courses of action in some R&D department.

One of the most underestimated academic techniques, in my opinion, to resolve this regarding research activity is the Systematic Review.

Personally, I cannot imagine serious scientific research starting without the use of this tool, and by the end of this article, this issue will be clearer.

But what is a systematic review? Here I borrow the citation from Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997):

Systematic reviews are scientific investigations in themselves, with pre-planned methods and an assembly of original studies as their “subjects.” They synthesize the results of multiple primary investigations by using strategies that limit bias and random error (9, 10). These strategies include a comprehensive search of all potentially relevant articles and the use of explicit, reproducible criteria in the selection of articles for review. Primary research designs and study characteristics are appraised, data are synthesized, and results are interpreted.

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997)

The perception I have is that countless works start full of expectations and promises of originality, but most of the time they are works that only reinvented the wheel over other people’s works who didn’t get the credit. Had there been a more thorough systematic review, these works would either receive fewer resources or not exist at all, and resources could be allocated to other areas with greater potential for relevance/return.

If I had to summarize the basic points of the importance of systematic reviews, I would consider the following:

1. The Systematic Review helps to understand the past of a topic within a field of science and its evolution over time;
2. It presents the current state of the art for researchers of the present;
3. It is an important tool to discover gaps and limitations (e.g.** methodological) in the current literature;**
4. It makes your research converse with the current literature from the day of publication;
5. It shows ways of monitoring the literature by bringing in relevant sources; and last but not least;
6. It prevents researchers from reinventing the wheel, allocating resources for the development of works with a higher degree of originality.

Here I agree with the statement by Webster and Watson (2002) that the Systematic Review unites concepts over time but occasionally prepares for the future, where the review understands theory and practice and the ontological relationship of the field of study.

As Webster and Watson (2002) state, systematic or literature reviews cannot be a compilation of citations like a phone book, but rather an active exercise in analyzing the studies being reviewed.

Systematic reviews can help practitioners keep abreast of the medical literature by summarizing large bodies of evidence and helping to explain differences among studies on the same question. A systematic review involves the application of scientific strategies, in ways that limit bias, to the assembly, critical appraisal, and synthesis of all relevant studies that address a specific clinical question.

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997)

One of the points I want to highlight at some point in the future is how systematized research should be the goal of any company to incorporate data into decision-making processes and the architecture of new corporate solutions. My argument is the same as Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997), which I quote below:

Review articles are one type of integrative publication; practice guidelines, economic evaluations, and clinical decision analyses are others. These other types of integrative articles often incorporate the results of systematic reviews. For example, practice guidelines are systematically developed statements intended to assist practitioners and patients with decisions about appropriate health care for specific clinical circumstances (11). Evidence-based practice guidelines are based on systematic reviews of the literature, appropriately adapted to local circumstances and values. Economic evaluations compare both the costs and the consequences of different courses of action; the knowledge of consequences that are considered in these evaluations is often generated by systematic reviews of primary studies. Decision analyses quantify both the likelihood and the valuation of the expected outcomes associated with competing alternatives.

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997)

A point about systematic reviews is that many journals, due to space limitations, generally limit the number of pages of works, and unfortunately, the first area to be sacrificed is the literature review because it does not have as much focus as the methodology, results, or conclusions.

Here I will gather some tips from 3 articles that I consider to be good references regarding systematic reviews. These are some notes from these articles, along with some comments on what I do when I have to perform a systematic review to see the state of the art of a research topic.

The fundamental basis of this article is in the ideas of Cook, Mulrow, & Haynes (1997); Webster and Watson (2002), and Brereton et al. (2007). This will only be a non-exhaustive list of topics, and reading the originals is essential.

If I had to choose a framework for adopting systematic reviews, it would be this one by Brereton et al. (2007):

(Note: The original article includes an image of the framework here)

Authors and Prospective Topics

Positioning on progress and learning and embarking on new projects for the development of new theoretical models according to Webster and Watson (2002), where these reviews can be on a) a mature topic with a vast body of knowledge or b) an emerging topic with a higher speed of development.

The review of topics gives directions on concepts and their evolution and directions, and the review of authors helps the work communicate with major laboratories or researchers who will assist in the debate about the scientific field.

Writing a Systematic Review Article

What is being sought? What are the keywords? What is the expected contribution?

One of the most important points is to perform a disclosure of the limitations of the review such as:

• Scope of search (e.g. keywords used, article database)
• Time limit of the articles
• Summary of past research, highlight on gaps, proposals on how to shorten this gap, and implications of theory in practice.

This disclosure signals that your work is taking a snapshot of the literature at the moment, and that it may not be perfect due to methodological biases or even factors exogenous to your research, such as if a database changes the article indexer and the query with the same parameters brings other results.

Identification of Relevant Literature

Here the systematic review focuses on the concept regardless of where these concepts are.

This implies saying that the focus is not only:

• On the best journals
• On some most productive authors
• In some areas of knowledge
• On issues of geographical scope of the country

Choice of Databases

Here the review takes on more of an air of art than science indeed, and here comes a very personal view: I particularly like to deal with more than 5 research databases. I found this number through some experimentation, but it was the number that gives me a certain breadth regarding the articles that are indexed in the best journals and helps to catch some good articles and especially PhD theses that might be hidden on page 8 of some obscure keyword.

Another point of the database is to understand its selectivity, and selectivity here I call how much the search tool can bring me a sufficiently relevant number of articles with the lowest index of signal and noise.

And as I couldn’t fail to mention, it is always tempting to go only where we are most familiar, such as Google Scholar and Microsoft Research; but the tip here is to look for databases from other areas of knowledge.

Review Structure

It should focus mainly on concepts and not on authors.

One thing that helps a lot is the qualitative categorization of articles, where aspects of gaps, type of methodology, and nature of the work can be compiled later.

Theoretical Development

The point here is based on the past, the current state of things, and the limitations and gaps present; how to use this for the future?

Here I recommend a modest expansion of ideas, something that is not a 10-year research into the future and has plausibility.

Reason for Proponents

• Theoretical explanations (The why?): This will be the glue that will stick all practice in a systematized, reproducible, observable, transferable, and replicable way;
• Empirical findings from the past: The support of what was observed over time, and the quality of the evidence presented and how these observations were performed; and
• Practice and experience: Mechanisms for theory validation and subsequent refinements of what is being theorized and done.

Conclusion

I think at this point I managed to show my point regarding the importance of systematic review as a tool for understanding the past and the present, as well as a method that helps plan the future in terms of research.

I personally recommend always using this tool before any academic project whenever there is some kind of practice adoption.

References

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Systematic reviews: synthesis of best evidence for clinical decisions. Annals of internal medicine, 126(5), 376-380. - Link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.733.1479&rep=rep1&type=pdf

Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lessons from applying the systematic literature review process within the software engineering domain. Journal of systems and software, 80(4), 571-583. - Link: https://www.sciencedirect.com/science/article/pii/S016412120600197X

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS quarterly, xiii-xxiii. Link: https://www.researchgate.net/profile/Harald_Kindermann/post/How_to_write_the_academic_review_article_in_the_field_of_management/attachment/5abe1af54cde260d15d5d477/AS%3A609838266593280%401522408181169/download/2002_Webster_Writing+a+Literature+Review.pdf