Welcome back intrepid systematic meta fans. In the last exciting post, we covered the steps involved in planning review studies. This sequel post will guide us through the next steps, which involves, conducting searches, the screening and selection of studies, data extraction, quality assessment, analysis (and synthesis), and finally reporting.
Are you ready for further systematic meta-ing?
- Systematic and Manual Search.
We developed our review questions and search strategy, we published our protocol, and now we are ready to conduct our systematic search. We sometimes use a gateway to access a range of databases and conduct the search at the same time. For example, Ovid gateway can be used to perform the search on EMBASE, MEDLINE, and PsycINFO. These gateways can be discussed with the academic librarian. Some studies are being indexed in multiple databases and you will get the same study in each database separately. These studies should be treated as a single paper, and we need to remove the duplications., You may also want to manually look for studies. For example, you can identify some relevant journals and check their publications, particularly the recent publications. Furthermore, you can check the reference lists of some included studies and check studies that cited a specific relevant study. Manuel search is usually conducted after completing the systematic search. Sometimes, we obtain a study that it is relevant, but it did not report the data we need such as an effect size. In this case, we can contact the corresponding author of the paper and request this data. Our aim is to include all the literature based on our eligibility criteria, and as you increase your methods for search you will get more comprehensive results. Lastly, do not forget to note the number of studies yielded in each search separately. The search might be updated later, and the most recent studies can be included before the final analyses.
- Screening and Selecting Studies.
You will probably end up with 100s of studies after conducting the search. First thing to do is to record the number of studies yielded from the search, and then remove all duplicates from the different databases. Then, you need to export all titles, abstracts, and their links (DOIs) from databases for study selections. In order to manage all this process, a reference manager such as Endnote, Zotero, Mendeley can be used. In addition to these reference manager tools, Covidence software can be used for screening and selecting studies and for data extraction. Covidence is developed for intervention reviews, and you can use your university account to have free access. If these tools are not suitable for your review, you can simply use Microsoft excel or Microsoft word. During the screening, it is important to take notes about your screening decisions (i.e., why did you exclude the study). Now it is time to review all studies and see which shall be included and which shall be excluded. The screening and study selection stage should be completed by two reviewers independently if possible. The first step is to read titles and abstracts and exclude any further duplicates, and any clearly irrelevant studies based on your eligibility criteria. Next, we need to obtain full texts of each study and review them to decide if a study should be included or excluded. At this stage, the reason for exclusion should be noted. It is recommended to use PRISMA flow diagram to demonstrate the screening and study selection process.
- Data Extraction.
All included studies are ready. Now we need to extract the data we need for analyses and for descriptive purposes. We can develop a data extraction form and extract the data we need such as sample sizes, the measurement tools used, the location of data collection, the effect sizes etc. This can be piloted with few studies first to check if everything works properly. While Covidence software can be used for data extraction, Microsoft excel is also a useful tool at this stage. Like the study selection process, multiple reviewers are preferred for extracting data to avoid any errors. The descriptive information and the key findings of included studies are usually presented in a table.
- Quality assessment.
This is assessing the quality of your included studies (e.g., case-control design, validity of the measures, blinding). What you may assess and what tool you may use its going to depend on your review questions. Ideally, the quality assessment of your included studies should be done by at least two raters independently (i.e., two reviewers). We suggest piloting the tool, to assess some studies independently (e.g., 10% of them) and then meeting to check the results. This allows for calibration of the criteria you are using, to ensure clarity and consistency between reviewers (e.g., discuss use of the tool, and adjust any criterion if possible). If there is still disagreement after rating the studies, you may ask for help from another reviewer of your team. There are several guidelines and tools for the quality appraisal. The existing tools may be developed specifically for cross-sectional reviews or intervention reviews. Or they might have a specific focus, such as assessing the reporting quality or methodological quality of studies. So, first check the existing tools. If you can’t find something suitable for your review questions, then you may consider modifying/adapting one of the existing tools or developing your own . For new tools, you may focus on the sampling, measurement, and/or design among other options.
- Analysis & Synthesis (meta-analysis itself).
Meta-analysis is a way (among others) of analysing and synthesising quantitative data collected within a systematic-review framework. What data you my analyse and how you may analyse your data, depends on your review questions. The most common meta-analytic questions are based on correlation data (r) or mean differences data (X̅1 – X̅2). For this, we need to convert the data into ‘effect sizes’ (Cohen’s d or Hedge’s g) so that we can compare studies and compute a summary effect (i.e., weighted mean of these effect sizes).
If your interest is to compare the correlation between two variables across different studies, then you may make a question like: are the effect sizes of the association between two variables different across the studies? (See Spruit et al., 2019).
If your focus is on checking whether the mean difference is in favour of a specific treatment amongst multiple studies, then you may have a question like: are the effect sizes of an intervention different between studies? (See Spruit et al., 2016).
If you are wondering whether the accuracy of a measurement varies across several studies, then may make a question like: are the effect sizes of instrument accuracy different between studies? (See Youngstrom et al., 2015).
If the answer to any of the above questions is ‘yes, there is a difference between the effect sizes’, then you may want to explain why there is such heterogeneity. Having heterogeneity does NOT make your meta-analysis less robust. It is a great opportunity for you to understand why the effect sizes are different across your included studies.
This is called ‘moderator analysis’ or ‘subgroup analysis’. You may have look at whether the differences between the effect sizes of the association between the variables are affected by gender. You might also check whether the differences between the effect sizes of the treatment are influenced by the type of control group. Finally, you could revise whether the differences between the effect sizes on the instrument accuracy are moderated by the type of informant (e.g., parent, teacher, child). Also, you may use your quality assessment data as a moderator (e.g., has the quality of your studies something to do with the heterogeneity of the effect sizes?). Moderator variables should be pre-defined, that is, you may want to select what relevant ‘moderators’ you should analyse based on the previous literature.
The main software tools are R studio, RevMan (by Cochrane Reviews), Stata Macros, and Comprehensive Meta-Analysis (CMA). R studio is as powerful as the other software tools, if not more so. R Studio is highly flexible, with access to multiple packages, allowing you to run almost any kind of analysis you have in mind. Moreover, R is free and has a huge platform of community support (see https://stackoverflow.com/).
- Reporting.
The way to report a systematic review and meta-analysis is structured. There are several tools and plenty guidance for doing so. We suggest checking for the relevant tools you may need to use according to type of review questions you have. Cochrane Library and PRISMA group guidance is widely used. They have a very comprehensive list of resources that can help with writing your report. Also, you may want to check the reporting requirements of the journal you want to submit your review to.
Thank you very much for reading our post. If you want to find out more resources for conducting your systematic review and meta-analysis, please click here or drop us an email!
