A Realist Systematic Review of Evidence from Low- and Middle-Income Countries of Interventions to Improve Immunization Data Use

"Few immunization data use interventions have been rigorously studied or evaluated. The review highlights gaps in the evidence base, which future research and better measures for assessing data use should attempt to address."

The use of routine immunisation data by healthcare professionals (HCPs) in low- and middle-income countries (LMICs) is recognised as an underutilised resource. Applying a realist review methodology, this study synthesised evidence of effective interventions for improving data use in decision-making.

As outlined here, realist review is a theory-driven literature review that aims to test and refine the underlying assumptions for how an intervention is supposed to work and under what conditions. After developing a review protocol with input from a technical steering committee, the researchers articulated a theory of change (TOC) (see above) for how they expected data use interventions to influence data use. Two rounds of searches conducted in 2018 led to 49 articles from peer-reviewed journals and 53 from grey literature that met inclusion criteria. The researchers then plotted the results of the 66 articles that reported on immunisation data use interventions (vs. those in other health sectors) by primary intervention type and evidence of intermediate outcomes and data use actions in an evidence gap map (see Figure 3 in the paper and/or click here). The gap map visualises all pieces of evidence, including the strength and directionality of the evidence, as well as promising strategies. (A complete list of the immunisation data use articles and their quality appraisal scores is included in Appendix B, and a detailed synthesis of evidence by intervention category and data use outcomes is included in Appendix C.)

Selected findings include:

• The published literature provides insights into the barriers related to data use (e.g., insufficient skills in data use core competencies among health workers, lack of trust in data due to poor quality, and inadequate availability because of fragmented data across multiple sources), but there are few rigorous evaluations of interventions designed to address these barriers.
• The review unearthed evidence of interventions impacting the intermediate outcomes in the TOC, such as data quality, availability, analysis, synthesis, interpretation, and review, but there was less evidence on what works to support data-informed decision-making. This issue can be explained by the lack of consensus around how to define and measure data use.
• Multicomponent interventions with mutually reinforcing strategies to address barriers at various stages of data use (e.g., electronic immunisation registries (EIRs) combined with training and/or supportive supervision) were found to be most effective.
• Interventions were more likely to succeed and be sustainable if they institutionalised data use through dedicated staff positions for data management, routine data review meetings, national training curricula, and guidelines on data use for frontline staff.
• While the transition from paper to digital systems has made higher-quality data more available to decision-makers in real time, it has not automatically translated into greater data use. It may help to pair digital systems with activities that reinforce data use.
• Data quality is an important barrier and necessary precursor to data use, but there is limited evidence that interventions focused singularly on data quality improve data use. Health workers may lack the necessary skills to analyse and translate data into information that is useful for making decisions on programme implementation. However, there is more compelling evidence to suggest that data use interventions can lead to improvements in data quality. As health workers began using data, they were able to identify inconsistencies and take corrective action. Data use also generated demand for higher-quality data, and as quality improved, users increasingly trusted the data, which in turn reinforced use of the data.
• There were particular gaps in the evidence on what works to improve data use at the facility level. Interventions at this level have focused more on improving data collection practices and data quality, and less on data use.

Among the suggestions for action:

• The operational barriers and administrative challenges faced by digital information system interventions point to the need for a phased approach, ensuring that data use infrastructure, human resources capacity, and skills-building are in place before a full digital transition.
• There is a need to develop better measures for assessing data use in decision-making. Evaluation designs should account for complex interventions. The researchers do not necessarily recommend investment only in experimental design studies to establish effectiveness; rather, they found that the most useful and richest evidence came from mixed-methods studies and evaluations that described why and how the intervention worked, for whom, and where it worked.

Among the suggestions for future research is to study whether more emphasis on building data use skills during frontline health worker pre- and in-service training and continuing education (e.g., statistics, data interpretation, and data management) - as well as cultivating a culture of data use - would have an impact on strengthening data quality and use.

In conclusion, the researchers suggest that: "The evidence on effective interventions and promising strategies detailed in this review will help program implementers, policymakers, and funders choose approaches with the highest potential for improving vaccine coverage and equity....[T]hese findings [may] also be of interest to researchers and evaluators to prioritize gaps in the existing knowledge."