Immunization Data: Evidence for Action (IDEA) - A Realist Review of What Works to Improve Data Use for Immunization

"The use of high-quality data is a cornerstone of well-functioning immunization programs."

Calls to improve the quality and use of data feature prominently in several national plans of action and in global strategies like the Global Vaccine Action Plan (GVAP). Sound and reliable information is thought to enable more effective health education and training, more targeted care, more efficient policy development and implementation, and better resource allocation. PATH and the Pan American Health Organization (PAHO) designed this report - with supporting campaign website - to provide the immunisation community with strategies for improving the quality and use of immunisation data. The review, funded by the Bill & Melinda Gates Foundation, draws on nearly 550 documents, including published literature, working papers, project evaluations, and reports.

The realist review approach, unlike a traditional systematic review, does not exclude evidence based on study design or quality. In order to guide the review, PATH and PAHO developed a theory of change (TOC) that framed their hypothesis of the theorised mechanisms and contextual factors that work together to help decision-makers translate immunisation data into information, and ultimately action.

Findings of the review are organised into the following intervention categories, which correspond to the chapters of the report:

1. Electronic immunisation registries (EIRs)
2. Logistics management information systems (LMISs)
3. Health management information systems (HMISs)
4. Decision support systems
5. Data quality assessments
6. Data review meetings
7. Peer learning networks
8. Supportive supervision, mentorship, and on-the-job training
9. Training
10. Multicomponent interventions

As an example, to focus on #7 above, the report shows how peer learning networks can connect health workers and allow them to interact, share information and feedback, and review and discuss data. By establishing connections across multiple levels of the health system, peer learning networks can help bridge the gap between data producers and data users. Increasingly, peer learning may take place on social network platforms like WhatsApp. Although the review found no evidence that peer learning networks led to data use outcomes in the TOC, PATH and PAHO found moderate-certainty evidence that peer learning networks led to improved vaccine availability. They also found evidence of peer learning networks leading to improvements in knowledge, motivation, and skills related to data use. Peer learning networks were found to be most effective at problem-solving when they bring together individuals from across departments and levels of the health system and when they adopt structured approaches for continuous quality improvement.

Some overarching findings of the review include:

• Multicomponent interventions were the most prevalent and often more effective. Successful intervention packages included strategies that addressed:
  • skill sets and capacity of data users;
  • gaps in feedback mechanisms;
  • data use within existing systems, workflows, and workloads;
  • user-centred design principles;
  • interaction between data producers and data users, and structured problem-solving;
  • data use culture and motivation to use data; and
  • long-term commitment of financial and human resources.
• Interventions that took a health systems approach to institutionalising data use were more likely to be successful and sustainable. This occurred by routinely conducting data review meetings at all levels, making national guidelines and protocols on data use available to frontline staff, creating dedicated staff positions at all levels of the health system, and incorporating training in data use in national in-service and pre-service training curricula.
• There was limited or mixed evidence on the effectiveness of HMISs, including EIRs, on data use, but they remain promising interventions for improving data use when accompanied by complementary activities.
• Moderate- to high-certainty evidence exists to suggest that computerised LMISs have made higher-quality data more available to decision-makers to improve management of supply chains.
• There is a dynamic, cyclical relationship between data quality and data use.

The IDEA review identifies 5 strategies to improve data use and outlines how funders, policymakers, and programme implementers can incorporate these best practices to improve the efficacy of state, regional, and national immunisation programmes. These include:

• Strategies that address barriers to immunisation should be interconnected and mutually reinforcing.
• The more that data is used, the more likely its quality will improve.
• Data should be an integral part of health decision-making.
• Digital information systems should be used to provide high-quality data to decision makers in real time.
• The use of digital systems should be phased to ensure that the proper infrastructure is in place to manage them.

In addition, recommendations for improved monitoring and evaluation (M&E) are provided. For instance, PATH and PAHO propose a set of indicators for monitoring immunisation data use interventions, with a focus on the intermediate outcome and data use action levels of the TOC.

The review was guided by the IDEA steering committee, which was composed of 10 global and regional senior leaders from PAHO, the World Health Organization (WHO) headquarters, Bill & Melinda Gates Foundation, PATH, US Centers for Disease Control and Prevention (CDC), United Nations Children's Fund (UNICEF), Gavi, and the Swiss Tropical and Public Health Institute (Swiss TPH), as well as country representatives from both the BID Learning Network (BLN) and Improving Data Quality for Immunizations (IDQi) project core countries.

IDEA partners launched the aligned #FindYourFinding interactive campaign, inviting practitioners across the immunisation and global health community to explore the IDEA review and support them in acting on its findings. In additon to testimonials from partners and supporters, the campaign website is designed to bring IDEA strategies to life through:

• An evidence gap map: For each primary intervention type identified in the review, the gap map visualises all the pieces of evidence and promising strategies according to the intermediate data use outcomes, data use actions, and impact indicators to which they pertain.
• Find your finding: Included are separate pages featuring top findings for: policymakers; implementers; funders; and by intevention.
• A partner toolkit: Handouts, social media content, event outreach materials, and communication materials are designed to facilitate offline and online action on the part of decision-makers around the world in using the findings to improve immunisation outcomes.