Outcomes Mapping Study for Childhood Vaccination Communication: Too Few Concepts Were Measured in Too Many Ways

"Communication around childhood vaccination is a developing area with many innovative and often complex interventions that are not yet rigorously evaluated or their mechanisms of effect well understood..."

Responding to this challenge of measuring the effects of the diversity of communication interventions designed to inform, educate, remind, support, and/or change behaviour around vaccination, this paper presents a map that is designed to catalog the outcomes that have been measured in trials of childhood vaccination communication interventions. The premise is that "[e]ffective communication with parents and communities is critical to addressing vaccine hesitancy and improving global childhood vaccination uptake." It emerges from research being carried out as part of Communicate to Vaccinate (COMMVAC 2) (see Related Summaries, below), which is an international project addressing evidence-based communication - e.g., via the development and use of core outcome sets (COSs) - about childhood vaccination in low- and middle-income countries. The paper describes the methods for creating the Trial Outcomes Map, reports the results, and reflects on a number of trends in outcome reporting identified during this process.

In November 2013, the researchers searched the Cochrane Central Register of Controlled Trials (CENTRAL) for trials of childhood vaccination communication interventions, identifying 112 relevant trials containing 209 outcomes. (Included were trials written in English (randomised, quasi randomised, or cluster controlled) that evaluated a communication intervention, focused primarily on a vaccination population of children up to 6 years of age, addressed routine childhood vaccines only, and reported communication which was addressed to caregivers or community members (communication interventions directed to health professionals were not included unless they specifically impacted the encounter between the provider and parents. This focus reflects the scope of the COMMVAC project.) Thematic analysis revealed 3 overarching outcome categories: (i) consumer-related (i.e., outcomes relevant to the thoughts, feelings, experiences, and decision making of parents or community members), (ii) vaccination-related (i.e., outcomes assessing changes in vaccination status or behaviour), and (iii) health-system-related outcomes. These categories contain 21 outcome types (the concepts being measured - e.g., "knowledge"), measured using 66 different outcome variables (the ways in which different trials define and measure outcomes).

The process at each level of the resulting Trial Outcomes Map (see Table 1, pages 7-8 to view the map) was iterative, and the final map structure and organisation was based on several rounds of discussion within the research team, which includes experts in vaccination and communication research with experience in taxonomy development.

Of the 112 trials identified, 89 (80%) measured at least one vaccination-related outcome. Though consumer outcomes are increasingly measured in more recent trials, "this Map again shows the lack of consensus around which Outcome Types and Variables are and should be assessed in trials. Consumer-related outcomes were measured by 31 trials, or 28% of the total mapped trials. 'Knowledge' and 'attitudes or beliefs' were the two most commonly measured consumer-related Outcome Types....'Intervention evaluation,' which addresses consumers' views, impressions, and assessments of the communication intervention itself, was measured by five trials....The variation in the consumer-related outcomes assessed by trials is evidenced by the fact that 31 trials measured 14 different Outcome Types using 33 different Outcome Variables. Health system-related outcomes were measured by 20% of the trials (22 trials)."

The researchers note that "trial literature defined key Outcome Types using many different Outcome Variables, meaning there is considerable inconsistency at the level of actual measurement....Stakeholder consensus in the form of a COS could help reduce unnecessary outcome variation, help decide which outcomes are more important for trials of communication interventions as compared to other types of vaccination interventions, and ensure that recommended outcomes are based on appropriate evidence. In the next steps of the COS research process, we will build on the Outcomes Map by consulting with parents, health providers, researchers, and policy makers to identify additional outcomes and determine which outcomes are most relevant and important to these stakeholders." COMMVAC researchers are also conducting a qualitative evidence synthesis focusing on parents' and carers' views and experiences of routine early childhood vaccination communication. This synthesis may help clarify which outcomes are most relevant and important to parents and carers and also the pathways between intervention delivery and important end points.

The hope is that this map and process going forward may improve consistency in outcome selection and reporting for trials that examine vaccine-related communication strategies.