Selecting the Most Effective Nudge: Evidence from a Large-Scale Experiment on Immunization

"...the social network can be used in creative and cost-effective ways to amplify the effect of other policies."

There is mounting evidence that, despite efforts to make immunisation more widely available, insufficient parental demand for immunisation has contributed to a stagnation in immunisation coverage. This challenge has motivated experimentation with "nudges", which can be used separately or in combination. Drawing on techniques from statistics and econometrics for model selection and subsequent inference, this paper evaluates a large-scale set of interventions to increase demand for immunisation in Haryana, India.

For several years, the government of Haryana had developed various strategies to make reliable supply of immunisation services available to rural areas, but the uptake of immunisation remained low. Studies have found that part of the low demand reflects deep-seated doubts about the effectiveness of immunisation, its side effects, or the motives of those trying to immunise children. However, another part of the low uptake reflects a combination of indifference, inertia, and procrastination. In surveys the researchers conducted in Haryana, many parents reported being in favour of immunisation and had their children vaccinated with the first dose of a vaccine, but a large fraction of children did not complete the vaccination schedule, which is consistent with high initial motivation but difficulty with following through.

With support from the United States Agency for International Development (USAID) and the Gates Foundation, the government of Haryana entered into a partnership with J-PAL to test out different interventions. The objective was to pick out the best policy for a possible scale-up throughout the state. The researchers therefore experimented with nudges that have been shown to be effective in other contexts. They set about finding the best combination and dosage of those nudges.

The experiment was a cross-randomised design of three main nudges: providing monetary incentives (a small value of mobile phone credit given to caregivers each time they brought their child to get immunised), sending short message service (SMS) reminders (also sent through an automated voice call to reach illiterate parents), and seeding immunisation "ambassadors", who were either selected randomly or through a nomination process. In the latter case, a small number of randomly selected villagers are asked to identify those members of their village who are either particularly trusted or are information hubs for their community, or both. To identify information hubs, the researchers asked these respondents to name the community members best positioned to spread information (e.g., about the existence of immunisation camps) most widely. (See Related Summaries, below.)

For each of these nudges, the experiment involved varying the level and schedule of the incentives, the number of people receiving reminders, and mode of selecting the ambassadors. In total, there were: one control group, four types of incentives interventions, four types of ambassador interventions, and two types of SMS interventions. Since they were fully cross-randomised (in the sample of 915 villages), there were 75 unique policy combinations.

The researchers developed a new statistical technique - what they call a smart pooling and pruning procedure - for finding a best policy from a large set, which also determines which policies are effective and the effect of the best policy. The key outcomes are (i) the number of measles immunisations and (ii) the number of immunisations per dollar spent.

The policy that has the largest impact (information hubs, SMS reminders, incentives that increase with each immunisation) increases the number of immunisations by 44% relative to the status quo. The most cost-effective policy (information hubs, SMS reminders, no incentives) increases the number of immunisations per dollar by 9.1%. The best policy was found to be one where the policymaker uses information hubs and sends SMS reminders to 33% of the community.

A substantive finding from this analysis is that using information hubs magnifies the effect of other interventions. Neither incentives nor SMS reminders are selected on their own, but are selected in combination with information diffusion via information hubs. Conversely, the information hubs are not selected for efficacy on their own, but only when combined with SMS reminders with or without incentives that grow with the number of shots. One possible interpretation of this finding, especially given that the most effective ambassador is the information hub (this is the person best placed to circulate information according to the community) is that the ambassador ensures widespread diffusion about the presence of the incentives (in incentives villages) and is able to explain and de-mystify the content of the personalised reminders (in SMS villages, even without incentives). In either case, the ambassador has something concrete to discuss with the people they talk to (which is not the case in villages without SMS or incentives, perhaps explaining why they have no effect in this case).

Among the study's takeaways: From the perspective of public health policy, standard tools that have previously been championed (e.g., SMS reminders) may not be particularly effective at large scale, such as at a state or national level. This could be due to the fact that complex information circulated to influential people can lead to better understanding and action than complete broadcasting, because people may feel insecure asking questions. But using such instruments in combination, particularly with policy insights from social network analysis, can yield effective, cost-effective policies. The results are consistent with recent literature pointing to the importance of leveraging networks to diffuse information in a variety of contexts and might be something policymakers could consider.