Development and Feasibility Testing of an Artificially Intelligent Chatbot to Answer Immunization-related Queries of Caregivers in Pakistan: A Mixed-methods Study

"...findings make the case for LMICs [low- and middle-income countries] to consider utilizing chatbots to optimize health education and information provision to caregivers to improve immunization uptake."

Amongst multiple demand- and supply-side barriers that impede immunisation uptake, lack of reliable information and inadequate counselling from healthcare providers cause caregivers' simple questions to remain unresolved, causing them to unnecessarily delay or forgo vaccines. Current information dissemination strategies in Pakistan, such as mass media campaigns, paper-based immunisation cards, and short messaging service (SMS/text message) reminders, rely on static and unidirectional platforms instead of providing a two-way, conversational experience to caregivers. This paper describes the development of an artificial intelligence (AI) chatbot to respond to caregivers' immunisation-related queries in Pakistan and investigates its feasibility and acceptability in a low-resource, low-literacy setting.

The study was conducted in Korangi and Landhi Towns of Karachi, Pakistan. The districts within which the two towns are located had a full immunisation coverage of 51.8% and 56.3%, compared to a national full immunisation coverage of 66%. Mobile penetration in Pakistan is high, with almost 78% of the total population having mobile connections and over 60 million people having access to the internet.

Bablibot (Babybot), a local-language, text-based immunisation chatbot, can operate as a standalone system but can also be integrated with electronic immunisation registries (EIRs). Unlike conventional chatbots that respond through rule-based actions based on keywords, Bablibot tries to replicate human conversations, and it deploys machine learning (ML) algorithms to continuously self-improve. That is, the ML model ensures that as the dataset grows, Bablibot becomes increasingly sophisticated and autonomous, improving in terms of accuracy (ability to respond correctly) and breadth (ability to handle a broad range of queries) over time. The human in the loop (HiTL) feature feature allows the bot to default to the human responder when required. Since March 2020, Bablibot was available via SMS only; in November 2020, a WhatsApp feature was added in response to community feedback. Bablibot does not necessitate internet or smartphone access and poses no additional cost to users other than the regular charges of their existing SMS or internet packages.

Bablibot is linked to a web-based portal that displays all conversations the bot engaged in and highlights ongoing conversations that require human intervention. Conversation logs can also be downloaded for detailed analysis.

Leveraging its interoperability with EIRs, the team integrated Bablibot with Sindh Government's provincial EIR after completing development and testing of the bot's core features. However, Bablibot's ability to engage caregivers is not dependent on EIR data, and it continues to provide guidance and counselling on immunisation-related queries to caregivers who are not enrolled in the EIR or who are messaging from a number not linked to their record in the EIR.

For the sequential mixed-methods study, the study field officers visited 12 Expanded Program on Immunization (EPI) centres in Landhi and Korangi towns on a rotational basis between March 9 2020 and to August 20 2020 to enroll caregivers in the study. Additional caregivers were reached through targeted text message communication. The researchers assessed Bablibot's feasibility and acceptability by tracking user engagement and technological metrics and through thematic analysis of in-depth interviews with 20 caregivers.

In total, 2,202 caregivers were enrolled in the study, of which 677 (30.7%) interacted with Bablibot (users). Bablibot responded to 1,877 messages through 874 conversations. Conversation topics included vaccination due dates (32.4%; 283/874), side effect management (15.7%; 137/874), and the appropriateness of delaying vaccination due to child's illness or COVID-19 lockdown (16.8%; 147/874).

Over 90% (277/307) of responses to text-based exit surveys indicated satisfaction with Bablibot. Qualitative analysis showed that caregivers appreciated Bablibot's usefulness. Many caregivers approached Bablibot on repeated occasions. The researchers suggest that the high satisfaction rate was in part due to the presence of human monitoring (HiTL feature), which ensured that users received the information they were seeking, even in cases where the bot was unable to provide an accurate response.

Although unsolicited, two out of ten users reported having informed others in their social circle about Bablibot as well. Non-users, although did not feel the need for using Bablibot during the study period, still believed in the potential of Bablibot to have an impact on immunisation uptake by providing remote and immediate access to immunisation information, especially for mothers who otherwise have limited access to information.

Users provided feedback for further improvement of the system. For example, most of the caregivers suggested that Bablibot should be accessible in multiple languages so more people can use it. Caregivers would also like the answers to reach them quickly, with reliable, complete, and to-the-point information

Thus, the researchers found Bablibot to be "a feasible solution for providing guidance to caregivers by acting as a bi-directional, 24/7 chat service, easily available to them at minimal cost....[R]esults demonstrate that caregivers in Pakistan require ongoing guidance throughout their child's immunization journey for simple yet crucial information such as vaccination dates or side effects. Chatbots, such as Bablibot, have the potential to address these concerns instantaneously, which if left unaddressed, trigger vaccine-hesitant behavior....Moreover, by being a remote and confidential information source, chatbots can address gender gaps in information access in areas where women lack the ability to seek information and health services due to local gender norms."

Next steps will be to leverage incoming Bablibot data to train the model further and explore avenues for additional similar training data sets, considering the upsurge in local language chatbots that have emerged since this study.

In conclusion, this study shows "the potential of incorporating local language AI-based chatbots into the clinical immunization workflow to enhance patient engagement and improve patient education regarding immunization. Integration of such Chatbots with electronic health records could revolutionize the delivery of accurate, personalized information while dispelling misconceptions and misinformation regarding immunizations."