An Assessment of the Quality of Vaccination Data Produced through Smart Paper Technology in The Gambia

"High-quality health data is paramount to health systems planning, implementing new programs, and evaluating their effectiveness..."

The World Health Organization (WHO) deems health information systems as one of the key building blocks of health systems, their main functions being data generation, compilation, analysis and synthesis, and communication and use. The goal of this study is to characterise the quality of routine immunisation data generated using MyChild Solution, an electronic immunisation register (EIR), compared to data obtained through the conventional health management information system (HMIS) used in The Gambia.

In the conventional system in place before MyChild Solution, health workers would register infants in paper-based immunisation registers. Completing monthly return forms required a lengthy process involving manually extracting vaccine doses consumed using tally books, stock balances from the previous month, and vaccine doses received during the current month.

Introduced in The Gambia by Shifo Foundation and its partners in 2017, MyChild Solution is an HMIS for immunisation based on smart paper technology (SPT), optimised for use in low-resource settings. (Click here to learn more about MyChild.) SPT relies on customised paper forms that are designed to be automatically digitised after being scanned using optical character recognition (OCR) technology, machine learning algorithms, and built-in data validation and recognition rules. This allows sustaining a digital registry while maintaining the reliability and low cost of paper at the point of care, thus ensuring that patient data collection will not be affected by power outages or connectivity issues during immunisation sessions.

MyChild Solution produces several outputs. A set of 16 key performance indicators (KPIs), including data quality, coverage of selected antigens, missed opportunities for vaccination, dropout and wastage rates, and vaccination timeliness, are sent to health workers through SMS (text message) at the end of each month for them to update their paper KPI monitoring graphs. This is designed to empower health workers to identify actions to improve their own performance based on their KPI trends. The system also generates a defaulters list for each health facility at the end of every month to facilitate defaulter tracing. Vaccine needs for the next month are also automatically estimated by the solution and used to inform logistics management.

The researchers used the WHO's Data Quality Review (DQR) Toolkit to evaluate MyChild Solution's data quality in the 19 health facilities across across Western Region 1 (WR1) and Western Region 2 (WR2). In most cases, data quality indicators under the DQR Toolkit were evaluated using data over a 12-month period (March 2018 - February 2019) and included all the health facilities using MyChild Solution in the 2 regions. For the conventional HMIS, the most recent 12 months before the introduction of MyChild Solution were used.

Results are shared along the 4 dimensions of the WHO DRQ Toolkit, which proposes several data quality indicators (metrics) that are grouped as follows:

1. Completeness and timeliness of data: Both MyChild Solution and the conventional HMIS produced 100% complete and timely data in their reference years.
2. Internal consistency of data: Both systems had no moderate or extreme outliers and showed the expected Pentavalent vaccine 1 to Pentavalent 3 dropout direction.
3. External consistency of data, i.e. its agreement with other sources of data such as surveys: All the Penta 3 coverages are within 33 percentage points of each other for both systems - within the acceptable limits set by the DQR Toolkit.
4. External comparisons of population data, i.e. a review of denominator data used to calculate rates for performance indicators: The proportion of verification factors that are not acceptable was higher in the conventional HMIS. Also, MyChild Solution was found to near perfectly (99.98%) digitise scanned documents.

Based on these data quality metrics and additional metrics of interest (e.g., the incidence of recording errors, the incidence of incomplete indicator level data, and implausible dates), the study found that MyChild Solution generates more internally consistent data across data sources than the conventional HMIS and has the capability to eliminate inconsistencies between monthly returns and District Health Information Software 2 (DHIS-2) values because of its ability to be integrated with DHIS-2. Most of the inconsistencies in the conventional HMIS are due to not updating immunisation registers, followed by a missing data source.

Furthermore, all immunisation sessions were correctly captured in the system. According to the researchers, this feature of MyChild Solution gives it an edge over the conventional HMIS because it allows remotely comparing sessions captured by the system against those scheduled for each health facility using the solution's interactive dashboard. This enables quasi-real-time (as forms get scanned) monitoring and follow-up with health facilities from the regional and national levels.

The researchers conclude by recommending the following going forward: "it is important to maintain robust data quality assurance processes including continuously providing feedback to health workers on recording-related issues, monitoring data verification processes, and continuously assessing system-wide data quality metrics to minimise the risk of potential data quality issues."