FITS – Forecasting for Immunization Test Suite

Background

The need for accurate and timely immunization evaluation history and forecasting recommendations is paramount to the success of Immunization Information Systems (IIS). The CDC has developed a set of resources for immunization clinical decision support (CDS) systems based on the Advisory Committee for Immunization Practices (ACIP). It is critical that systems implement the ACIP recommendations as specified. By their inherent nature, however, ACIP recommendations can be very complex; the availability of directed test descriptions and test cases can help in realizing a clear and consistent interpretation of the rules. Additionally, test tools are needed to test CDS engines for immunization. NIST has built a test tool (FITS – Forecasting for Immunization Test Suite, https://fits.nist.gov) that allows immunization experts to create and manage test cases and allows for those test cases to be executed on CDS engines. FITS provides advanced features used for importing, creation, management, exporting, and sharing of test cases and the reporting of test results (See Figure 1). The CDC is using FITS to import existing test cases and creating new test cases. The American Immunization Registry Association (AIRA) has operationalized the tool and is using in their assessment of IIS implementations

Credit: Robert Snelick

Fig 1. Diagram of FITS architecture

Problem: Producing Accurate Up to Date Forecasts

Immunization forecasting software provides clinicians (physicians, nurses, etc.) with patient-specific immunization recommendations during visits. This software, known as a "forecaster," analyzes patient medical history and profiles to generate a forecast, which includes a list of recommended immunizations and their corresponding dates.

Vaccination schedules, written in complex clinical language, require software developers to translate them into computer code for algorithmic analysis. This translation process is prone to errors due to potential misinterpretations by developers and even disagreements among medical experts. Frequent updates to these schedules further complicate matters, leading to outdated software and incorrect forecasts.

Solution: Improved Forecasting Through FITS Testing

NIST has developed the Forecasting for Immunization Test Suite (FITS), a software tool for evaluating immunization forecasting systems. FITS assesses performance via direct API connection, user-selected test case execution, and analysis of forecast results. Forecast software developers use FITS for testing their software prior to release, and public health agencies like the CDC uses FITS to create and maintain test cases. FITS is also used by American Immunization Registry Association (AIRA) to conduct quarterly forecasting performance assessments across all major Immunization Information Systems (IIS) in the US.

The FITS Success Story

Since AIRA began testing IIS forecasters with FITS, the number of forecasters which align with test cases created by CDC (collectively) rose from 47% in 2019 to 73% in 2024. [1]

Credit: American Immunization Registry Association

Fig 2. Results of Quarterly testing of IIS forecasters using FITS

Why are IISs and Accurate Forecasting Important?

According to the US Centers for Disease Control and Prevention (CDC): Immunization clinical decision support — also referred to as evaluation and forecasting [1], or simply forecasting — is an automated process that determines the recommended immunizations needed for a patient and delivers these recommendations to the healthcare provider. Immunization forecasting is a software-based service built into nearly all Immunization Information Systems (IIS). The software implementation itself is called a forecaster because it outputs a forecast [2] which is a list of recommended immunizations and dates, based on a patient medical history and profile.

The information contained within IIS forms the basis for many important decisions regarding immunizations at both the local and national levels, and is utilized by a range of stakeholders, from individual healthcare providers, hospitals and clinics, to schools, insurers, and public health agencies like the CDC.

Determining which vaccinations a patient should have — along with the correct dosage and formulation, and when and how to optimally administer them — is an increasingly complex task, even for experts.

This has led to the development of automated forecasters — a variety of clinical decision support system (CDS) software seeing increased usage within the modern medical profession.

The guidance given by the CDC and others, as to which vaccines dosages should be given to whom and at what time, are referred to as vaccine schedules. These vaccination schedules, and the associated guidance (e.g. “contraindications”) published along with them, are lengthy, and undergo constant updates as new vaccines and vaccine formulations are produced, and as new medical knowledge is published. Forecasting software automates the complex analysis of vaccination records using all available schedules and expert guidance which speeds up the process and leads to more accurate and consistent results.

1. The term “evaluation” refers to determinations of which vaccinations in a patient’s medical record are considered valid or invalidly administered, according to the latest criteria published by the CDC, et al.
2. In addition to a “forecast”, an evaluation of the patient’s vaccination history is typically included

Electronic Health Record (EHR) systems used by hospitals, clinics, schools, and others interact with local IIS to upload and query individual’s vaccination records. EHR systems typically use either the IIS or their own built-in software to provide immunization forecasts. These immunization forecasts are then used by doctors, nurses, school administrators, and others to determine how “up to date” an individual is with respect to the CDC’s recommended vaccination schedule (based on things like their age, gender, and medical history).

By automating these tasks, clinicians can better serve their patients, by reducing the potential for mistakes, and maximizing the “opportunities” of a clinical visit. For example, patients who have not maintained regular doctor visits may benefit from “catching up” on their vaccines, which can be done in accordance with CDC guidelines. (This is unfortunately the situation for many young children during the recent COVID-19 pandemic, whose parents might avoid regular pediatric visits and checkups due to fear of infection.)

Schools, for example, also use immunization forecasting systems to more quickly and accurately assess which students are up to date with vaccine eligibility requirements and notify them accordingly. Insurance companies likewise use this technology to remind their customers about needed vaccinations. Any setting where vaccinations are administered can potentially benefit from this technology. In addition to its use for individuals, Immunization forecasting is also used for analysis of large populations. For example, immunization forecasting is often used by States to measure vaccination rates— also referred to as vaccination coverage (which are statistics reported regularly to the CDC) — on whole, or cross sections of the State's population. For example, as vaccines become available for

Analysis of large quantities of data to determine the rates of vaccination cannot be performed as accurately without use of an immunization forecasting and evaluation systems. Ultimately, successful public health efforts to track immunization rates and identify deficiencies (“pockets of need”) depends on the ability of the IIS to maintain accurate and timely information and utilize accurate immunization evaluation and forecasting services.

Additional Information:

See NIST Presentation from 2019.

See NIST Presentation at 2017 AIRA National Meeting.

Additional Information from AIRA:

https://repository.immregistries.org/resource/iis-assessment-aggregate-status-report-clinical-decision-support/