ONC Patient Matching Challenge: The Search for a Solution

Last month, the Office of the National Coordinator for Health Information Technology (ONC) announced the winners of its “Patient Matching Algorithm Challenge.” Designed to “bring about greater transparency and data on the performance of existing patient matching algorithms,” and to “spur the adoption of performance metrics for patient data matching algorithm vendors,” the five-month competition saw 7,000 submissions from more than 140 competing teams.

In the end, the competitors with the best “F-Score” were:

1. Vynca ($25,000)

2. PICSURE ($20,000)

3. Information Softworks ($15,000)

The fact that each of the winning organizations employed different methodologies (more information below) is a great demonstration of why such a competition was needed in the first place.

The Catalyst

Section L of Connecting Health and Care for the Nation: A Shared Nationwide Interoperability Roadmap (aka “the Roadmap”), published in 2015 by the ONC, highlights the significant challenges around accurate individual data matching—and its importance to interoperability and the nation’s health IT infrastructure. While providers must be able to share a patient’s health information, they must also be aware that matching mistakes can have broad impacts that include adverse events, compromised safety and privacy, and over-burdened clinical resources due to longer stays, duplicate testing, treatment delays, and other implications.

Costs are also impacted. Administratively, the primarily manual process of correcting mismatches is estimated to cost $60 per record. Clinically, a survey from The Ponemon Institute puts costs associated with patient impacts caused by mismatched records at upwards of $17 million a year.

These are the realities that prompted the ONC to develop the Patient Matching Algorithm Challenge, the first step of which was to tap Just Associates once again to put its data integrity expertise to work.

The Contest

The ONC called upon Just Associates to create a synthetically-generated test data set of about 1 million records, based on real-world data and discrepancies that are typically found in the master patient index (MPI). This data set also incorporated the master answer key that would hold duplicate pairs that an algorithm should match/identify.

After unlocking and downloading the test data set, competitors ran their algorithms and submitted results to the scoring server on the Patient Matching Challenge website. The submissions received performance scores and appeared on a Challenge leaderboard that was updated throughout the competition.  Each set of results received objective evaluation metrics (F-scores) that could be used to guide system improvements. 

After up to 100 re-runs, participant answers were evaluated and scored against the master answer key, which was also provided by Just Associates. Winners were ultimately determined based on their F-scores, a measure of accuracy that factors both precision and recall.

As noted previously, each winning organization took a different approach to the Challenge:

• PICSURE used an algorithm based on the Fellegi-Sunter (1969) method for probabilistic record matching and performed a significant amount of manual review.

• Vynca used a stacked model that combined the predictions of eight different models, enabling them to manually review less than .01 percent of the records.

• Information Softworks also used a Fellegi-Sunter-based enterprise master patient index system with some additional tuning, but reported needing extremely limited manual review.

In addition to the overall winners (Vynca, PICSURE, Information Softworks), ONC selected three “category” winners, each of whom received $5,000:

• Best First Run: Information Softworks

• Best Recall: PICSURE

• Best Precision: PICSURE

Just Associates extends its warmest congratulations to each of the Challenge winners!

A Lasting, Positive Impact

Those who participated in the Challenge took the proactive step to identify and address gaps and blind spots in their current system and make the changes necessary to create an algorithm that would more accurately identify potential duplicate patient records as well as patients at the point of care.

“Many experts across the healthcare system have long identified the ability to match patients efficiently, accurately, and to scale as a critical interoperability need for the nation’s growing health IT infrastructure,” said Don Rucker, M.D., national coordinator for health information technology. “This challenge was an important step towards better understanding the current landscape.”

---