Abstract: Consistent Algorithm Monitoring Performance at Distributed Sites with Disparate Data Sources (7th Annual Public Health Information Network (PHIN) Conference)

Tuesday, September 1, 2009: 10:50 AM

Hanover A/B

Jian Xing, PhD, MS , National Center for Public Health Informatics, Centers for Disease Control and Prevention, Atlanta, GA

Howard S. Burkom, PhD , Johns Hopkins University Applied Physics Laboratory, Laurel, MD

Jerome I. Tokars, MD, MPH , National Center for Public Health Informatics, Centers for Disease Control and Prevention, Atlanta, GA

OBJECTIVE:

A federated monitoring system requires sharing of analytic services such as alerting algorithms. As Grid technology makes sharing convenient and safe, the analytic community must ensure that algorithms work reliably. Our study objective is to find dynamic threshold adjustments to satisfy sensitivity and specificity requirements.

BACKGROUND:

The BioSense Strategic Plan includes monitoring at local, state, and regional levels. BioSense alerting methods include a modified C2 algorithm and regression. Using the same alerting threshold across monitoring regions may lead to poor sensitivity. We present a selective threshold adjustment method.

METHOD:

We studied performance of 4 control-chart-based methods and 5 regression models for time series of syndromic hospital visits aggregated for 20 Metropolitan Reporting Areas (MRAs) during flu (1/1/2008-3/31/2008) and non-flu seasons (7/1/2008-9/30/2008). Mean MRA syndromic counts varied from 69‑1,143 in flu season and from 33-624 in non-flu season. Detection sensitivity was estimated by applying the algorithms to the original series with simulated signals injected into MRA-day. We devised a threshold adjustment method, using a function ZDT of an area-wide threshold and the local mean Z-score, to enforce a minimum sensitivity for each MRA to signals of fixed size while keeping a region-wide false alarm rate.

RESULTS:

For the regression method applied to Respiratory and GI syndrome data, thresholds at only 7 and 5 MRAs needed adjustment, respectively, for a minimum sensitivity of 36%. For a few MRAs, the sensitivity increased by over 20% with only a small overall increase in false alarm rates. For the control-chart-based method, 13 and 15 MRAs needed adjustment, giving preference to the regression model for these syndrome groups at the MRA level.

DISCUSSION:

The ZDT method can be adjusted for other combinations of group and individual constraints and other signal types. It is intended to enable robust monitoring for distributed computing analytical services.

21065 Consistent Algorithm Monitoring Performance at Distributed Sites with Disparate Data Sources