Challenges in Evaluating Syndromic Surveillance Systems

When considering evaluation methods for syndromic surveillance systems, the first question to be addressed is “What is the purpose of this system?” In large measure, the answer to this question determines the system's performance specifications and the attributes to be evaluated.  As first conceived in the last decade of the Twentieth Century and implemented following the attacks of September 11, 2001 and the anthrax events occurring later that Fall, syndromic surveillance systems were developed to enable public health to detect and respond to deliberate biological attacks on a population. For such systems, the attributes of sensitivity, timeliness, and public health utility were paramount.  As time has passed without new biological attacks and users have gained familiarity with systems, syndromic surveillance systems' uses have expanded to include injury and illness surveillance following natural disasters, monitoring the health status of defined populations, and surveillance for illness during significant meteorological events such as prolonged periods of excessively cold or hot temperature, and following chemical and other environmental exposures, food contamination related illness, case finding for outbreaks reported and under investigation, and monitoring trends in seasonal disease such as influenza.   Many of these uses depend upon the surveillance system to provide an assurance that adverse health events are not occurring in numbers that would be of public health significance.

            Previously reported system evaluation methods are reviewed.  The challenges to the comprehensive evaluation of syndromic surveillance systems are discussed and several approaches to evaluation of sensitivity, specificity, and predictive value positive are proposed.