Validation of a New Instrument: A Possible Assessment Tool to Evaluate Programs Designed to Change Health Behaviors

Theoretical Background and research questions/hypothesis:   This presentation overviews an assessment model recently used to validate the impact of suicide prevention (gatekeeper) training programs and holds promise for assessing the effectiveness of public health initiatives designed to drive changes in health and behavioral health behaviors.   The model is drawn from Kirkpatrick's evaluation paradigm and three major theories of motivation.  Kirkpatrick identifies four levels of assessment: reaction or satisfaction levels, learning or the impact on attitudes, knowledge and/or skills, behavior or the actual change in behaviors and results or overall impact on health care.  Assessment items developed focused on attitudinal measures drawn from motivation theories that relate to goal driven behaviors and prediction of future outcomes.  Specifically, the belief variable of preparedness from Social Cognitive Theory for it influences one's confidence to take action and complete a task. A behavioral intention variable which is a function of beliefs about the likelihood of performing a behavior, was drawn from  Reasoned Action Theory. Lastly, based on Bandura’s integrative framework of personal efficacy, a self-efficacy variable that examines confidence in one’s abilities which predicts behavioral control. We hypothesized that behavioral intent measures of preparedness, likelihood and self-efficacy can increase our ability to assess the impact of programs designed to bring about behavioral change.  

Methods:   To test the model an 11-item Gatekeeper Behavior Scale (GBS) was developed to measure likelihood, preparedness and self-efficacy to aid people in psychological distress, including those at risk for suicide, and assist them in finding appropriate mental health services.  To validate the scale, 8,931 users completed one of five Kognito gatekeeper training simulations for educators or students between 2010 to 2013.  Pre-training, post-training, and follow-up GBS surveys were administered.   The GBS’s construct and content validity was assessed via confirmatory factor analysis (CFA). Criterion-validity was examined via correlations with behavioral measures, and convergent validity was assessed via correlations with similar but distinct constructs.  

Results:   The three factor CFA model based on the subscales of preparedness, likelihood and self-efficacy fit the data best. Factor loadings showed all items correlated highly with theoretical constructs from which the items were derived (r≥0.84, p<.001).  The full 11-item GBS was found to have high internal consistency (α=.93).  Criterion-related validity for likelihood to discuss concerns (measured in the post-training survey) significantly related to actually approaching people that they believe are in psychological distress (r=.219, p<.001; measured in the follow-up survey). Likelihood to refer someone to psychological services significantly correlated with the actual number of people referred (r=.235, p< .001).  Convergent validity was established via a robust correlation between self-efficacy in motivating someone to seek help and general self- efficacy (r=.519, p<.001). 

Conclusions:   The GBS appears to be a valid tool in measuring the impact of online gatekeeper training simulations and holds promise for assessing the efficacy of other delivery methods. 

Implications for research and/or practice:   This assessment model which is based on three major theories of motivation holds great promise as a tool to measure the impact of public health initiatives designed to change health and behavioral health behaviors.