Using a Mixed-Methods, User-Centered Evaluation to Improve Usability for the National Center for HIV/AIDS, Viral Hepatitis, STD, and TB (NCHHSTP) Atlasplus

Background:  Data visualization has become crucial for the dissemination of disease data to target audiences. To maximize effectiveness and use, developers of digital data visualization tools should identify key users and seek to understand their data needs. Evaluation of key users through quantitative and qualitative methods can identify important information regarding perceptions, usability, and satisfaction surrounding data visualization tools and contribute to improvement the tool.

Program background: AtlasPlus is NCHHSTP’s data visualization tool that allows users to create maps, charts, and detailed reports and analyze trends and patterns in HIV, viral hepatitis, STD, and/or TB surveillance data. NCHHSTP conducted a mixed-methods, user-centered evaluation to develop recommendations for improvement.

Evaluation Methods and Results:  Surveys and interviews were employed to identify user characteristics, perceptions, behaviors, and usability concerns for the NCHHSTP AtlasPlus. Surveys were distributed to internal users—NCHHSTP staff (N=122)—and external users—CBOs, state and local health departments, and academic researchers (N=63)—of AtlasPlus. In-depth interviews were conducted with the internal (N=1) and external (N=17) groups. Qualitative data were pulled from the comments provided by internal audiences in surveys. Descriptive data and thematic analysis were used to analyze the quantitative and qualitative data, respectively.                 Over half of internal users of AtlasPlus identified as epidemiologists (N=39), program-related staff (e.g. public health analysts) (N=31), or behavioral scientists (N=26). Internal users reported being satisfied (61%) with AtlasPlus, and reported it as easy to use (40%). Internal users most frequently engaged with the change over time feature (65%)—a component that shows an animated update of disease trends over time—followed by other maps (53%) and data tables (44%).                 Qualitative themes around use of AtlasPlus for CDC users included generating data for internal reports, completing grantee-related tasks, and responding to outside requests for disease data. Themes surrounding usability included the desire for a more user-friendly, intuitive experience; more clarity between the basic query (mapping data for a single disease) and advanced query (comparing diseases and demographics); and provision of additional data visualizations.                 Most external users of AtlasPlus identified as disease program managers (N=20) or epidemiologists (N=16). External users reported being satisfied (65%) with AtlasPlus, and reported it as being easy to use (54%). External users engaged most with the change over time feature (62%), followed by generating data tables for a single disease (41%), the advanced query (35%), and mapping (27%). Qualitative themes revealed the utility of AtlasPlus in comparing diseases where data are not otherwise available (e.g. comparing border and peer counties). Usability issues discussed included the advanced feature’s layout, directions on use, and functionality.

Conclusions: Internally and externally to NCHHSTP, disease program staff and epidemiologists were frequent users of AtlasPlus. Users reported positive perceptions of the site and identified similar areas for usability improvement—increased data and graphical displays and more overall user-friendliness—particularly with the advanced query.

Implications for research and/or practice: Future development of AtlasPlus should consider the needs of frequent users’ characteristics, perceptions, behaviors, and usability concerns in order to effectively align with their data visualization and disease data needs.