HCI 5073 Module 4 Technology Evaluation Example

A Dashboard Worth Opening at 7 a.m.? Evaluating a County Heat-Health Dashboard Against Seven Features of Actionable Public Health Dashboards

Student Name

American College of Education

HCI5073: Public Health Informatics

Module 4 Assignment

Instructor Name

October 26, 2026

The Tool Being Evaluated

Earlier modules of this project examined how a composite Midwestern county detects heat-related illness through emergency department syndromic data and how good those data are. This paper evaluates the tool that turns the data into something people can use: the county's heat-health dashboard. Built by the county's epidemiologist in a commercial visualization program and refreshed each morning from the state syndromic platform, the dashboard has two versions. The internal version, used by public health, emergency management and outreach staff, shows daily counts and rates of heat-related emergency visits, a seven-day trend line, breakdowns by age group and sex, a ZIP code map and the day's heat index forecast. The public version, posted on the county website during summer, shows weekly counts, the map at a coarser level and prevention messages.

The question is not whether the dashboard looks polished but whether it helps people act: to decide where outreach teams go, when to open cooling centers and what to tell residents.

Evaluation Approach

A team of European and North American researchers assessed 158 public COVID-19 dashboards from 53 countries and asked an international expert panel to judge how actionable they were (Ivanković et al., 2021). Only 20 were rated highly actionable, and the panel identified seven features those dashboards shared: they know their audience and information needs; manage the type, volume and flow of information; report data sources and methods clearly; link time trends to policy decisions; provide data that are close to home; break down the population into relevant subgroups; and use storytelling and visual cues. Although the features came from pandemic dashboards, they describe what any surveillance dashboard must do, so this evaluation uses them as criteria.

The criteria were applied in two ways. First, the dashboard was reviewed feature by feature against its documentation. Second, eight users completed a short set of tasks while thinking aloud: four staff members from public health and emergency management, two outreach workers and two community members from a neighborhood association. Tasks included finding which ZIP codes had the most heat-related visits yesterday, judging whether visits were rising and locating advice for older adults.

Audience, Volume and Sources

The internal dashboard serves its main audience reasonably well. Staff completed most tasks within a minute, and the epidemiologist's daily summary panel, a three-sentence note at the top, was the element users read first. The problem is volume. The internal version has eleven charts on one screen, including several that no user consulted, such as visits by sex and a cumulative seasonal total. Emergency managers said they wanted three things: whether visits were above normal, where, and whether the forecast suggested tomorrow would be worse. Those answers were present but scattered.

Reporting of sources and methods is weak in both versions. Neither explains that the counts come from emergency visits only, that the query misses some heat illness recorded under other complaints, or that recent days may rise as late diagnosis codes arrive. In the international review, only about one in five dashboards explained the quality and meaning of their data (Ivanković et al., 2021), and this dashboard is in the majority. A user who compares Monday's count with the same count viewed on Thursday may see it change and lose trust.

Trends Linked to Decisions

The seven-day trend line is the most useful display for staff, but it is not linked to the decisions it should inform. Heat advisories, cooling center openings and outreach days are not marked on the timeline, so users cannot see whether visits fell after an action or whether the dashboard's rise preceded the warning. Adding markers for heat alerts and county actions would let managers learn from each event. The value of this link is shown nationally: Schramm et al. (2021) used syndromic data to document how heat-related emergency visits in the Northwest rose sharply during the record June 2021 heat wave, far above visits in the same period of earlier years, which is the kind of comparison against a baseline that makes a trend actionable. The county's dashboard shows no seasonal baseline, so users could not tell whether 14 visits in a day was high or typical.

Close to Home and Subgroups

The ZIP code map is the feature users liked most and the one most likely to mislead. Because heat-related visits are counted by the patient's residence, a ZIP code with few residents can show a high rate from two or three visits, and the default color scale gave those areas the darkest shading. Two staff members identified the wrong ZIP code as the highest-risk area during testing for this reason. Suppressing or flagging rates based on fewer than five visits, or using three-day totals, would reduce the problem. The age breakdown is valuable, since adults over 65 are the county's priority group, but the dashboard reports only two age groups, under and over 65, which hides the pattern among working-age adults who are exposed outdoors. Vaidyanathan et al. (2024), reporting on the 2023 warm season, found that more males than females sought emergency care for heat-related illness, especially those aged 18 to 64, which suggests that outdoor workers deserve their own view.

Storytelling and Visual Cues

The public version tells almost no story. It shows a weekly bar chart and a map without explaining what residents should do with them, and both community testers said they would not return to it. The prevention messages sit below the charts in small type. A version that opened with a plain statement, such as the number of residents who needed emergency care for heat last week and the three neighborhoods most affected, followed by where to find cooling centers, would connect the data to action for residents. Visual cues also need work: the internal dashboard uses red for both high counts and for the forecast heat index, which two users confused, and the dashboard does not display well on phones, where outreach workers usually view it.

When a Dashboard Adds Noise

Dashboards can also do harm. Daily counts in a county of this size are small and fluctuate, and staff sometimes reacted to a single-day rise that disappeared the next day. Several users said they looked at the dashboard less as the summer went on because it seldom changed their plans. A dashboard that updates constantly but rarely signals something important trains people to ignore it. The fix is not more charts but clearer thresholds: an alert that appears only when visits exceed a statistical baseline for two consecutive days would help staff separate signal from routine variation.

Recommendations and Conclusion

The dashboard meets two of the seven features well, audience and subgroup breakdown for older adults, meets three partly and misses two, clear methods and storytelling for the public. Recommendations follow directly. The internal version should be reduced to four panels: an above-normal indicator against a seasonal baseline, the trend with actions marked, a map with small numbers flagged and the forecast. A short methods note should state where the counts come from, which cases they miss and why recent days may rise. The public version should lead with a plain-language summary and cooling center information and be designed for phones. With these changes the dashboard could become what staff need: a tool they open each morning because it helps them decide what to do that day.

References

Ivanković, D., Barbazza, E., Bos, V., Brito Fernandes, Ó., Jamieson Gilmore, K., Jansen, T., Kara, P., Larrain, N., Lu, S., Meza-Torres, B., Mulyanto, J., Poldrugovac, M., Rotar, A., Wang, S., Willmington, C., Yang, Y., Yelgezekova, Z., Allin, S., Klazinga, N., & Kringos, D. (2021). Features constituting actionable COVID-19 dashboards: Descriptive assessment and expert appraisal of 158 public web-based COVID-19 dashboards. Journal of Medical Internet Research, 23(2), e25682. https://doi.org/10.2196/25682

Schramm, P. J., Vaidyanathan, A., Radhakrishnan, L., Gates, A., Hartnett, K., & Breysse, P. (2021). Heat-related emergency department visits during the northwestern heat wave: United States, June 2021. MMWR. Morbidity and Mortality Weekly Report, 70(29), 1020-1021. https://doi.org/10.15585/mmwr.mm7029e1

Vaidyanathan, A., Gates, A., Brown, C., Prezzato, E., & Bernstein, A. (2024). Heat-related emergency department visits: United States, May-September 2023. MMWR. Morbidity and Mortality Weekly Report, 73(15), 324-329. https://doi.org/10.15585/mmwr.mm7315a1