Attachment B - Did You Know 2011 Results

“Did You Know?” Evaluation

Presented by
Melissa McAvoy
Health Communication Specialist
Applied Systems Research and Evaluation Branch
Division of Public Health Performance Improvement

December 17, 2012
Division of Public Health Performance Improvement
Office for State, Tribal, Local and Territorial Support

Acknowledgements

Did You Know? Leadership
– Chelsea C. Payne, MPH
– Karen Resha, MA

Did You Know? Email
Content Development
– Angela Jiles, MPH
– Julissa Portillo, MPH

Evaluation Design and
Analysis
– LCDR Bobby Rasulnia,
PhD, MPA, MPH, CHES
– Angela Jiles, MPH

Purpose of the “Did You Know?” Evaluation

To quantify success of the project thus far, as well as any
improvements that can be made to increase effectiveness and
usability of the product and satisfaction of health department
staff.

Specifically, the survey was designed to determine respondent







Characteristics
Preferences for email delivery
Preferences for content
Decision making with regards to clicking on links
Opinion on utility
Suggestions for improving product

Methodology, Intended Audience, and Response Rate

Data was gathered through a survey and email metrics
 Survey Dates: August 23, 2011 – September 12, 2011
 Email Metrics: February 11, 2011 to March 9, 2012
• Metric tracking first began for emails on February 11, 2012

Intended Audience: state, tribal, local, and territorial public
health agency staff (STLTs)

Respondent Universe: ~3,000 subscribers positively
identified as STLT staff by email address out of 9,700 total
subscribers

Response Rate
 Respondent Goal: 500
 Final # of respondents: 230
 Response rate: 46%

Respondents came from a variety of states and
a majority were local health departments.

41 states and 1 territory/pacific island

Jurisdictions






76% local (n=175)
17% state (n=39)
6% tribal (n=14)
<1% territory/pacific island (n=1)
<1% could not determine (n=1)

Most respondents were leaders in their health organization.
78% were leaders (n=174) and of
those:

24% were non-leaders (n=56) and of
those:

•

58% public health directors, and/or
their deputies

•

17% nurses

•

33% health officials and/or their
deputies

•

7% emergency preparedness staff

•

There were twenty two other roles
including:

•

There were nine other leadership
roles including:
– Program directors, managers, or
coordinators
– Nursing administrators and/or
supervisors
– Division or branch directors and/or
their deputies
– Board of health members

– Administrative/clerical staff
– Health educators
– Environmental health staff
– Epidemiologists

* Respondents could select multiple roles and hold a leadership and non-leadership role simultaneously.

IS THE CONTENT APPROPRIATE FOR AND
REACHING THE INTENDED AUDIENCE?

Most respondents believe that they are the primary
audience for “Did You Know?”

Most frequent responses
 87% Local public health professionals
 75% Public health leadership
 72% Healthcare providers
• This could be due to the fact that many local health departments are
healthcare providers.

 41% consummers / general public

Public health professionals who did not think they were
part of the primary audience for “Did You Know?”
 43% tribal (n=14)

WHAT ARE THE PREFERENCES FOR
DELIVERY?

Respondents like the current delivery
day and frequency.

Best Day
 58% Friday (current delivery day)
 16% Monday

Frequency
 70% Once a week (current frequency of delivery)
 11% once a month

WHAT ARE THE PREFERENCES FOR
TYPES OF CONTENT AND TOPICS?

Leaders want more strategies while
non-leaders want more free resources and materials.
Most frequent responses by all respondents
•
•
•
•
•

74% Newly released data and statistics
73% Evidence-based strategies
65% New CDC guidelines and recommendations
49% Free resources and materials
47% Information relevant to my local community

Commonalities Between Top 5 Answers Among Leaders and Non-Leaders
Newly released data, evidence-based strategies, new CDC guidelines and recommendations, and
information relevant to my local community

Major Differences in Top 5 Between Leaders and Non-Leaders
Leaders

Non-Leaders

(n=174)

(n=56)

Evidence-based strategies

78%

57%

Information on what other jurisdictions are doing

51%

36%

Free resources and materials

44%

64%

Top 5 Content Types Respondents Would Link
in Future Editions of “Did You Know?”
• Most frequent responses
–
–
–
–
–

74% Newly released data and statistics
73% Evidence-based strategies
65% New CDC guidelines and recommendations
49% Free resources and materials
47% Information relevant to my local community

• Least frequent responses
–
–
–
–
–

21% Upcoming events such as conferences, meetings, grand rounds, etc
25% New journal articles and publications
35% Updated policy information
35% Workforce development materials including training
40% Performance improvement information

Chronic disease information was
of most interest.
What topics are you interested in seeing for "Did You Know?" in the future?
Topics where # Responses > 10 (n=183)
Chronic Disease

50

Practices and Strategies

43

New, Timely, Updated

36

Prevention

29

Infectious Diseases

27

PMQI and Evaluation

24

Local-level information

23

Audience-specific Information

19

Environmental Health

18
0

* Free response question

10

20

30

40

50

60

WHAT MOTIVATES READERS TO
CLICK ON LINKS?

Respondents and actual click data show a preference
for link #1 and bullet #1.
Click-through Rate controls for the number of
emails opened each week which typically
increases or decreases as the number of
subscribers increase or decrease.
# Unique Link Clicks
# Unique Email Opens

Number of survey respondents selecting a
link declined as link position and bullet
position declined.

•

Link Selection by Link Position
– 41.7% Link #1
– 21.3% Link #2
– 22.2% Link #3
– 14.8% Link #4

Actual Mean and Median Click-through Rates
by Link Position
12.0%
10.0%
8.0%
6.0%
4.0%
2.0%
0.0%
Link #1

Link #2
Mean

Link #3

Median

Actual Mean Click-through Rates by Bullet
Position
12.00%
10.00%
8.00%
6.00%

•

Link Selection by Bullet Position
– 49.1% Bullet #1
– 23.0% Bullet #2
– 27.8% Bullet #3

4.00%
2.00%
0.00%

* See speaker notes for details on how this analysis was conducted.

Bullet #1

Bullet #2

Bullet #3

HOW DO RESPONDENTS USE “DID
YOU KNOW?”

More than half of respondents have used “Did You
Know?” to support or inform practices or policies.
60% have used “Did You Know?” to inform or support practices and/or policies at their
work (62% of leaders and managers, 55% of non-leaders)

Use
1 Used to Educate the Public
2 Used to Prepare Materials
3 Started, Changed, or Supported Internal Programs or Practices
Shared with Task Forces, Work Groups, Committees, and/or Community
4
Organizations
5 Discussed or Presented to Board of Health
6 Shared with Internal Staff
7 Changed or Supported Internal Policy and/or Educated External Policy Makers
8 Discussed with Other Public Health Leaders
9 Evaluated Health Issues, Practices, or Policies
10 Shared with Colleagues or Peers
11 Used for Individual Professional Use or Knowledge
12 Applied for a Grant
13 Discussed with Elected Officials

% of
Respondents
(N=100)
27%
20%
20%
13%
11%
10%
9%
6%
4%
3%
2%
2%
1%

CONCLUSIONS AND
OPPORTUNITIES TO CONSIDER

Opportunities to consider

Moving the most important content to the first link and first bullet

Developing content that targets (or can be used by) multiple
audiences, including the public

Using these results to help characterize and understand Gateway
users in general

Continuing to provide:
 New data and statistics (preferably broken down by STLT jurisdiction)
 Evidence-based strategies,
 New CDC guidelines and recommendations.

Recommendations for further evaluation

Use findings to inform an evaluation of all / more OSTLTS
communication products

Conduct an A/B test to determine whether the preference for the
first link and first bullet is caused by the link order, writing style, or
content

Limitations

Survey responses did not include non-subscribers, thus
introducing bias

Survey response rate was low which is typical for internet and
email survey
 Dividing tasks into seven parts created small sample sizes for individual tasks which
was in part caused by the lower overall survey response rate

Survey did not capture a large enough sample of tribal or
territorial subscribers to draw conclusions

Questions?

For more information please contact Centers for Disease Control and Prevention
1600 Clifton Road NE, Atlanta, GA 30333
Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348
E-mail: [email protected] Web: http://www.cdc.gov
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the
Centers for Disease Control and Prevention.

Office of the Director
Place Office Title Here