Supporting Statement A

CDC Prevention Status Reports: User Satisfaction and Impact

OSTLTS Generic Information Collection Request

OMB No. 0920-0879

Supporting Statement – Section A

Submitted: 12/10/14

Program Official/Project Officer

Garry Lowry, MPH

Health Scientist

Centers for Disease Control and Prevention, Office for State, Tribal, Local and Territorial Support, Division of Public Health Performance Improvement

4770 Buford Hwy, NE, MS E-70, Atlanta, GA 30024

Phone: 404-498-0361

Fax: 404-315-2369

Email: [email protected]

Section A – Justification

1. Circumstances Making the Collection of Information Necessary

Background

This information collection is being conducted using the Generic Information Collection mechanism of the OSTLTS OMB Clearance Center (O2C2) – OMB No. 0920-0879. The respondent universe for this information collection aligns with that of the O2C2. Data will be collected from US state health department officials and staff from all 50 states and Washington D.C., acting in their official capacities. The web-based assessment will collect data from 612 state health department staff. The key informant interviews will collect data from a sample of 20 respondents who completed the web assessment for an overall total of 632 respondents.

This information collection is authorized by Section 301 of the Public Health Service Act (42 U.S.C. 241). This information collection falls under the essential public health service of evaluating effectiveness, accessibility, and quality of personal and population-based health services.¹

CDC’s Office for State, Tribal, Local and Territorial Support (OSTLTS) was formed in 2010, under the leadership of Dr. Judith Monroe, to oversee a breadth of crosscutting programs and projects that increase CDC’s support and technical assistance to STLT health agencies. OSTLTS acts as an internal and external liaison to identify gaps, opportunities for collaboration, and strategies to support CDC’s public health work. An effective and coordinated communication strategy is at the heart of OSTLTS’ work to build the framework to identify, validate, disseminate, and adopt the highest standards, the most effective policies, and the best evidence-based practices.

In 2011, CDC Director Dr. Thomas R. Frieden commissioned OSTLTS with creating and disseminating the Prevention Status Reports (PSRs), a communication tool meant to help public health leaders and key decision makers advance evidence-based public health policy and practice in all 50 states and the District of Columbia by sharing the status of key public health indicators and performance on key policy and practice indicators, and identifying areas where improvements can be made. The purpose of this data collection is to assess the quality of the PSRs and their effectiveness in stimulating evidence-based public health policy and practice.

The PSRs highlight—for all 50 states and the District of Columbia—the status of public health policies and practices designed to prevent or reduce the following ten important public health problems and concerns:

• Excessive Alcohol Use

• Food Safety

• Healthcare-Associated Infections

• Heart Disease and Stroke

• HIV

• Motor Vehicle Injuries

• Nutrition, Physical Activity, and Obesity

• Prescription Drug Overdose

• Teen Pregnancy

• Tobacco Use

Each report follows a simple framework:

• Describe the public health problem or concern using public health data

• Identify potential solutions to the problem drawn from research and expert recommendations

• Report the status of those solutions (using a three-level rating scale—green, yellow, or red—to indicate the extent to which selected policies or practices are being implemented in each state and the District of Columbia)

A state’s complete set of PSRs consists of ten topic-specific reports and the state summary—a two-page summary of the state’s PSR ratings. A complete set of the PSRs for the state of Georgia is included as an example (see Attachment A—Georgia PSR). Other PSR products include:

• Quick Start Guide—Provides tips and tools for using the PSRs to increase the use of evidence-based public health practices and improve health outcomes (see Attachment B—PSR Quick Start Guide)

• National Summary—A national report that shows the percent of states rated green, yellow, or red for each PSR public health policy and practice indicator (see Attachment C—PSR National Summary)

All PSR materials are available from CDC’s website.

A one-time web-based assessment of the PSRs was conducted in the fall of 2011, OMB No. 0920-0879 (see Attachment D—PSR 2011 Assessment Results). Since this was OSTLTS’ first pilot of the PSRs, the 2011 reports were sent to a limited audience—all state health officials and senior deputies via US mail. The purpose of the prior assessment (web-based data collection) was to assess health official perceptions of the utility of the PSRs, their utilization of the PSRs, and their recommendations for improvement. Analysis of qualitative data resulted in the following themes regarding improvement of the PSRs: clarify purpose, improve the development process, enhance content, maintain basic format, and improve dissemination. Assessment feedback was used to improve the PSRs. For example, more detailed information is provided on each topic report stating the purpose of the reports. The development process was enhanced to include engagement of state health department leaders and staff in vetting indicators and reviewing data. Content was enhanced by developing a Quick Start Guide to identify potential uses of the reports and providing steps for initiating use. Very minimal changes were made to the format of the reports due to favorable comments about format and design. Finally, the newly designed PSRs were released publicly via CDC’s website in January 2014 resulting in much wider dissemination. An email, including a direct link to CDC’s website, announcing the release of the PSRs was sent to public health program staff at all levels and other public health partners (e.g., hospital administrators, legislators, and medical residency staff).

Since the 2011 assessment, no further data collection has been done to assess the PSRs. Assessing the newly designed PSRs is important to understand the usefulness of the reports to a wider audience. OSTLTS will be working with Health Resources in Action (HRiA), in close collaboration with the National Network of Public Health Institutes (NNPHI), to conduct the proposed current assessment to determine reach, user satisfaction, usefulness, and impact of the PSRs. A multiphase assessment, using mixed methods—both online assessment and key informant interviews (beyond health officials and senior deputies)—will be conducted to assess:

• The extent to which the PSRs are distributed to the target audience (i.e., US state health department officials and staff)

• The extent to which recipients report that they are satisfied with the quality of the PSRs

• The extent to which recipients report that they are using the PSRs

• How recipients report using the PSRs

• Any actions users report taking to advance evidence-based and expert-recommended policies and practices due to the PSRs

• Opportunities for continuous quality improvement of the PSRs

Assessment data will ultimately be used to understand the impact of the PSRs and identify opportunities for improvement. NNPHI/HRiA will produce a final report that is thematically organized and describes key findings and strategic directions for OSTLTS to consider to improve upon and support the use of PSRs in the future. A final internal assessment report will be prepared for CDC staff, including an executive summary. In addition to a final report, a core PowerPoint presentation of key assessment findings will also be developed for CDC to deliver and tailor by audience. Assessment findings will also be used to develop manuscripts to submit for publication in peer-reviewed journals focused on assessment and public health practice.

Overview of the Information Collection System

The information collection consists of a web-based questionnaire (see Attachment E—Online Instrument: Word version and Attachment F—Online Instrument: Web version) and subsequent follow-up key informant interviews (see Attachment G—Interview Guide). The online instrument will be used to gather quantitative information from STLT public health staff that have accessed the reports, to provide a portrait of users’ experiences with the PSRs including successes, challenges, and suggestions for improvement. Open-ended questions will also be used to provide an opportunity for respondents to offer more detailed and specific feedback (e.g., suggestions for improvement). The instrument will include skip patterns in order to tailor questions depending on specific reports they have accessed. The online assessment was programmed using Qualtrics and was pilot tested by three public health professionals. Feedback from this group was used to refine questions as needed, ensure accurate programming and skip patterns, and establish the estimated time required to complete the online instrument.

Following the online data collection, 20 follow-up interviews will be conducted with a sample of health department staff (e.g., state health officials, program directors, etc.) that completed the web-based instrument. The purpose of the interviews is to gather more in-depth information from respondents to the online assessment regarding their experiences using the PSRs. The online tool, Qualtrics, will create an email link for each respondent with a unique identifier that will facilitate conducting the follow-up interviews. Interviews will be conducted by telephone immediately after the online assessment data are collected and analyzed. Discussions will explore online assessment topics in more depth and provide greater context to understand perceptions and use of the PSRs, including perceived successes, challenges, and suggestions for improvement. The interview guide was pilot tested by three public health professionals and feedback from this group was used to refine questions as needed and estimate the time required to complete the interviews.

Items of Information to be Collected –

Online Assessment

The online data collection instrument consists of 25 main questions of various types, including dichotomous (yes/no), multiple response, interval (rating scales), and open-ended. Due to the automatic skip patterns, it is expected that not all respondents will be presented with all questions. An effort was made to limit questions requiring narrative responses from respondents, by limiting the number of open-ended questions and using the key informant interviews as a more appropriate tool to elicit more detailed responses. The instrument will collect information on the following:

• Awareness and access to the PSRs;

• Current and intended use of the PSRs;

• Further dissemination of the PSRs (reach);

• User satisfaction with the quality of the PSRs, including satisfaction with other components (i.e., CDC PSR website, PSR Quick Start Guide);

• Recommendations for improving the PSRs; and

• Respondent characteristics – state/district current position, and level of position in organization.

Key Informant Interviews

To gather more in-depth information from users, 20 follow-up interviews will be conducted with a sample of health department staff (e.g., state health officials, program directors, etc.) that completed the web-based instrument. There are 5 main questions in the semi-structured telephone interview guide with probes under each question. If all probes are asked, then the guide totals 25 questions. All questions are open-ended and ask the respondent to provide specific examples that directly build on his/her responses from the web-based instrument to provide a more nuanced and comprehensive portrait of the health departments’ experiences.

The interview guide will collect information on the following:

• Specific examples of use/intended use of the PSRs that were identified on the web-based instrument (open-ended question); additional probes will be asked regarding facilitators and barriers to use of the PSRs.

• Specific examples of challenges to using the PSRs that were identified on the web-based instrument

• Elaboration on implementation of specific suggestions to improve use of the PSRs that were identified on the web-based instrument(open-ended question with a probe on what CDC would need to consider in implementation);

2. Purpose and Use of the Information Collection

The main purpose is to assess the extent to which PSRs support planning and decision-making about strategies to improve public health, lead to specific actions intended to advance evidence-based and expert-recommended public health policies and practices (e.g., targeted quality improvement efforts), and increase the use of evidence-based and expert-recommended policies and practices. This information does not exist for the newly designed and publicly available PSRs.

A final report would be organized thematically and describe any challenges encountered, factors associated with these challenges, successes identified by users, factors that have facilitated success, and lessons learned and strategic directions for OSTLTS to consider to improve upon and support the use of PSRs in the future. Based on findings from the data collection, OSTLTS may make additional modifications to the PSRs, augment the PSRs with additional supporting products, and/or enhance communication and dissemination efforts. A final internal assessment report will be prepared, including an executive summary. In addition to a final report, a core PowerPoint presentation of key assessment findings will also be developed and tailored by audience. Assessment findings will also be used to develop manuscripts to submit for publication in peer-reviewed journals focused on assessment and public health practice.

3. Use of Improved Information Technology and Burden Reduction

The data collection will employ a mixed methods approach to reduce unnecessary burden on respondents. All respondents will complete the online assessment (programmed using Qualtrics), allowing them to complete and submit their responses electronically. The online assessment was designed to collect the minimum information necessary for the purposes of this project (i.e., limited to 25 questions) and includes skip patterns. Qualtrics will create an email link for each respondent with a unique identifier that will facilitate conducting the follow-up interviews. Assuming a sufficient survey sample is achieved, quantitative data analyses will compare responses by user type and geographic location to identify differences by these characteristics.

The purpose of the interviews is to gather more in-depth information from respondents to the online assessment regarding their experiences using the PSRs, particularly to collect contextual information missing from a quantitative assessment. The interviews will be conducted with a sample of 20 respondents who completed the web-based instrument to complement the limitations of close-ended questions on web-based tool and provide a more nuanced and comprehensive portrait of the user’s experiences. To minimize burden, interviews will be scheduled at the respondents’ convenience within a two-week period. While questions are open-ended, they directly build off of the respondents’ answer on the web-based tool and ask for specific examples and illustrations of challenges, facilitators, and suggestions. The conversation is designed to be brief and focused. Using NVivo software, qualitative data from interviews will be coded and analyzed thematically for main themes and sub-themes.

4. Efforts to Identify Duplication and Use of Similar Information

The information being collected is specific to the PSRs and there is currently no information available that can substitute for direct responses from the target audience—STLT public health staff. Because these are unique products and target respondents are a critical stakeholder group for CDC and OSTLTS, there is no existing data which could replace the need to gather data through this data collection. Although OSTLTS conducted an initial assessment of the PSRs in 2011 to assess state health official satisfaction with and use of the PSRs, the PSRs were redesigned in response to respondent feedback. The 2011 assessment was also only sent to a limited audience (based on initial limited dissemination of pilot product), and the current data collection expands the number and types of respondents, and includes telephone interviews to collect contextual information missing from a quantitative assessment. The metrics data in SiteCatalyst/Omniture will be used measure the amount and type of activity on the website. However, this tool cannot capture user demographics, preferences, satisfaction, use, or identify areas for improvement.

5. Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this information collection.

6. Consequences of Collecting the Information Less Frequently

This request is for a one time information collection. There are no legal obstacles to reduce the burden. If no data are collected, OSTLTS will be unable to understand:

• The extent to which recipients report that they are able to effectively use the data and information presented in the PSRs

• How recipients are using the PSRs

• Any actions taken due to the PSRs to advance evidence-based and expert-recommended policies and practices

• Opportunities for continuous quality improvement of the PSRs

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances with this information collection package. This request fully complies with the regulation 5 CFR 1320.5 and will be voluntary.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

This information collection is being conducted using the Generic Information Collection mechanism of the OSTLTS OMB Clearance Center (O2C2) – OMB No. 0920-0879. A 60-day Federal Register Notice was published in the Federal Register on October 31, 2013, Vol. 78, No. 211; pp. 653 25-26. No comments were received.

CDC partners with professional STLT organizations, such as the Association of State and Territorial Health Officials (ASTHO), the National Association of County and City Health Officials (NACCHO), and the National Association of Local Boards of Health (NALBOH) along with the National Center for Health Statistics (NCHS) to ensure that the collection requests under individual ICs are not in conflict with collections they have or will have in the field within the same timeframe.

9. Explanation of Any Payment or Gift to Respondents

CDC will not provide payments or gifts to respondents.

10. Assurance of Confidentiality Provided to Respondents

The Privacy Act does not apply to this information collection. STLT governmental staff will be speaking from their official roles and will not be asked, nor will they provide individually identifiable information.

This information collection is not research involving human subjects.

1. Privacy Impact Assessment Information

No individually identifiable information (IIF) will be collected.

11. Justification for Sensitive Questions

No information will be collected that are of personal or sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Online Assessment

The estimate for burden hours is based on a pilot test of the online instrument by three public health professionals. In the pilot test, the average time to complete the instrument including time for reviewing instructions, gathering needed information and completing the instrument, was approximately 11 minutes. Based on these results, the estimated time range for actual respondents to complete the instrument is 10-15 minutes. For the purposes of estimating burden hours, the upper limit of this range (15 minutes) is used.

Key Informant Interviews

The estimate for burden hours is based on a pilot test of the interview guide by three public health professionals. In the pilot test, the estimated time to complete the telephone interviews ranged from 15 to 20 minutes. For the purposes of estimating burden hours, the upper limit of this range (i.e., 20 number minutes) is used.

Estimates for the average hourly wage for respondents are based on the Department of Labor (DOL) National Compensation Survey estimate for management occupations – medical and health services managers in state government (http://www.bls.gov/ncs/ocs/sp/nctb1349.pdf). Based on DOL data, an average hourly wage of $57.11 is estimated for all 632 number respondents. Table A-12 shows estimated burden and cost information.

Table A-12: Estimated Annualized Burden Hours and Costs to Respondents

Data Collection Instrument: Form Name	Type of Respondent	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Online Assessment	State health department officials and staff who received the PSRs	612	1	15/60	153	57.11	8737.83
Interview Guide	State health department officials and staff who reported using the PSRs	20	1	20/60	7	57.11	399.77
	TOTALS	632	1		160		9137.60

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There will be no direct costs to the respondents other than their time to participate in each information collection.

14. Annualized Cost to the Government

There are no equipment or overhead costs. Contractors, however, are being used to support development of the assessment tool, data collection, and data analysis. The only cost to the federal government would be the salary of CDC staff and contractors.

The lead FTE for this project is a Senior Health Scientist. The CDC lead will provide consultation and oversight on the development of the assessment instrument and interview guide, analysis plan, documentation of findings, and OMB application package. The Health Communication Specialist will provide additional support on instrument development and lead development of the OMB package. The majority of work on this project will be carried out by external contractors, NNPHI and HRiA, including primary development of the assessment tool, data collection, data review and analysis, and documenting findings.

The total estimated cost to the federal government is $38,901. Table A-14 describes how this cost estimate was calculated.

Table A-14: Estimated Annualized Cost to the Federal Government

Staff (FTE)	Average Hours per Collection	Average Hourly Rate			Average Cost
Health Scientist (GS-14) Consultation on and oversight of development of OMB package; Consultation with and oversight of contractors for instrument development, data collection, data analysis, quality control and report preparation	160	$57.04			$9,126
Health Communication Specialist (GS-9) Development of OMB package; Consultation with contractors on instrument development, data collection, data analysis, quality control and report preparation	120	$24.79			$2,975
NNPHI/HRiA Contractors Instrument development, pilot testing, web-based instrument programming, data collection, data coding and entry, quality control, data analysis, report preparation	400	$67.00			$26,800
Estimated Total Cost of Information Collection					$38,901

15. Explanation for Program Changes or Adjustments

This is a new information collection.

16. Plans for Tabulation and Publication and Project Time Schedule

The results will be used to create both internal and external reports.

• A final internal assessment report will be prepared, including an executive summary. This final report would be organized thematically and describe any challenges encountered, factors associated with these challenges, successes identified by users, factors that have facilitated success, and lessons learned and strategic directions for OSTLTS to consider to improve upon and support the use of PSRs in the future.

• A core PowerPoint presentation of key assessment findings will also be developed and tailored by audience.

• Manuscripts will be developed for submission to peer-reviewed journals (focused on assessment and public health practice) for publication.

Project Time Schedule

• Design survey questionnaire (COMPLETE)

• Develop survey protocol, instructions, and analysis plan (COMPLETE)

• Pilot test survey questionnaire (COMPLETE)

• Prepare OMB package (COMPLETE)

• Submit OMB package (COMPLETE)

• OMB approval (TBD)

• Conduct online assessment (Assessment open 4 weeks)

• Code, quality control, and analyze data from online assessment (4 weeks)

• Conduct telephone interviews (4 weeks)

• Collect, code, enter, quality control, and analyze qualitative data from interviews (3 weeks)

• Prepare reports (5 weeks)

• Disseminate results/reports (Date TBD)

17. Reason(s) Display of OMB Expiration Date is Inappropriate

We are requesting no exemption.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.

LIST OF ATTACHMENTS – Section A

Note: Attachments are included as separate files as instructed.

1. Georgia PSR

2. PSR Quick Start Guide

3. PSR National Summary

4. PSR 2011 Assessment Results

5. Online Instrument: Word version

6. Online Instrument: Web version

7. Interview Guide

REFERENCE LIST

1. Centers for Disease Control and Prevention (CDC). "National Public Health Performance Standards Program (NPHPSP): 10 Essential Public Health Services." Available at http://www.cdc.gov/nphpsp/essentialservices.html. Accessed on 8/14/14.