ICR - SSA_CoreSIPP_ProgramEval_final

SUPPORTING STATEMENT: PART A

Program Evaluation of CDC’s Core State Injury Prevention Program

OMB# 0920-New

October 12, 2021

Point of Contact:

Brandon Nesbit

Contact Information:

Centers for Disease Control and Prevention

National Center for Injury Prevention and Control

4770 Buford Highway NE MS F-64

Atlanta, GA 30341-3724

Phone: 770-488-0637

Email: [email protected]

CONTENTS

Section Page

A. Summary table 3

Justification 3

A.1. Circumstances Making the Collection of Information Necessary 3

A.2. Purpose and Use of Information Collection 4

A.3. Use of Improved Information Technology and Burden Reduction 7

A.4. Efforts to Identify Duplication and Use of Similar Information 8

A.5. Impact on Small Businesses or Other Small Entities 8

A.6. Consequences of Collecting the Information Less Frequently 8

A.7. Special Circumstances Relating to the Guidelines of

5 CFR 1320.5(d)2 9

A.8. Comments in Response to the Federal Register Notice and

Efforts to Consult Outside the Agency 9

A.9. Explanation of Any Payment or Gift to Respondents 9

A.10. Protection of the Privacy and Confidentiality of Information Provided by Respondents 9

A.11. Institutional Review Board (IRB) and Justification for Sensitive Questions 10

A.12. Estimates of Annualized Burden Hours and Costs 10

A.13. Estimates of Other Total Annual Cost Burden to Respondents

or Record Keepers 13

A.14. Annualized Cost to the Government 13

A.15. Explanation for Program Changes or Adjustments 14

A.16. Plans for Tabulation and Publication and Project Time Schedule 14

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate 14

A.18. Exceptions to Certification for Paperwork Reducation Act

Submissions 15

Attachments

A Authorizing Legislation: Public Health Service Act

B Published 60-Day Federal Register Notice

B1 Public Comments

C Research Determination

D Privacy Act Applicability

E Implementation and Capacity Rubric

F Economic Indicators

G Recipient-level Group Interviews

H Death Injury Indicators Spreadsheet

I Emergency Department Injury Indicators Spreadsheet

J Hospital Discharge Injury Indicators Spreadsheet

K Special Emphasis Report

L Partner’s Portal Screen Shoots

• Goal of the study

This program evaluation is intended to assess both recipient-level and program-level outcomes associated with the NCIPC’s Core SIPP funded state injury prevention program. Evaluation metrics will assess injury prevention-focused infrastructure development, surveillance system development and use, and partnerships, to prevent Adverse Childhood Experiences (ACEs), Traumatic Brain Injury (TBI), and transportation-related injuries. Recipient identification of disproportionately affected populations and subsequent public health actions taken to address injury-related health disparities will also be assessed.

• Intended use of the resulting data

Data collected in support of this program evaluation will be used to monitor progress toward program goals, identify technical assistance needs of recipients, to identify practice-based evidence for injury prevention public health actions as defined under this program, and for Continuous Quality Improvement (CQI) purposes.

• Methods to be used to collect

Mixed methods data collection. Data will be reported annually on quantitative metrics and qualitative surveys, and interviews. No research design or human subjects involved.

• The subpopulation to be studied

100% of the populations are in funded recipient jurisdictions.

• How data will be analyzed

The data will be analyzed using descriptive and summary statistics as well as qualitative analysis and summaries.

A. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

CDC requests OMB approval for 3 years for this new data collection. Approval is requested to collect information from awardees funded under the Core State Injury Prevention Program cooperative agreement, hereafter known as Core SIPP. This program is a new initiative. As part of the annual program evaluation data collection, recipients will submit data on enhancements in program implementation capacity (Attachment E), leveraged resources/funds through economic indicators (Attachment F), and challenges and successes, programmatic improvements, and impact through interviews (Attachment G). Finally, awardees will annually submit injury and violence prevention surveillance data using an Excel-based Injury Indicator Spreadsheets (Attachment H, I, and J) and Special Emphasis Reports (Attachment K).

Information to be collected will provide crucial data for program evaluation and provide CDC with the ability to respond in a timely manner to requests for information about the program from the Department of Health and Human Services (HHS), the White House, Congress, and other increased capacity, understand how the cooperative agreement increases potential sustainability though improved capacity, provide data-driven technical assistance, and disseminate the most current surveillance data on unintentional and intentional injuries.

Authority for CDC’s National Center for Injury Prevention and Control (NCIPC) to collect these data is granted by Section 301 of the Public Health Service Act (42 U.S.C. 241). This act gives federal health agencies, such as CDC, broad authority to collect data and participate in other public health activities, including this type of program implementation evaluation (Attachment A).

A.2. Purpose and Use of Information Collection

The Core SIPP evaluation will collect several types of information from recipients over the course of the funding cycle. This information will be used to:

1) Evaluate and track outcomes at the recipient- and program-levels as they relate to injury prevention-focused infrastructure development, surveillance system development and use, and partnerships, to prevent Adverse Childhood Experiences (ACEs), Traumatic Brain Injury (TBI), and transportation-related injuries. Recipient-and program-level identification of disproportionately affected populations and subsequent public health actions taken to address injury-related health disparities will also be assessed.

2) Identify technical assistance needs of individual recipients and this recipient cohort, so that the CDC team can appropriately deploy resources to support recipients.

3) Identify practice-based evidence for injury prevention public health actions to advance the field through future partnerships, program design, and publications.

4) Inform continuous quality improvement activities over the course of the funding period, to include quarterly and annual strategic planning for current and later iterations of this program under future funding.

Information will be collected by CDC through the following modes to address the purposes identified above (also see Table. A.2.1)

1) The Core SIPP Implementation Capacity Development Rubric will be implemented once at the start of program funding (baseline collection), and subsequently during the middle of each reporting year. Recipients will self-administer the rubric via CDC’s Partner Portal, where they will self-score their state injury prevention programs according to their current level of capacity for components of interest. These scores will be used to identify recipient strengths, areas for improvement, and additional needs for CDC TA support. Measuring recipient improvements in implementing public health actions in this standard way will greatly increase the ability for CDC to measure the impact of the program investment. CDC will also aggregate these scores across recipients to identify larger program needs and to inform internal Continuous Quality Improvement (CQI) activities. This information will be shared back with recipients individually during annual technical review calls, as well as in aggregate at annual partnership meetings. Additionally, increased capacity will increase the likelihood of sustainability beyond the funding cycle.

2) Recipient-level Group Interviews will take place at the end of Program Years 2 and 3. The purpose of these interviews is to evaluate progress and challenges in implementing the Core SIPP program within the individual recipient-level context to inform tailored supports from CDC and partners. The tailored support will be in effort to facilitate solutions to programmatic barriers, adjust recipient strategies as needed, and ensure the quality of data reported annually to CDC.

3) Economic Indicators will be collected to better understand the cost of IVP implementation by strategy as well as how recipients have leveraged funds and resources to increased sustainability for injury and violence prevention work.

4) Injury Indicator Spreadsheets and Special Emphasis Reports will be collected annually to track state level injury and violence morbidity and mortality data. This will allow CDC to measure trends over time within a state, across states, and against the national average to identify changes during the Core SIPP funding period. Completion of the spreadsheets and reports will also ensure recipient surveillance capacity and reporting is in alignment with best practices.

The Implementation Capacity Development Rubric is an adaptation of a validated and previously published instrument currently being used by State Health Department Injury Units for state plan assessment. Because the Core SIPP funded-cohort will be small (N=23), the analytical design will use basic graphing techniques to illustrate change over time at the recipient- and aggregate-levels, there will not be sufficient power for meaningful significance testing. However, charting change from year-to-year will allow the program to monitor recipient growth in an ongoing way and to shape technical assistance agendas when areas of slow progress are identified. Technical assistance provided is intended to enhance current implementation efforts as well as increase long-term sustainability for the program.

Information generated in these data collections will be shared with recipients (individually and in aggregate), program partners, CDC leadership, inform evidence-based practice and evaluation-related publications, and with internal CDC IVP research, evaluation, funding, and implementation strategy and planning sessions.

Recipients will be asked to identify and describe populations within their jurisdictions who experience disproportionately high injury burden; however, the race, ethnicity, or geographic location of any one person will not be requested.

Table A.2.1. Evaluation Questions by Method

Information Collection Type and Frequency	Evaluation Questions
Type	Frequency
Implementation Capacity Development Rubric (quantitative) Attachment E	Y1 = 2 Y2 = 1 Y3 = 1	How are recipients improving/increasing their capacity for IVP? How are recipients attending to improving sustainability of efforts beyond the funding cycle? Do recipients require technical assistance to improve progress? If so what technical assistance does the recipient require? How are recipients leveraging partnerships to increase/improve capacity for IVP?
Recipient-level Group Interview (qualitative) Attachment G	Y1 = 0 Y2 = 1 Y3 = 1	Do recipients require technical assistance to improve progress? If so what technical assistance does the recipient require? What are barriers that impede progress toward goals and outcomes? How are recipients facilitating solutions to identified barriers? How are recipients identifying, recording, and disseminating practice-based evidence? How are recipients using progress and evaluation data to make programmatic improvements?
Economic Indicators (Quantitative) Attachment F	Y1 = 1 Y2 = 1 Y3 = 1	What are the costs associated with implementing selected IVP strategies and activities? What other sources of funding contribute to the selected IVP strategies and activities? What in-kind support contribute to the selected IVP strategies and activities? How are CDC funds distributed across functions for IVP implementation (i.e. staffing, mini grants, etc.)?
Injury Indicators (Quantitative) Attachment H	Y1 = 1 Y2 = 1 Y3 = 1	How are state injury indicators changing over time? Is Core funding improving state capacity to collect and analyze surveillance data?
Special Emphasis Reports (Quantitative) Attachment I	Y1 = 1 Y2 = 1 Y3 = 1	What are state indicators related to topical areas of focus? How do state indicators compare to national averages? How do states compare to each other? How are state implementation efforts impacting topic specific indicators?

Table A.2.2. Methods to measure SIPP Outcome Indicators

SIPP Outcome Indicators	Methods/Instruments
Increased recipient knowledge and utilization of: • Emerging data sources for injury surveillance • Robust data/surveillance best practices	E,F
Increased understanding of injury among disproportionately affected populations	E,F
Increased stakeholder inclusion in program planning, implementation, and evaluation	E,F,G
Increased integration among multi-sectoral partners with shared commitment to injury prevention	E,F,G
Increased understanding of risk and protective factors for identified disproportionately affected communities	E,G
Increase understanding of appropriate evidence-informed strategies to address identified needs	E,G
Increased adoption of continuous quality improvement practices	E,G
Increased understanding of ongoing efforts and gaps in jurisdiction to address NOFO priority areas	E,G
Increased recipient ability to identify and respond to emerging injury threats	E,G
Increased recipient capacity to strengthen communities by increasing protective factors for injuries using best available evidence	E,G
Increased recipient capacity to strengthen communities by reducing risk factors for injuries using best available evidence	E,G
Sustain recipient injury prevention public health actions supported by best available evidence	E,G
Reduce and sustain injury morbidity and mortality associated with ACEs, Transportation Safety, and TBI	H,I,J
Reduce and sustain risk factors for ACEs, Transportation Safety, and TBI	H,I,J
Increase and sustain protective factors for ACEs, Transportation Safety, and TBI.	H,I,J

A.3. Use of Improved Information Technology and Burden Reduction

The CDC developed the Partner’s Portal (Attachment L) as well as excel spreadsheet (Injury Indicators) to collect the data outlined in this ICR. The data entry interface of the Partner’s Portal was developed using NCIPC-owned, Microsoft Azure, and Platform as a Service (PaaS) cloud solution approved for use by CDC programs. The use of the Partner’s Portal provides several advantages:

• This user-friendly online interface requires little training and will be easy and intuitive for recipients to use to enter data for the information collection.

• Standard data elements, definitions, and specifications at all levels improve the quality and comparability of information that recipients submit, and enhance the consistency of reports to examine information across recipients.

• The structure of the data collection in Partner’s Portal is flexible such that different recipients are still able to capture and report information relevant to their program context and structure.

• The ability to carry information and populate from one reporting period to the next increases the efficiency of data entry, reduces errors and redundancies, and therefore increases the quality and reliability of information that recipients submit each year.

Another advantage of the Partner’s Portal is that recipients can generate reports directly from the system, which allows recipients to fulfill their annual reporting obligations efficiently by submitting necessary information for both progress reports and continuation applications into the system once. This ability to save and update pertinent information from one reporting period to the next, will reduce the administrative burden of the annual reporting on recipients, and the review process on both recipients and CDC staff. Respondents will only need to modify or update the information, report data on measures, provide updates, or add new items as applicable.

These tools improve information quality by minimizing errors and redundancy. Having information consistently collected from all funded jurisdictions in the same manner year-over-year will reduce the level of burden attributable to redundancy and reduce the workload to enter and maintain the data. Additionally, jurisdictions will have data self-populated from one year to another, which minimizes data re-entry, burden, and potential errors. Finally, by providing data collection tools, which all will be using, jurisdictions will experience less burden because each location will not need to figure out how to collect data on their own.

Further, standardization will enhance the consistency of information collected, thereby enabling examination of cross-program strategies. The report generation capabilities of the web-based tools used will reduce the respondent burden associated with paper-based reports. Without the reporting tools and the integrated approach to information collection and reporting, funded jurisdictions and CDC would need to continue to use time consuming, labor-intensive procedures for information collection and reporting.

A.4. Efforts to Identify Duplication and Use of Similar Information

No effort to collect these evaluation data from Core SIPP recipients is being conducted within the agency. A performance monitoring collection (OMB# 0920-1120) for the previous Core SVIPP program was conducted under a previous funding cycle; however the previous collection was performance monitoring only and did not collect evaluation data as is being proposed here. This new collection enables CDC to evaluate the Core SIPP CDC-RFA-CE21-2101, which is the next iteration of the program. The collection of this qualitative information is part of a federal reporting requirement for funds received by recipients. The tools and methods will provide information necessary for a mixed methods approach to programmatic evaluation. This qualitative evaluation data will be supplemented by the performance monitoring to offer a rich picture of the implementation factors that contribute to program success. Core SIPP is a unique program and these data are not available from any other source.

A.5. Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this data collection.

A.6. Consequences of Collecting the Information Less Frequently

Data reported in the Partners Portal web interface will be collected annually. Data will be collected in alignment with annual progress reporting requirements which are due 120 days before the end of the budget period and serves as a non-competing continuation application. Less frequent reporting would undermine accountability efforts at all levels and negatively impact monitoring awardee progress. The annual reporting schedule ensures that CDC responses to inquiries from HHS, the White House, Congress and other stakeholders are based on timely and up-to-date information. The qualitative data that will come from interviews will be collected annually as well, however these data collections will not be associated with the partner’s portal and will most likely be collected at different times in the reporting year. Collecting these data less frequently would also hamper CDC’s ability to provide timely and appropriate technical assistance to recipients.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The request fully complies with the regulation 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A.8.a) Federal Register Notice

A 60-day Federal Register Notice was published in the Federal Register on June 10, 2021, vol. 86, No. 110, pp. 30939 (Attachment B). There were 3 anonymous comments to the 60-day Federal Register Notice (Attachment B1).

A.8.b) Efforts to Consult Outside the Agency

The data collection instruments were designed collaboratively by CDC staff and selected contractors. Consultation will continue throughout the implementation process. As many components of this ICR are based on existing tools; feedback from partners, both internal and external, may have occurred during their implementation in previous funding opportunities.

In addition, listening sessions with partners (ASTHO, Safe States Alliance, CSTE) informed the development of the evaluation design CDC has also conducted listening sessions with funded and unfunded states to further inform evaluation planning and data collection activities from a participatory perspective.

A.9. Explanation of Any Payment or Gift to Respondents

Respondents will not receive payments or gifts for providing information.

A.10. Protection of the Privacy and Confidentiality of Information Provided by Respondents

The Office of the Chief Information Officer at the CDC has determined that the Privacy Act does not apply (Attachment D). No personal contact information will be collected. All data will be reported in aggregate form, with no identifying information included. Because data are maintained in a secure, password protected system, and information will be reported in aggregate form, there is no impact on respondent privacy. Key program staff will provide information related to programmatic improvement and they will be notified that their responses on the electronic information system will be treated in a secure manner. Staff identifiers will not be used in any progress reports. The information collection does not require consent from individuals. All procedures have been developed, in accordance with federal, state, and local guidelines, to ensure that the rights and privacy of key awardees’ program staff (e.g. program director) will be protected and maintained.

While consent is not required to report non-research aggregate data, awardee approval will be obtained if specific state data is used for publications, reports, or other publicly disseminated information. Respondents are state governmental agencies. Although contact information is obtained for each awardee, the contact person provides information about the organization, not personal information. No system of records will be created under the Privacy Act. Submission and access to state data will be controlled by a password-protected login to the secure Partners Portal. Access levels vary from read-only to read-write, based on the user’s role and needs. CDC staff, and evaluation contractors will have varying levels of access to the system with role-appropriate security training, based on the requirements of their position(s). Aggregated information will be stored on an internal CDC Access server subject to CDC’s information security guidelines.

A.11. Institutional Review Board (IRB) and Justification for Sensitive Questions

IRB Approval

The CDC National Center for Injury Prevention and Control (NCIPC)’s OMB and human subject research officer has determined that this collection is non-research and therefore IRB approval is not needed (Attachment C). The information does not involve the collection of personal information or participation of Human Subjects.

Sensitive Questions

There are not sensitive questions in this collection.

A.12. Estimates of Annualized Burden Hours and Costs

Respondents will be the awardees of the Core SIPP cooperative agreement. Awardees will report information to NCIPC about their evaluation performance measures, functional capacity, leveraged funding and resources, and surveillance data. Information collection tools to be used include the functional capacity rubric (Attachment E). Leveraged resources/funds through economic indicators (Attachment F). Challenges and successes, programmatic improvements, and impact through interviews (Attachment G). Excel-based Injury Indicator Spreadsheets (Attachment H, I and J) and Special Emphasis Reports (Attachment K).

Implementation Capacity Development Rubric (Attachment E)

Annually, all 23 respondents will complete the web-based capacity rubric. The annual burden per response is estimated at 2 hours. The implementation capacity rubric was pilot tested by CDC staff from within and beyond the funding program team. Average time estimate was taken to determine burden level. Testing with partners external to CDC was not possibly due to competitive advantage that could have provided in advance of applying for CDC funding.

Economic Indicators (Attachment F)

Annually, all 23 respondents will complete the Excel-based Economic Indicator Spreadsheet. The annual burden per response is estimated at 1 hour. The economic indicators spreadsheet was pilot tested by CDC staff from within and beyond the funding program team using old recipient budget information as examples. Average time estimate was taken to determine burden level. Testing with partners external to CDC was not possibly due to competitive advantage that could have provided in advance of applying for CDC funding.

Qualitative Interviews (Attachment G)

Annually, all 23 recipients will participate in individual interviews. The annual burden per respondent is 1.5 hours.

Injury Indicator Spreadsheets (Attachment H, I, and J)

Annually, all 23 respondents will complete the Excel-based Injury Indicator Spreadsheet. The annual burden per response is estimated at 5 hours. Burden was estimated based on level of effort estimated by recipients for completing Injury Indicator Spreadsheets under OMB# 0920-1120 - Monitoring and Reporting for the Core State Violence and Injury Prevention Program Cooperative Agreement (CORE SVIPP). No changes are made year to year to the Injury Indicator Spreadsheet requirements, so burden is anticipated to remain the same.

Special Emphasis Reports (Attachment K)

Annually, all 23 respondents will complete the Excel and pdf based special emphasis reports. The annual burden per response is estimated at 10 hours.

Burden was estimated based on discussion between CDC subject matter experts and epidemiologists in the field. Special Emphasis Report structure and guidance is developed by the Council of State and Territorial Epidemiologists (CSTE) and has been tested and validated by epidemiologists across the Country.

The total estimated annualized burden for the current awardees is 679 hours, as summarized in Table A.12.1.

Table A.12.1. Estimated Annualized Burden Hours

Type of Respondents	Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in hours)	Total Burden (in hours)
Core SIPP Program Awardees	Implementation Capacity Rubric (Attachment E)	23	1	2	46
	Economic Indicators (Attachment F)	23	1	1	23
	Recipient-level Group Interviews (Attachment G)	23	1	1.5	35
	Injury Indicators Spreadsheet (Attachment H)	23	1	5	115
	Emergency Department Injury Indicators Spreadsheet (Attachment I)	23	1	5	115
	Hospital Discharge Injury Indicators Spreadsheet (Attachment J)	23	1	5	115
	Special Emphasis Reports (Attachment K)	23	1	10	230
Total					679

A.12.b) Annual burden cost

A program manager will prepare the progress reports for each area. The average hourly wage for a program manager is $31. The hourly wage rates for program managers are based on wages for similar mid-to-high level positions in the public sector. The total estimated annualized cost is $21,033.50, as summarized in Table A.12-2.

https://www.bls.gov/oes/current/oes_stru.htm

Table A.12.2. Estimated Annualized Burden Costs

Type of Respondent	Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden per Response	Average Hourly Wage Rate (in dollars)	Total Respondent Cost
Core SIPP Program	Implementation Capacity Development Rubric (Attachment E)	23	1	2	$31	$1426
	Economic Indicators (Attachment F)	23	1	1	$31	$713
	Annual Recipient Interviews (Attachment G)	23	1	1.5	$31	$1069.50
	Injury Indicator Reports (Attachment H)	23	1	5	$31	$3565
	Emergency Department Injury Indicators Spreadsheet (Attachment I)	23	1	5	$31	$3565
	Hospital Discharge Injury Indicators Spreadsheet (Attachment J)	23	1	5	$31	$3565
	Special Emphasis Reports (Attachment I)	23	1	10	$31	$7,130
Total						$21,033.50

A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

This data collection will not result in costs for respondents or record keepers. No capital or maintenance costs are expected. Additionally, there are no start-up, hardware or software costs.

A.14. Annualized Cost to the Government

Table 3. Estimated Annualized Cost to the Government

Type of Cost	Description of Services	Annual Cost
CDC Personnel	100% GS-12@$73,347/year = $73,347 50% GS-13 @ $87,219 /year = $43,610 25% GS-14 @ $103,065/year = $25,766	$142,723
Contractor	Data Collection Contractor	$100,000
Total Annual Estimated Costs		$242,723

A.15. Explanation for Program Changes or Adjustments

This is a new collection.

A.16. Plans for Tabulation and Publication, and Project Time Schedule

A. Time schedule for the entire project

The cooperative agreement cycle is five years. OMB approval is being requested for three years. Per the NOFO, data collection must begin 3 months post award with the Rubric. Other collections will occur per the NOFO requirements once a year due 120 days before the end of the budget period. Data collection began with the awarding of the grants and will continue throughout the funding cycle.

B. Publication plan

Information collected by the awardees will be reported in internal CDC documents and shared with state-based programs. Publication in a peer-reviewed scientific journal will be determined post-data collection.

C. Analysis plan

CDC will use statistical methods for analyzing information. For example, the difference between baseline rates and achieved rates on indicators will be documented and analyzed. Furthermore, the data collected in the mixed methods design will allow for CDC staff to evaluate implementation and provide technical assistance to awardees after an internal qualitative review has been completed.

Project Time Schedule
Activity	Time Schedule
Annual data collection as described in Section A.2 and A.12	Ongoing once annually 1-30 months after OMB approval.
Data cleaning and analysis	Ongoing annually 4-36 months after OMB approval.
Reporting of evaluation data and findings to recipients and stakeholders	Ongoing annually 8-36 months after OMB approval.

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

The display of the OMB expiration date is not inappropriate.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.