Supporting Statement A

Assessment of Performance Management and Improvement Practices in States, Tribes, Locals and Territories

OSTLTS Generic Information Collection Request

OMB No. 0920-0879

Supporting Statement – Section A

Submitted: October 3, 2012

Program Official/Project Officer

Anita McLees

Health Scientist

Applied Systems Research and Evaluation Branch

Division of Public Health Performance Improvement

Office of State, Tribal, Local and Territorial Support

Centers for Disease Control

1825 Century Center MS E-70

Tel: 404-498-0316

Email: [email protected]

Section A – Justification

1. Circumstances Making the Collection of Information Necessary

Background

This data collection is being conducted using the Generic Information Collection mechanism of the OSTLTS Survey Center (OSC) – OMB No. 0920-0879. The respondent universe for this data collection aligns with that of the OSC. Data will be collected from Performance Improvement Managers acting in their official capacities at State, Tribal, Local, or Territorial (STLT) health agencies. This data collection is authorized by Section 301 of the Public Health Service Act (42 U.S.C. 241).

The Centers for Disease Control and Prevention’s (CDC) National Public Health Improvement Initiative (NPHII) is a five-year Cooperative Agreement funded through the Prevention and Public Health Fund of the Affordable Care Act. NPHII is designed to strengthen the nation’s health by optimizing agency resource utilization and improving program performance and quality of program services. For the first year of NPHII (FY2010), CDC’s Office for State, Tribal, Local and Territorial Support (OSTLTS) awarded $42.5 million to seventy-six STLT health agencies. All 76 STLT awardees received funds to hire a Performance Improvement Manager and initiate performance management activities. Of the 76 STLT awardees, nineteen received larger awards to focus on one or more of four key areas: health promotion and disease prevention, public health policy and public health law, health IT and communications infrastructure, and workforce and systems development (Attachment A – Original NPHII Funding Opportunity Announcement). In FY2011, OSTLTS awarded $33.5M dollars to seventy-four of these original 76 STLT agencies for the second year of NPHII, including 48 state health departments and the District of Columbia, 9 local health departments, 8 territory and Pacific Island health departments, and 8 American Indian/Alaska Native Tribes/Organizations. South Dakota and Virgin Islands did not reapply for funding for the second year of NPHII (Attachment B – Map and Listing of FY 2011 NPHII Awardees). For this second year, NPHII focused more specifically on promoting performance management, quality improvement, and accreditation readiness activities to advance the goal of improving the efficiency and effectiveness of services and programs in STLT awardee organizations. In September 2012 (FY2012), an additional $33.5M dollars was awarded to seventy-three STLT agencies to continue work in these same areas. South East Alaska Regional Health Consortium decided not to apply for FY 2012 NPHII funding, and will not be included in the respondent population for the Year 3 data collection (Attachment C - Continuation guidance for Year 2 of NPHII, Attachment D - Continuation guidance for Year 3 of NPHII).

As referenced above, NPHII seeks to optimize the value of America’s investment in public health by systematically increasing the performance management capacity of public health departments and improving their ability to meet public health goals. Its focus on performance management and quality improvement is directed at STLT public health agencies dedicated to advancing the efficiency and effectiveness of services and programs.  Increasing the effectiveness and efficiencies of these STLT agencies is intended to mitigate the impact of decreases in the public health workforce over the past several years and thereby introduce practices to help keep Americans healthy and productive.

Though NPHII is a five-year cooperative agreement, the evolving nature of the public health environment and its impact on changes to the cooperative agreement requirements have led to the fine-tuning of NPHII’s strategies after the first year of the program. The shift between Year One and Year Two of NPHII towards a greater emphasis on accreditation readiness, performance management, and quality improvement as mechanisms to achieve greater efficiency and effectiveness of services and programs is reflective of the launch of a voluntary national public health accreditation program in 2011 by the Public Health Accreditation Board (PHAB) (Attachment E – PHAB Standards and Measures) and an increased understanding of the benefits of quality improvement on the efficiency and effectiveness of STLTs’ daily operations. Years 2 and 3 Continuation Guidance documents for NPHII (Attachments C & D) reflect this refinement in their primary objectives:

• Increase STLTs’ readiness to achieve voluntary public health accreditation through the PHAB

• Improve STLT health agencies’ ability to more efficiently and effectively implement programs and deliver services through implementation of performance management and quality improvement activities

To evaluate progress made towards intended outcomes of NPHII, CDC funded the National Network of Public Health Institutes (NNPHI) through a Cooperative Agreement to collaborate with CDC to implement a formative evaluation of NPHII. As the concentration of the NPHII program was clarified in Years 2 and 3 per the description above, the evaluation questions and approaches were also refined to reflect the more specific focus and requirements of the NPHII program. NNPHI worked with their expert consultants and CDC to revise the NPHII logic model (Attachment F – Logic Model for NPHII Evaluation) and evaluation questions (Attachment G – NPHII Evaluation Questions) to examine the NPHII investment and its initial impact on:

• Efficiency and effectiveness of processes, services, and/or programs

• Organizational foundation, including the development of the capacity to systematically conduct performance management and quality improvement activities and the development of a culture or environment that supports this type of work

• Accreditation readiness, including completion of key pre-requisites and addressing gaps associated with national public health standards developed by the PHAB

• Progress towards achieving NPHII Cooperative Agreement goals as outlined in STLTs’ work plans

The NPHII Annual Assessment of Performance Management and Improvement Practices (hereafter referred to as NPHII Annual Assessment) is intended to capture STLTs progress towards achieving NPHII goals in the areas of accreditation readiness, performance management and quality improvement, and the extent to which their STLT agency’s environment is conducive to achieving these goals. The NPHII Annual Assessment data collection tool will be used to collect information from all 74 NPHII STLTs as to their activities in these key areas of the Cooperative Agreement during the second year (September 30, 2011 – September 29, 2012) and from the 73 NPHII-funded STLTs during the third year of funding (September 30, 2012-September 29, 2013).

Privacy Impact Assessment

Overview of the Data Collection System – The data collection system to be used for both assessments consists of a web-based questionnaire (Attachment H – NPHII Assessment Screen Shots; Attachment I –NPHII Assessment Word Version) to be administered to all 74 NPHII STLTs (Year 2 NPHII activities) and 73 NPHII STLTs (Year 3 NPHII activities). The primary respondent for the assessment is the STLT’s Performance Improvement Manager (PIM). Each STLT participating in NPHII was expected to hire or appoint a PIM as a requirement of funding in order to oversee and manage performance management / quality improvement activities across their agency. If the PIM is unable to respond to the data collection tool, the NPHII principal investigator is to respond. As mentioned above, these individuals will report progress that their agency has made towards achieving NPHII goals in the areas of accreditation readiness, performance management, and quality improvement. The data collection instrument will be administered as a web-based tool, with a PDF copy of the instrument made available to STLTs prior to the data collection period. The data collection tool was pilot tested by three public health professionals. Feedback from this group was used to refine questions as needed, ensure accurate programming and skip patterns and establish the estimated time required to complete the data collection tool.

The data collection tool will collect information from all 74 NPHII STLTs as to their activities in these key areas of the Cooperative Agreement during the second year (September 30, 2011 – September 29, 2012) and from 73 NPHII STLTs during the third year of funding (September 30, 2012-September 29, 2013).

Items of Information to be Collected – The data collection tool consists of 66 questions of various types including dichotomous, multiple response, interval, filter and open-ended questions. The data collection tool is organized into four sections.

1. PIM qualifications – respondents are asked about their current job position, their level of experience in public health and quality improvement, and their level of proficiency on competencies in areas of quality improvement and performance management

2. Accreditation readiness – respondents are asked to identify activities they have undertaken to build awareness of, or readiness for, accreditation within their organization

3. Performance management systems and quality improvement – respondents are asked about their organization’s capacity for performance management and quality improvement in the following areas:

   1. Establishment of the components of a performance management system (performance standards, performance measures, routine performance reporting and quality improvement), as well as barriers to organization-wide performance management

   2. Quality improvement methods, tools and resources used by PIMs; percent of the organization’s staff trained in quality improvement

   3. Quality improvement initiatives focused on gains in efficiency and effectiveness

4. Environment for quality improvement and performance management – includes items from a Quality Improvement Maturity Tool developed for the Multi-State Learning Collaborative (a collaborative project between the Robert Wood Johnson Foundation and NNPHI) which assess the extent to which an environment is conducive to the implementation of quality improvement. This section includes items about leadership and staff training in quality improvement techniques, as well as the nature of decision-making within the organization.

Identification of Website(s) and Website Content Directed at Children Under 13 Years of Age – The data collection system involves using a web-based data collection tool. Respondents will be sent a link directing them to the online data collection tool only (i.e., not a website). No website content will be directed at children.

2. Purpose and Use of the Information Collection

The purpose of the NPHII Assessment (Years 2 & 3) is to capture progress made by the 74 and 73 STLTs participating in NPHII program respectively in the areas of accreditation readiness, performance management and quality improvement. Accreditation readiness activities have the potential to improve the performance of public health systems as health departments/organizations implement strategies to address gaps associated with the national public health standards. Development of the capacity for performance management and quality improvement in STLT health departments/organizations is expected to yield gains in efficiency and effectiveness of business- and service-related operations.

The assessment data will be used for the following purposes:

• Technical Assistance - The findings will help identify facilitators and barriers to achieving program goals, and will inform the development of technical assistance tools and resources.

• Program recommendations - Findings from the annual assessment will be used to understand the successes and areas for further improvement of the program and generate recommendations for how NPHII can be improved.

• Report Deliverables - Findings will be shared with various audiences through the following deliverables:

• Individual grantee reports – these reports will include analysis of individual STLT progress over time and/or a comparison of their progress to all other STLTs. These reports will be shared with the individual STLT only (i.e., the Alabama health department will receive a report that includes only data on its progress and/or a comparison of its responses to other STLTS in aggregate).

• Lunch and learn – An hour-long session for OSTLTS staff, to include a presentation on aggregate findings as well as a question and answer session

• Brief reports – A series of brief one to two page reports focusing on specific sections of the assessment (accreditation readiness, efficiency and effectiveness, and STLT environment for performance management and QI). These reports are intended to inform stakeholders of the progress made by NPHII STLTs.

• Summary report – mid-length report that will highlight key findings from the assessment as well as programmatic implications and future evaluation opportunities

The key stakeholders for the evaluation are Congress, the U.S. Department of Health and Human Services, CDC, national partners, and the NPHII STLTs.

Privacy Impact Assessment: No sensitive information is being collected. No individually identifiable information is being collected. The proposed data collection will have little or no effect on respondent privacy as no names or other personally identifiable information will be collected via the assessment. Respondents are participating in their official capacity as Performance Improvement Managers in state, tribal, local and territorial departments of health.

3. Use of Improved Information Technology and Burden Reduction

Data will be collected via a web-based tool allowing respondents to complete and submit their responses electronically. The web-based tool allows respondents to either complete the assessment in one sitting or to save and return to it at a later time. This method was chosen to reduce the overall burden on respondents. The data collection tool was designed to collect the minimum information necessary for the purposes of this project (i.e., limited to 66 questions with appropriate skip patterns). Screen shots of the web-based data collection tool can be found in (Attachment H – NPHII Assessment Screen Shots).

4. Efforts to Identify Duplication and Use of Similar Information

This data collection tool is intended to measure progress made by NPHII-funded STLTs in the areas of accreditation readiness, performance management and quality improvement. There is no information available that can substitute for this data collection as this universe of STLTs has not reported data on these activities for this particular time period through other mechanisms. The NPHII assessment captures more standardized, outcome-based data than is available in the interim and annual progress reporting, which focus more on administrative and process-oriented information. Additionally, NNPHI and CDC staff cross-walked the NPHII data collection tool with the Association of State and Territorial Health Officials (ASTHO) and the National Association of County and City Health Officials (NACCHO) profile surveys to ensure that there was no duplication in data items requested of respondents for the given timeframes.

5. Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this data collection.

6. Consequences of Collecting the Information Less Frequently

This request is for two instances of data collection – the assessment at the end of Year Two of NPHII (covering the period September 30, 2011 – September 29, 2012), and the same assessment repeated at the end of Year Three of NPHII (covering the period September 30, 2012 – September 29, 2013). There are no legal obstacles to reduce the burden.

The consequences to the program of not collecting this information under this mechanism and within these timeframes are as follows:

• Inability to provide performance measure data for Affordable Care Act / Prevention and Public Health Fund reporting on NPHII, for which annual updates on relevant measures are expected

• Inability to assess STLT progress on key program outcomes and obtain timely data to inform the provision of assistance to STLTs moving forward

• Inability to inform key evaluation questions and adequately assess the initial impact of NPHII

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances with this information collection package. This request fully complies with the regulation 5 CFR 1320.5 and will be voluntary.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

This data collection is being conducted using the Generic Information Collection mechanism of the OSTLTS Survey Center (OSC) – OMB No. 0920-0879. A 60-day Federal Register Notice was published in the Federal Register on October 22, 2010, Vol. 75, No. 204; pp. 65353-54. Two comments were received from the Association of State and Territorial Health Officials (ASTHO), and the National Association of County and City Health Officials (NACCHO).

CDC partners with professional STLT organizations, such as the Association of State and Territorial Health Officials (ASTHO), the National Association of County and City Health Officials (NACCHO), and the National Association of Local Boards of Health (NALBOH) along with the National Center for Health Statistics (NCHS) to ensure that the collection requests under individual ICs are not in conflict with collections they have or will have in the field within the same timeframe.

9. Explanation of Any Payment or Gift to Respondents

CDC will not provide payments or gifts to respondents.

10. Assurance of Confidentiality Provided to Respondents

The Privacy Act does not apply to this data collection. Employees of state and local public health agencies will be speaking from their official roles and will not be asked, nor will they provide individually identifiable information.

This data collection is not research involving human subjects.

11. Justification for Sensitive Questions

No information will be collected that are of personal or sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

The estimate for burden hours is based on a pilot test of the data collection tool by three public health professionals. In the pilot test, the average time to complete the data collection tool including time for reviewing instructions, gathering needed information and completing the data collection tool, was approximately 22 minutes. Based on these results, the estimated time range for actual respondents to complete the data collection tool is 20-25 minutes. For the purposes of estimating burden hours, the upper limit of this range (i.e., 25 minutes) is used.

Estimates for the average hourly wage for respondents are based on the Department of Labor (DOL) National Compensation Survey estimate for management occupations – medical and health services managers in state government (http://www.bls.gov/ncs/ocs/sp/nctb1349.pdf). Based on DOL data, an average hourly wage of $47.49 is estimated for all 74 respondents. Table A-12 shows estimated burden and cost information.

The total burden hours for two annual administrations of the NPHII assessment is 61 hours.

Table A-12: Estimated Annualized Burden Hours and Costs to Respondents –NPHII Annual Assessment

Type of Respondent	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
NPHII Assessment STLT Performance Improvement Managers*	74	1	25/60	31	$47.49	$1472.19
TOTALS				31		$1472.19

* South East Alaska Regional Health Consortium decided not to apply for FY 2012 NPHII funding, and will not be included in the respondent population for the Year 3 data collection. Therefore, the number of respondents in Year 3 data collection will be 73 and a request of 30 burden hours.

13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There will be no direct costs to the respondents other than their time to participate in each survey.

14. Annualized Cost to the Government

There are no equipment or overhead costs. Contractors are not being used to support this data collection. The only cost to the federal government would be the salary of CDC staff supporting the data collection activities and associated tasks.

Table A-14: Estimated Annualized Cost to the Federal Government

Staff (FTE)	Average Hours per Collection	Average Hourly Rate	Average Cost
Health Scientist (GS-14) Assisting with instrument development, OMB package preparation, data quality assurance, data analysis and report preparation	250	$54.87	$13,717.50
ORISE fellow Assisting with instrument development, OMB package preparation, data quality assurance, data analysis and report preparation	250	$23.55	$5,887.50
Cooperative Agreement Partner NNPHI Instrument development, pilot testing, OMB package preparation, data collection, data coding and entry, quality control, data analysis, and report preparation	1248	$75.00	$93,600.00
Estimated Total Cost of Information Collection for Year 2 NPHII Assess.			$113,205.00
Estimated Total Cost of Information Collection for Year 3 NPHII Assess			$113,205.00

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

The results will be used internally to set priorities for NPHII and externally to communicate results with STLT partners through the following deliverables:

• Individual grantee reports – these reports will include analysis of individual STLT progress over time and/or a comparison of their progress to all other STLTs. These reports will be shared with the individual STLT only (i.e., the Alabama health department will receive a report that includes only data on its progress and/or a comparison of its responses to other STLTS in aggregate).

• Lunch and learn – An hour-long session for OSTLTS staff, to include a presentation on aggregate findings as well as a question and answer session

• Brief reports – A series of brief one to two page reports focusing on specific sections of the assessment (accreditation readiness, efficiency and effectiveness, and STLT environment for performance management and QI). These reports are intended to inform stakeholders of the progress made by NPHII STLTs.

• Summary report – mid-length report that will highlight key findings from the assessment as well as programmatic implications and future evaluation opportunities

Additionally, opportunities for publication of the NPHII findings will be explored including conference presentations and publications in peer-reviewed journals.

Both quantitative and qualitative analyses will be performed. Quantitative analyses will involve using descriptive statistics to determine frequency distributions and corresponding variances for responses to each survey question. Qualitative thematic analyses will be conducted on open-ended questions. The findings will be used to describe the NPHII program and inform programmatic recommendations as well as additional evaluation opportunities (Attachment G – NPHII Evaluation Questions).

Project Time Schedule

Year 2 NPHII Assessment (October 2012 – April 2013)

Design survey tool………………………………………………….. (COMPLETE)

Develop protocol, instructions, and analysis plan…….. (COMPLETE)

Pilot test survey questionnaire ……………………………….. (COMPLETE)

Prepare OMB package…………………………………………….. (COMPLETE)

Submit OMB package……………………………………………… (COMPLETE)

OMB approval………………………………………………………... (Tentative - 10/14/12)

Conduct data collection …………………………………………. (Tentative - 11/5/12 – 12/5/12)

Collect, code, enter, quality control, and analyze data.. (Tentative – 1/31/13)

Prepare report………………………………………………………. (Tentative – 3/31/13)

Disseminate results/reports…………………………………... (Tentative – 4/25/13)

Year 3 NPHII Assessment (October 2013 – April 2014)

Conduct data collection …………………………………………. (Tentative - 11/5/13 – 12/5/13)

Collect, code, enter, quality control, and analyze data.. (Tentative – 1/31/14)

Prepare report………………………………………………………. (Tentative – 3/31/14)

Disseminate results/reports…………………………………... (Tentative – 4/25/14)

17. Reason(s) Display of OMB Expiration Date is Inappropriate

We are requesting no exemption.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.

LIST OF ATTACHMENTS – Section A

Note: Attachments are included as separate files as instructed.

1. Original NPHII Funding Opportunity Announcement

2. Map and Listing of FY 2011 NPHII Awardees

3. Continuation guidance for Year 2 of NPHII

4. Continuation guidance for Year 3 of NPHII

5. PHAB Standards and Measures (version 1.0)

6. Logic model for NPHII Evaluation

7. NPHII evaluation questions

8. NPHII Assessment – Screen shots

9. NPHII Assessment – Word version