The Performance Measures Project: Improving Performance Measurement and Monitoring By CDC Programs
Revision
Part A: Justification
January 5, 2023
Contact: J. Stan Lehman
Telephone: (404) 639-2041
E-mail: [email protected]
Program Performance and Evaluation Office
Office of the Associate Director for Policy and Strategy
Office of the Director
Centers for Disease Prevention and Control
Atlanta, Georgia
TABLE OF CONTENTS
A. Justification
Circumstances Making the Collection of Information Necessary
Purposes and Use of Information Collection
Use of Improved Information Technology and Burden Reduction
Efforts to Identify Duplication and Use of Similar Information
Impact on Small Businesses or Other Small Entities
Consequences of Collecting the Information Less Frequently
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
Explanation of Any payment or Gift to Respondents
Protection of the Privacy and Confidentiality of Information Provided by Respondents
Institutional Review Board (IRB) and Justification for Sensitive Questions
Estimates of Annualized Burden Hours and Costs
Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
Estimates of Annualized Cost to the Federal Government
Explanation for Program Changes or Adjustments
Plans for Tabulation and Publication and Project Time Schedule
Reason(s) Display of OMB Expiration Date is Inappropriate
Exceptions to Certification for Paperwork Reduction Act Submissions
List of Attachments
Authorizing Legislation
Sample Performance Measure Technical Specification Instrument
Sample Performance Measure Reporting Instrument
GenIC Request Template for CDC/ATSDR Programs
60 Day Federal Register Notice
Federal Register Notice Comments
Research Determination (1 of 2)
Research Determination (2 of 2)
GENIC Completion and Carryover Report (includes a list of previously approved GENICs submitted for continuation as part of the Revision ICR)
Rationale for renewal/revision: Although the Performance Measures Project (PMP) was approved in January 2020, the COVID-19 pandemic severely impacted CDC/ATSDR programmatic activities, including implementation of PMP. Currently five CDC/ATSDR programs have received OMB approval to collect performance measure data using the 0920-1282 Generic Information Collection. In addition, seven programs are enrolled in PMP and actively developing applications. As CDC/ATSDR programs begin to normalize operations following the COVID-19 pandemic, numerous other CDC/ATSDR programs have showed strong interest in participating in PMP when: 1) they develop new Notices of Funding Opportunities (NOFO) or; 2) transition current performance measure data collection from the HHS PRA waiver for public health emergency – Coronavirus Disease 2019 [COVID-19] to the PMP GenIC for ongoing performance data collection. This revision is requested to allow participating CDC/ATSDR programs to continue performance measure data collection through the remaining approval period and for additional programs to use the GenIC for future performance measure data collection.
Goal of the project: To provide a flexible and timely approval framework that facilitates the collection of information needed to measure and improve the performance of recipients of CDC/ATSDR funding. Through participation in the Performance Measures Project (PMP), CDC/ATSDR programs and recipients will: 1) Develop strong performance measurement tools and practices; 2) define and operationalize priority performance measures tailored to a specific public health initiative; and 3) establish common data collection and reporting expectations across all recipients. |
Intended use of the resulting data: The intended use of the resulting data is to generate general purpose statistics, to assess program progress (at both the local recipient-specific and CDC/ATSDR program-wide levels), to provide data that can inform program evaluation, to assist with program planning/management, and to improve the accountability of federal funds provided to recipients. |
Methods to be used to collect: Participating CDC/ATSDR programs will provide the program-specific information collection forms to each recipient to facilitate standardized performance data collection at the local level. Recipients will report these standardized performance data directly to CDC/ATSDR programs on a periodic basis to assess progress toward achieving desired program outcomes as defined by the program. The default periodic basis is annual reporting, but programs may propose alternate reporting schedules if needed.
|
The subpopulation to be studied: Respondents will be recipients of CDC/ATSDR funding. CDC’s Performance Measures Project (PMP) will work with up to 40 new CDC programs. The number of recipients per CDC/ATSDR program will vary dependent on program activity, and the expected average is approximately 35 recipients per program.
|
How the data will be analyzed: Data will be analyzed by the CDC/ATSDR program to assess recipient progress toward the key outcomes defined by the public health initiative. The information collected is meant primarily for program improvement and accountability; it is not intended to be used as the principal basis for policy decisions. |
1. Circumstances Making the Collection of Information Necessary
Each year, approximately 75% of the CDC/ATSDR’s congressionally appropriated funding goes to extramural organizations, including state and local partners, via contracts, grants, and, most commonly, cooperative agreements. The availability of funding for grants and cooperative agreements is announced through a Notice of Funding Opportunity (NOFO). CDC awards up to 100 new, non-research NOFOs each year (each funded for 1 to 5 years). These awards may have only a few funded recipients or more than 50 (such as when a CDC/ATSDR program provides funding to all states and territories). Monitoring and reporting of program performance is required of any non-federal entity receiving federal funds under 45 CFR 75.342 (Attachment 1); “The Non-Federal entity must monitor its activities under Federal awards to assure compliance with applicable Federal requirements and performance expectations are being achieved”.
CDC’s Program Performance and Evaluation Office (PPEO) provides technical assistance to CDC programs and external partners to improve the performance of CDC/ATSDR-funded public health initiatives. An important PPEO activity is providing technical assistance to CDC/ATSDR programs as they develop Notice of Funding Opportunities (NOFOs) for cooperative agreements and grants to award federal funds (e.g., to state and local recipients) to implement public health initiatives. An important element of program implementation is the development of program performance measures and the infrastructure for collecting these data by local recipients and reporting these data to CDC/ATSDR. The Performance Measures Project (PMP) works with selected CDC/ATSDR programs, providing tools, templates, and technical assistance to develop and implement priority performance measures that funded recipients report to CDC/ATSDR.
CDC/ATSDR programs develop logic models for each NOFO, describing the key programmatic strategies and activities and the short/medium/long-term outcomes funded recipients are expected to achieve during their period of performance. Programs develop performance measures customized to a NOFO-specific public health initiative to assess actions prescribed by the logic model with the immediate goal of monitoring progress and the long-term goal of improving performance.
Under this requested revision, CDC/ATSDR programs may customize the sample “Performance Measure Technical Specification Instrument” (Attachment 2) and the sample “Performance Measure Reporting Instrument” (Attachment 3) to measure, at the local level, the desired public health outcomes of a particular public health initiative, in compliance with the Paperwork Reduction Act (PRA). Individual collection requests submitted under this generic approval will include the tailored forms and a supplementary template (Attachment 4).
The sample Performance Measure Technical Specification Instrument provides standardized technical specifications for operationalizing performance measures. The CDC/ATSDR program, in collaboration with the funded recipients will document these technical specifications for each performance measure developed for a particular public health initiative. The sample Performance Measure Reporting Instrument is what the CDC/ATSDR programs will adapt and use for all recipients to report the performance data in a standardized way.
Since the initial generic information collection was approved in January 2020, multiple CDC/ATSDR programs have used this mechanism to improve and routinize collection and reporting of key performance data. Initially, participation in the PMP was limited to CDC/ATSDR programs developing new, domestic, non-research NOFOs funded through the cooperative agreement mechanism. In this Revision, CDC/ATSDR will expand eligibility to include international programs and programs funded under grants or contracts. CDC/ATSDR will continue to use the PMP generic framework for information collection that is uncontroversial, low burden, and provides a significant benefit to recipients. Each information collection activity will be submitted to OMB for review and approval as a new “GENIC”. OMB approval of the generic clearance is requested for three years.
2. Purpose and Use of the Information Collection
Information collection instruments that are customized to reflect program-specific performance measures have allowed CDC/ATSDR programs to collect data for a particular cooperative agreement in a uniform and systematic manner across all recipients, provide a single reporting format, and have reduced the burden on individual funded recipients. Standardized collection and reporting of performance measures results in higher quality performance data and allows CDC/ATSDR to monitor how individual recipients are progressing toward implementing the activities and achieving their outcomes and allows CDC/ATSDR to more easily aggregate data across all funded organizations. These data are used by CDC/ATSDR to monitor trends for each performance measure over the life of the public health initiative and help show program’s successes and challenges, make course corrections as needed, and identify technical assistance needs.
This generic clearance also allows the funded recipients to efficiently report their performance data using an instrument customized to a particular cooperative agreement. Having a standardized instrument makes it easier for them to report data consistently and accurately across reporting periods. The staff from funded recipient organizations are able to more easily enter data and review for completeness, enter basic summary data for reports, and save required data for use with other reporting systems.
3. Use of Improved Information Technology and Burden Reduction
CDC/ATSDR Programs and funded recipients will use some form of electronic data collection. Building on existing data infrastructure, this will be either a web page, email or On-line Data Collection Systems, such as Sharepoint, RedCap, AMP etc..
4. Efforts to Identify Duplication and Use of Similar Information
The information collected from funded recipients is not available from other sources in a consistent format.
5. Impact on Small Businesses or Other Small Entities
No small businesses will be involved in this data collection, unless contracted by local funding recipients.
6. Consequences of Collecting the Information Less Frequently
Performance measure data will be collected at least annually in accordance with program guidance and relevant public health initiative award terms and conditions. Less frequent reporting would undermine accountability efforts at all levels and negatively impact monitoring recipient progress. The periodic reporting schedule ensures that CDC/ATSDR responses to inquiries from HHS, Congress and other stakeholders are based on timely and up-to-date information.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This request fully complies with the regulation 5 CFR 1320.5.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A 60 Day Notice was published in the Federal Register on July 25, 2022; volume 87, number 141, page 44121 (Attachment 5).
No public comments were received in response to publication of the 60-day FRN (Attachment 6). No changes were made to the information collection plan.
CDC consulted with The Administration for Children and Families (ACF) on development of the first iteration of the Performance Measures project and the current ACF Generic (OMB Control Number: 0970-0490) informed revisions to the PPEO Generic.
9. Explanation of Any Payment or Gift to Respondents
Respondents will not receive payments or gifts for providing information.
10. Protection of the Privacy and Confidentiality of Information Provided by Respondents
Staff from the CDC/ATSDR Office of Science have reviewed this Information Collection Request and have determined that the Privacy Act is not applicable (Attachment 7, Attachment 8). The data collection does not involve collection of sensitive or identifiable personal information. Although contact information is obtained for each funded recipient (state or local jurisdiction), the contact person provides information about the organization, not personal information. No system of records will be created under the Privacy Act.
Recipients are required to provide data as a condition of cooperative agreement funding and will submit data via secure mechanism (e.g., email, password protected website, Sharepoint).
While consent is not required to report aggregate data, recipient consent will be obtained if their specific data are used for publications, reports or other publicly disseminated information.
Aggregated information will be stored on an internal CDC SQL server subject to CDC/ATSDR’s information security guidelines. CDC/ATSDR staff, technical assistance, and training contractors will have varying levels of access to the system with role-appropriate security training, based on the requirements of their position(s).
11. Institutional Review Board (IRB) and Justification for Sensitive Questions
The proposed generic clearance does not collect sensitive information.
12. Estimates of Annualized Burden Hours and Costs
A. Estimated Annualized Burden Hours
The burden request for this generic is based on estimates for two categories of GENICs.
New GENICs. CDC estimates that up to 40 CDC funded programs will be phased in over the next three-year project period. There will be an approximate average of 35 awards per participating cooperative agreement, grant, or contract, and up to 875 recipients (across all participating programs). For purposes of burden estimation, CDC is estimating biannual reporting, although some programs may report once per year (875 recipients x 2 responses/year = 1,750 responses per year). The estimated burden per response is 40 hours. The annualized request for new GENICs is 1,750 responses and 70,000 burden hours.
Continuation of previously approved GENICs (i.e., previously approved funding programs and information collection instruments). During the initial approval period for generic 0920-1282, information collection was initiated for 5 CDC/ATSDR programs. These are multi-year awards and there is a need to continue performance monitoring for 3 years. Due to the use of the 0920-1282 generic for cross-NOFO collaborations, the number of responses and average burden per response vary substantially and in some cases exceed the estimates provided above for typical new GENICs. CDC/ATSDR is requesting 27,049 annualized burden hours and 2,192 annualized responses for these activities.
Overall, CDC is requesting 3,942 annualized responses and 97,049 annualized burden hours.
OMB approval is requested for 3 years. Over this period, CDC is requesting 11,826 responses and 291,147 burden hours.
Attachment 9 provides breakdowns for the methodology used to calculate these estimates. CDC/PPEO will monitor capacity and usage, and as needed may request adjustments through the Change Request mechanism.
Previously approved GENICs are submitted with this Revision ICR and their continuation will be effective upon OMB approval of the Revision. New information collection activities will be submitted individually to OMB as additional GENICs. In all cases, information collection instruments will be based on PMP templates and tailored to the specific needs of the participating CDC/ATSDR program.
Table A.12-A. Estimated Annualized Burden to Respondents
Type of Respondents |
Form Name |
Number of responses |
Number of responses per respondent per year |
Average burden per response (in hours) |
|
CDC/ATSDR Award Recipients (new GENICs) |
Performance Measures Project Information Collection Tool |
1,750 |
1 |
40 |
70,000 |
CDC/ATSDR Award Recipients (continuation of previously approved GENICs) |
Performance Measures Project Information Collection Tool |
2,192 |
1 |
740.4/60 |
27,049 |
Total |
|
3,942 |
|
|
97,049 |
B. Estimated Annualized Cost to Respondents
Estimates for the average hourly wage for respondents are based on the U.S. Department of Labor Bureau (DOL) of Labor Statistics May 2014 National Occupational Employment and Wage Estimates (http://www.bls.gov/oes/current/oes_nat.htm). Based on DOL data, the average hourly wage for an epidemiologist is estimated to be $37.37. The total estimated annualized cost is $3,626,721 as summarized in Table A.12-B.
Table A.12-B. Estimated Annualized Cost to Respondents
Type of Respondents |
Total burden (in Hours) |
Average Hourly Wage |
Total Cost |
CDC/ATSDR Award Recipients |
97,049 |
$37.37 |
$3,626,721 |
13. Estimates of Other Total Annual Cost Burden to Respondents or Record
Keepers
No capital or maintenance costs are expected.
14. Estimates of Annualized Cost to the Federal Government
The cost to the Federal Government is approximately $117,154; 50% of a GS-14 step 5 epidemiologist and 50% of a GS-12 step 5 public health analyst).
15. Explanation for Program Changes or Adjustments
The scope of this revision reflects expanded technical assistance that the Program Performance and Evaluation Office (PPEO) provides to CDC/ATSDR programs and CDC/ATSDR program eligibility to participate in PMP will be expanded as follows:
Given the recent increase in grants and other funding mechanisms used at CDC/ATSDR to enhance programmatic flexibility, PMP eligibility will expand to include all available funding mechanisms for eligible programs.
PPEO is providing increasing technical assistance to international programs. Eligibility will expand to include both domestic and international programs.
Many CDC/ATSDR programs are operating under the HHS COVID-19 Emergency PRA waiver. This emergency waiver is expected to be discontinued at some point. PMP will prioritize transitioning CDC/ATSDR program performance measure data collection from the Emergency Waiver to PMP.
Some CDC/ATSDR programs are developing common performance metrics across multiple public health initiatives. PMP will prioritize cross-NOFO collaboration with these programs to increase efficiency.
As programs transition back to normal function after the Covid 19 Pandemic, there is substantially increased interest in PMP. With this in mind PPEO will increase the estimated number of new programs that may participate from 25 Programs to 40 Programs. In addition, PPEO’s burden estimate now includes an allocation for continuation of previously approved, multi-year information collections. resulting in an increase of estimated annualized burden from 35,000 hours to 97,049 hours.
CDC proposes changes to the GENIC Request Template (Attachment 4) that clarify the calendar years in which each program’s customized templates will be administered, and total burden hours for the entire period of information collection. The template will adopt the standard burden table format utilized throughout CDC/ATSDR which provides greater clarity with respect to the frequency of information collection (annual, semi-annual, quarterly, etc.). These changes will improve recordkeeping for the 0920-1282 generic and improve PPEO’s ability to monitor capacity and usage of the generic, while also providing increased flexibility for CDC/ATSDR programs to describe their data collection plans.
16. Plans for Tabulation and Publication and Project Time Schedule
A. Time schedule for the entire project
OMB approval is requested for three years. CDC/ATSDR programs will begin participating by the end of 2019. Each CDC/ATSDR program developing a new public health initiative will submit a generic information collection for use by all funded recipients. Reports will be generated by the recipients per the public health initiative requirements. Data collection will begin when funding is awarded and will continue throughout the funding cycle.
B. Publication plan
Information collected by the funded organizations will be reported in internal CDC/ATSDR documents and shared with recipients. Summary data will be provided for inquiries from HHS, Congress and/or other stakeholders. Individual programs that are planning other types of publications will include that information in their specific request.
C. Analysis plan
CDC/ATSDR will not use complex statistical methods for analyzing information. All information will be aggregated for each public health initiative and reported with no personal identifying information in external documents. This is intended primarily for program improvement and accountability; data will not be used to inform or influence policy decisions or budgetary justifications.
Most statistical analyses will be descriptive and will vary by individual cooperative agreement need.
17. Reason(s) Display of OMB Expiration Date is Inappropriate
The Sample Performance Measure Instruments (Attachment 2, Attachment 3) will display the expiration date for OMB approval. All collections under this generic will also include the OMB expiration date.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification statement.
-
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OFFICE OF MANAGEMENT AND BUDGET |
| Author | rst8 |
| File Modified | 0000-00-00 |
| File Created | 2024-07-22 |