Revised ORA SmartForms Pilot Updated SS Part B final 11-21-22

United States Food and Drug Administration

Pilot to Develop Standardized Reporting Forms for Federally Funded

Public Health Projects and Agreements

OMB Control No. or 0910-[NEW]

SUPPORTING STATEMENT

Part B. Statistical Methods

1. Respondent Universe and Sampling Methods

The universe for this collection includes State and local government units and professional supporting associations for State and local regulatory programs with oversight of FDA regulated products. The total number of unique entities for this collection is estimated to be 135, based on the number of registered entities who have, or have ever received, funding from the Office of Regulatory Affairs (ORA) for a public health project.

This population is considered to be relatively static as evidenced by the same entities remaining enrolled in the same ORA funded programs since the initiation of the program. The commodities and production types across the U.S. and its territories tend not to change and so the regulatory programs administered by State and local governments charged with protecting public health for these products, and subsequently their participation in specific ORA funded public health projects by commodity or product also tends to see little variation.

The table below represents the current enrollment for existing programs covered by this collection.

Program	Potential Respondents*	Applicable Forms
Animal Food Contract	32	Animal Food Contract Quarterly Summary Report
		Veterinary Medicine Course Preregistration Workbook
		Animal Food Safety Inspection Audit Form
		Corrective Action Plan for Program and Individual Performance Deficiencies
		State Implementation Agreement and Year End Evaluation
		Request for Audit Reduction
Animal Food Regulatory Program Standards	25	Animal Feed Regulatory Standards (AFRPS) Program Report
		Veterinary Medicine Course Preregistration Workbook
Egg Contract	6	Egg Contract Quarterly Summary Report
Egg Standards	2	General Program Report Form (non-specific for new cooperative agreement and grant programs)
Flexible Funding Model	44	Flexible Funding Model (FFM) Program Report
		Food Protection Task Force (FPTF) Program Report
		Manufactured Food Course Preregistration Workbook
		Emergency Response Course Preregistration Workbook
Human Food Contract	48	Human Food Contract Quarterly Summary Report
		FDA 3610 Field Inspection Audit
		Corrective Action Plan for Program and Individual Performance Deficiencies
		State Implementation Agreement and Year End Evaluation
		Request for Audit Reduction
Laboratory Flexible Funding Model	55	Laboratory Flexible Funding Model (LFFM) Program Report
		LFFM Sample and Activity Plan_Proposal Template
		LFFM_SRP-Lab Agreement Template_HAF Tracks
		LFFM QTR Chem_LFFM_HAF_Results_Sheet
		LFFM QTR Data Template Micro HAF Product Testing
Mammography Quality Standards Act (MQSA) Contract	41	MQSA MEU and Spending Update Report
Medical Device Contract	2	Medical Devices Contract Quarterly Summary Report
Produce Implementation	56	Produce Program Report
		Produce CAP_Project Plan outline
		Produce CAP Assessment Template
		Produce Course Preregistration Workbook
		Produce Educational Needs Assessment Submission Template
		Produce Instructions Inspection Aggregate Data
		Produce Inspection Aggregate Data Workbook
		Produce Instructions Inventory & Education Aggregate Data
		Produce Inventory & Education Aggregate Data workbook
Scientific Conference Grant	19	Scientific Conference Program Report

* Please note that a majority of enrollees participate in multiple programs accounting for a total of 330 responses across all programs from 135 unique entities.

Initial discussions and feedback from potential respondents regarding this pilot indicate ORA may expect a strong majority response rate. Factors informing that expectation include that program participants want ORA to have information regarding the effectiveness of their state and program specific effort to protect public health and document their alignment with national efforts. As noted in Part A, personal communications have also indicated substantial time and burdens associated with historical efforts to use existing instruments that were never designed to collect this data (i.e. OMB 0970-0334 and the Federal Financial Report (FFR) SF-425). We have confirmed enthusiasm for these new instruments by virtue of regular conversations with our program participants and ORA program staff. Additional positive feedback has been received from stakeholders representing programs not subject to PRA and established collections using comparable forms, attendees from listening sessions, and unsolicited comments at national program meetings. Further, if an awardee does not provide information using PRA approved information collections, as applied for here, it could be evaluated as a performance issue if the collections are incorporated into the terms and conditions of an awardees’ agreement. As an additional example of enthusiasm, one state program coordinator reached out to ORA to request time to present at an upcoming national program meeting so she could share all the ways these new reporting instruments would help their state program, including a reduction in time to complete program reports for their agreement.

Feedback from potential respondents on draft instruments submitted under this application and shared to solicit feedback with reference to the Federal Register Notice for public comment confirms their desire to use the new forms. FDA stakeholders also indicated making this collection available to the entire population of respondents rather than a smaller subset is highly desirable as it will greatly reduce the time and effort burden for program assessment for ORA and ORS technical staff. ORA data needs are not addressed currently available approved collections for grant, cooperative agreement, or contract reporting. The technical lead for one program recently reported that they received almost no useable data for their evaluation of programmatic and ROI/ROV concerns after reviewing over 50 report submissions. The traditional large pdf narrative based submissions resulting from the aforementioned collection instruments and generalized questions designed to evaluate grant or cooperative agreement performance are often unusable by staff charged with evaluating the program effectiveness or require numerous follow-up by both parties to collect the information needed to assess the program in the format necessary for evaluation and programmatic reporting.

2. Procedures for the Collection of Information

A list of the forms catalog was provided in Supporting Statement Part A.

The ORA has chosen to include the entire population listed above for this pilot. The ORA proposes this strategy as; the population size funded project recipients is relatively small, accessible as enrolled recipients of ORA funding projects, and cooperative as the nature of these projects are to facilitate partnerships and cooperative exchanges with State and local governments in protecting public health. By inviting the entire population, the ORA intends to maximize the statistical power of our pilot and reduce the risk of failing to detect differences in our response population, should they exist (i.e. Type II errors).

• Received data will be evaluated on the following metrics:

  • Comparative report page count.

  • Internal FDA Project manager evaluation of data quality.

  • Number of follow-ups to complete program assessments between historic reporting mechanisms and new forms.

  • Comparative analysis of staff hours required to aggregate data for historical reporting methods an new forms.

• What Success looks like:

  • Reduction in time required to evaluate reports.

  • Useable data at the program level to assess and report effectiveness.

  • Reduction in the number of follow-ups needed to complete individual assessments.

  • Reduction in time to aggregate data into useable reports.

  • ORA can respond effectively to requests from HHS, Congress and other sources for information on program effectiveness and ROI/ROV of funded program.

The population is estimated using all past and current unique entities that have or are receiving ORA funding for public health projects. This population is considered to be relatively static as evidenced by the same entities remaining enrolled in the same ORA funded programs since the initiation of the program. The commodities and production types across the U.S. and its territories tend not to change on a regional basis and so the regulatory programs administered by specific State and local governments charged with protecting public health for these products, and subsequently their participation in specific ORA funded public health projects by commodity or product also tends to see little variation.

The ORA intends to further develop and fine tune reporting instruments in an effort to achieve standardization where possible while collecting useable data to assess funded programmatic performance including ROI/ROV. Should a form element be deemed inadequate to achieve programmatic evaluation and reporting needs by assigned ORA program staff that element will be reviewed and revised. In addition, a threshold of 10% of respondents providing incorrect or unusable data, as determined by ORA technical program evaluators, will trigger a review of the question, data table, or section of the form in question. If the reason for failure is not obvious to the ORA technical reviewers, ORA program staff will seek feedback from respondents as to why they answered the way they did before seeking to revise the form.

Respondents will be asked to submit the data using a designated ORA email account or designated web portal for processing by the reporting deadline (2-4 times per year). Once a reporting deadline has passed, ORA will aggregate the report data into a filterable report for ORA technical evaluators. The technical evaluators, by program, include both subject matter experts in the public health risks targeted by that program and ORA program evaluators responsible for justifying and reporting on public health ROI/ROV metrics for ORA funded projects.

Proposed email correspondence regarding pilot:

Dear [insert program name] participants:

This email is a friendly reminder that the [insert report requirement(s)] is due [insert date]. This reporting is a program requirement within the Funding Opportunity Announcement [insert specific FOA reference or hyperlink].

The ORA Office of Partnerships (OP) is developing a template form that includes all requested fields to make it easier on program participants to structure, maintain, and submit their reports. At this time, use of templates provided by the OP is encouraged and voluntary, and is separate from program recipient’s official responsibilities in submitting the RPPR and other reporting required as described in the Notice of Award.

If you choose to use the template, you may submit it via [insert designated submission route].

As this report is a requirement of the award and use of templates cannot be considered an official reporting mechanism at this time, you will receive a second email from OM or OAGS requesting your report so it can be added to your official grant file. We apologize for any redundancy and inconvenience during this time and appreciate your understanding and patience.

Please feel free to reach out to your Project Manager and Program Official with any questions.  

[End of email]

Data will be aggregated into an easily filtered and sorted excel report for each reporting deadline for FDA program reviewers, allowing them to quickly review responses by form element for all submissions. They will be asked to report any form elements that failed to provide useable data for their evaluation for 10% or more respondents by program. If a review of the form element does not reveal an obvious error in form design to reviewers the ORA program staff will seek feedback from the respondents who answered incorrectly.

Proposed email for seeking feedback:

Hello,

You recently submitted a report for your participation in the [insert funded program name] using a template we are developing for this program.

Thank you for participating in our pilot, your submission is most helpful to us in evaluating and improving the form design for clarity and ease of use by participants for our funded programs.

We have identified a form element [insert tab, section and question or data table name] that was unable to achieve our purpose for data collection and would like your feedback regarding your submission for that element. Would you please let us know if we may contact you via phone or email for more information on how we could improve this form?

Thank you,

[insert Program Manager]

[End of email]

ORA expects the additional burden for each respondent contacted for a follow-up to not exceed 30 minutes per contact.

ORA expects to achieve a majority response rate and therefore does not foresee handling of non-response data. Should non-response be encountered program staff will follow-up requesting submission of the any required programmatic data not received.

Proposed email for non-response:

Hello,

Upon review of your [insert program] report submission we have identified the following deficiencies in meeting the reporting requirements listed in your Notice of Award:

[list criteria from NOA not addressed in the report submission]

We have developed the attached report template to assist you in meeting all program reporting requirements. If you have feedback regarding this template you would like to share you are welcome to submit it in responding to this email.

Thank you,

[insert Program Manager]

[End of email]

3. Methods to Maximize Response Rates and Deal with Non-response

Templates will be introduced via email and technical calls with award recipients. As ORA funded programs are generally cooperative in nature, and feedback regarding historical reporting methods indicates form templates would be welcomed, we do not expect non-response.

ORA intends to obtain the most accurate results possible by including the entire recipient population in the pilot. By doing so we expect to achieve a stable estimate for the utility of the collection and maximize the statistical power of our pilot.

Should program participants choose not to use the templates under the pilot and follow-up call will be made by the project manager to determine why and, if applicable, (e.g., the participant was unaware of the pilot template) encourage them to submit. As ORA awards are cooperative in nature, our program managers often need to extend reporting deadlines to accommodate an individual award recipient’s submission and that courtesy would be applied for pilot template submissions as needed.

4. Test of Procedures or Methods to be Undertaken

ORA expects the pilot as submitted will achieve a great reduction in the time and psychological burdens for ORA funded project recipients and time and effort burdens for FDA program staff charged with program evaluation.

Evaluation metrics to confirm this include:

• Comparative report page count.

• Internal FDA Project manager evaluation of data quality.

• Number of follow-ups to complete program assessments between historic reporting mechanisms and new forms.

• Comparative analysis of staff hours required to aggregate data for historical reporting methods and new forms.

• What Success looks like:

  • Reduction in time required to evaluate reports.

  • Useable data at the program level to assess and report effectiveness.

  • Reduction in the number of follow-ups needed to complete individual assessments.

  • Reduction in time to aggregate data into useable reports

  • ORA can respond effectively to requests from HHS, Congress and other sources for information on program effectiveness and ROI/ROV of funded program.

Data quality will also be improved for the recipients by providing targeted instruments with meaningful metrics specific to the funded project to document their program’s success. In addition, the ORA program staff will be able to provide useable data for their program evaluation and justify ORA’s funding decisions to proactively manage public health risks.

Evaluation metrics to confirm this include:

• Internal FDA Project manager evaluation of data quality

• Number of follow-ups to complete program assessments between historic reporting mechanisms and new forms.

• Comparative analysis of staff hours required to aggregate data for historical reporting methods and new forms.

• What Success looks like:

  • Reduction in time required to evaluate reports

  • Useable data at the program level to assess and report effectiveness

  • Reduction in the number of follow-ups needed to complete individual assessments

  • Reduction in time to aggregate data into useable reports

  • ORA can respond effectively to requests from HHS, Congress and other sources for information on program effectiveness and ROI/ROV of funded program.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The ORA Office of Partnerships will collect and analyze the information for this pilot.

Matt Avis

Phone: 301-796-5830

Email: [email protected]

13