ORA SmartForms Pilot Updated SS Part B final 9-22-2022

United States Food and Drug Administration

Pilot Survey to Develop Standardized Reporting Forms for Federally Funded

Public Health Projects and Agreements

OMB Control No. or 0910-[NEW]

SUPPORTING STATEMENT

Part B. Statistical Methods

1. Respondent Universe and Sampling Methods

The universe for this collection includes State and local government units and professional supporting associations for State and local regulatory programs with oversight of FDA regulated products. The total number of unique entities for this collection is estimated to be 135, based on the number of registered entities who have, or have ever received, funding from the Office of Regulatory Affairs (ORA) for a public health project.

This population is considered to be relatively static as evidenced by the same entities remaining enrolled in the same ORA funded programs since the initiation of the program. The commodities and production types across the U.S. and its territories tend not to change and so the regulatory programs administered by State and local governments charged with protecting public health for these products, and subsequently their participation in specific ORA funded public health projects by commodity or product also tends to see little variation.

The table below represents the current enrollment for existing programs covered by this collection.

Program	Potential Respondents*	Applicable Forms
Animal Food Contract	32	Animal Food Contract Quarterly Summary Report
		Veterinary Medicine Course Preregistration Workbook
		Animal Food Safety Inspection Audit Form
		Corrective Action Plan for Program and Individual Performance Deficiencies
		State Implementation Agreement and Year End Evaluation
		Request for Audit Reduction
Animal Food Regulatory Program Standards	25	Animal Feed Regulatory Standards (AFRPS) Program Report
		Veterinary Medicine Course Preregistration Workbook
Egg Contract	6	Egg Contract Quarterly Summary Report
Egg Standards	2	General Program Report Form (non-specific for new cooperative agreement and grant programs)
Flexible Funding Model	44	Flexible Funding Model (FFM) Program Report
		Food Protection Task Force (FPTF) Program Report
		Manufactured Food Course Preregistration Workbook
		Emergency Response Course Preregistration Workbook
Human Food Contract	48	Human Food Contract Quarterly Summary Report
		FDA 3610 Field Inspection Audit
		Corrective Action Plan for Program and Individual Performance Deficiencies
		State Implementation Agreement and Year End Evaluation
		Request for Audit Reduction
Laboratory Flexible Funding Model	55	Laboratory Flexible Funding Model (LFFM) Program Report
		LFFM Sample and Activity Plan_Proposal Template
		LFFM_SRP-Lab Agreement Template_HAF Tracks
		LFFM QTR Chem_LFFM_HAF_Results_Sheet
		LFFM QTR Data Template Micro HAF Product Testing
Mammography Quality Standards Act (MQSA) Contract	41	MQSA MEU and Spending Update Report
Medical Device Contract	2	Medical Devices Contract Quarterly Summary Report
Produce Implementation	56	Produce Program Report
		Produce CAP_Project Plan outline
		Produce CAP Assessment Template
		Produce Course Preregistration Workbook
		Produce Educational Needs Assessment Submission Template
		Produce Instructions Inspection Aggregate Data
		Produce Inspection Aggregate Data Workbook
		Produce Instructions Inventory & Education Aggregate Data
		Produce Inventory & Education Aggregate Data workbook
Scientific Conference Grant	19	Scientific Conference Program Report

* Please note that a majority of enrollees participate in multiple programs accounting for a total of 330 responses across all programs from 135 unique entities.

Initial discussions and feedback from potential respondents regarding this pilot indicate ORA may expect 100% response rate.

Feedback from potential respondents and internal stakeholders also indicated making this collection available to the entire population of respondents rather than a smaller subset is highly desirable as it will greatly reduce the current reporting burden required of project participants as well as the time and effort burden for program assessment for ORA technical staff. More specifically, the ORA has data needs that are not addressed or not in a useable format from currently available approved collections for grant, cooperative agreement, or contract reporting. The traditional large pdf narrative based submissions resulting from these generalized collections are often unusable by ORA staff charged with evaluating the programs or require numerous follow-up by both parties to collect the information needed to assess the program in the format necessary for evaluation and programmatic reporting.

2. Procedures for the Collection of Information

A list of the forms catalog was provided in Supporting Statement Part A.

The ORA has chosen to include the entire population listed above for this pilot. The ORA proposes this strategy as; the population size funded project recipients is relatively small, accessible as enrolled recipients of ORA funding projects, and cooperative as the nature of these projects are to facilitate partnerships and cooperative exchanges with State and local governments in protecting public health. By including the entire population, the ORA intends to maximize the statistical power of our pilot and reduce the risk of failing to detect differences in our response population, should they exist (i.e. Type II errors).

The population is estimated using all past and current unique entities that have or are receiving ORA funding for public health projects. This population is considered to be relatively static as evidenced by the same entities remaining enrolled in the same ORA funded programs since the initiation of the program. The commodities and production types across the U.S. and its territories tend not to change on a regional basis and so the regulatory programs administered by specific State and local governments charged with protecting public health for these products, and subsequently their participation in specific ORA funded public health projects by commodity or product also tends to see little variation.

The ORA intends to further develop and fine tune reporting instruments in an effort to achieve standardization where possible while collecting useable data to assess funded programmatic performance including ROI/ROV. Should a form element be deemed inadequate to achieve programmatic evaluation and reporting needs by assigned ORA program staff that element will be reviewed and revised. In addition, a threshold of 10% of respondents providing incorrect or unusable data, as determined by ORA technical program evaluators, will trigger a review of the question, data table, or section of the form in question. If the reason for failure is not obvious to the ORA technical reviewers, ORA program staff will seek feedback from respondents as to why they answered the way they did before seeking to revise the form.

Respondents will be asked to submit the data using a designated ORA email account or designated webportal for processing by the reporting deadline (2-4 times per year). Once a reporting deadline has passed, ORA will aggregate the report data into a filterable report for ORA technical evaluators. The technical evaluators, by program, include both subject matter experts in the public health risks targeted by that program and ORA program evaluators responsible for justifying and reporting on public health ROI/ROV metrics for ORA funded projects.

Proposed email correspondence regarding pilot:

Dear [insert program name] participants:

This email is a friendly reminder that the [insert report requirement(s)] is due [insert date]. This reporting is a program requirement within the Funding Opportunity Announcement [insert specific FOA reference or hyperlink].

The ORA Office of Partnerships (OP) is developing a template form that includes all requested fields to make it easier on program participants to structure, maintain, and submit their reports. Use of the template(s) is one way to meet this program reporting requirement. OP has submitted copies of the template(s), as well as other templates provided within the program, as part of an Office of Management and Budget (OMB) Paperwork Reduction Act (PRA) pilot program. At this time, use of templates provided by the OP is encouraged and voluntary, and is separate from program recipient’s official responsibilities in submitting the RPPR and other reporting required as described in the Notice of Award.

If you choose to use the template, you may submit it via [insert designated submission route].

As this report is a requirement of the award and use of templates cannot be considered an official reporting mechanism at this time, you will receive a second email from OM or OAGS requesting your report so it can be added to your official grant file. We apologize for any redundancy and inconvenience during this time and appreciate your understanding and patience.

Please feel free to reach out to your Project Manager and Program Official with any questions.  

Data will be aggregated into an easily filtered and sorted excel report for each reporting deadline for program reviewers, allowing them to quickly review responses by form element for all submissions. They will be asked to report any form elements that failed to provide useable data for their evaluation for 10% or more respondents by program. If a review of the form element does not reveal an obvious error in form design the ORA program staff will seek feedback from the respondents who answered incorrectly.

Proposed email for seeking feedback:

Hello,

You recently submitted a report for your participation in the [insert funded program name] using a template we are developing for this program.

Thank you for participating in our pilot, your submission is most helpful to us in evaluating and improving the form design for clarity and ease of use by participants for our funded programs.

We have identified a form element [insert tab, section and question or data table name] that was unable to achieve our purpose for data collection and would like your feedback regarding your submission for that element. Would you please let us know if we may contact you via phone or email for more information on how we could improve this form?

Thank you,

[insert Program Manager]

ORA expects the additional burden for each respondent contacted for a follow-up to not exceed 30 minutes per contact.

ORA expects to achieve a 100% response rate and therefore does not foresee handling of non-response data.

3. Methods to Maximize Response Rates and Deal with Non-response

Templates will be introduced via email and technical calls with award recipients. As ORA funded programs are generally cooperative in nature, and feedback regarding historical reporting methods indicates form templates would be welcomed, we do not expect non-response.

ORA intends to obtain the most accurate results possible by including the entire recipient population in the pilot. By doing so we expect to achieve a stable estimate for the utility of the collection and maximize the statistical power of our pilot.

Should program participants choose not to use the templates under the pilot and follow-up call will be made by the project manager to determine why and, if applicable, (e.g., the participant was unaware of the pilot template) encourage them to submit. As ORA awards are cooperative in nature, our program managers often need to extend reporting deadlines to accommodate an individual award recipient’s submission and that courtesy would be applied for pilot template submissions as needed.

4. Test of Procedures or Methods to be Undertaken

ORA believes the pilot as submitted will achieve a great reduction in the burden for both ORA funded project recipients and ORA program staff charged with program evaluation. Data quality will also be improved for the recipients by providing targeted instruments with meaningful metrics specific to the funded project to document their program’s success. In addition, the ORA program staff by providing useable data for their program evaluation and justify ORA’s funding decisions to proactively manage public health risks. A better solution for the ORA’s funded projects has been needed for some time, as may be evident from receiving no public comments during the PRA 60-day and 30-day notice periods.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design. Include the name of the FDA unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The ORA Office of Partnerships will collect and analyze the information for this pilot.

10