2593ssa02

The U.S. Environmental Protection Agency’s (EPA) Office of Research and Development (ORD) evaluates its scientific research products through a measure designed to track the number of research products meeting partner needs, which supports the EPA’s Cross-agency Strategy One: Ensure Scientific Integrity and Science-Based Decision Making. To prioritize robust science, ORD will conduct a survey of individuals currently using ORD’s products.

In determining an appropriate survey approach, EPA strives to achieve an acceptable level of statistical precision and avoid measurement errors while minimizing the burden placed on respondents and the Agency in conducting the survey effort. To do this, EPA implements a stratified random selection of products to evaluate using the satisfaction survey. Survey respondents, who are identified through an expert-led selection process, can participate voluntarily by completing and submitting the survey through the online tool Qualtrics, which replaces a tool previously used, SurveyGizmo. The mechanism used to conduct the survey was changed due to the Agency’s transition from using such tools as SurveyGizmo to Qualtrics. Information collected includes a series of questions ascertaining the quality, usability, and timeliness of the research products. From FY 20-22, the average number of respondents who completed the survey is 198. Based on this average, ORD is adjusting the number of survey respondents from 250 to 270 as this is the estimated maximum number of respondents based on a sample size of 60 products. Information gathered will be collected and securely stored in a restricted-access network drive and Microsoft SharePoint site. The survey responses will be securely maintained by ORD and will be withheld from the public pursuant to Section 107(a) of the Ethics in Government Act of 1978.

Summary data will be used to inform the annual end of year performance reporting to Congress and the general public by being published each year in the Annual Performance Report (APR), which is part of the President’s Budget Request and fulfills the EPA’s performance reporting obligations under the Government Performance and Results Act (GPRA). The survey results will also be used to estimate the degree to which ORD research products meet partner needs, which will enable the improvement of the development and delivery of products.

The public will benefit from EPA’s survey because of the transparency and dialogue that this endeavor will offer. States, and other external end users will gain a valuable opportunity to offer input into the development of ORD research products. Participation will ensure that EPA’s research is not only timely, but also highly relevant to intended end users.

A.2 NEED FOR AND USE OF THE COLLECTION

A.2.a Authority and Need for the Collection

Summary data will be used to inform the annual end of year performance reporting to the Congress and the general public by being published each year in the Annual Performance Report (APR), which is part of the President’s Budget Request and fulfills the EPA’s performance reporting obligations under the Government Performance and Results Act (GPRA). The survey results will also be used to estimate the degree to which ORD research products meet partner needs, which is an EPA/ORD performance metric and will enable the improvement of the development and delivery of research products.

2(b) Practical Utility/Users of the Data

The EPA has a responsibility to ensure that its scientific research products meet partner needs. This supports EPA’s Cross-agency Strategy One: Ensure Scientific Integrity and Science-Based Decision Making. The results of the survey will be used to improve ORD’s scientific research products, particularly the aspects of quality, usability, and timeliness.

A.3 NONDUPLICATION, CONSULTATIONS AND OTHER COLLECTION CRITERIA

The following sections verify that this information collection satisfies the OMB’s non-duplication and consultation guidelines and does not duplicate another collection.

A.3.a Non-duplication

To the best of EPA’s knowledge, formal, direct feedback from non-federal product users on the quality, usability, and timeliness of ORD’s research products is not available through any other avenue. Other data collection efforts EPA considered include the following:

Inclusion of comment boxes on web sites where products are available for use or download.
Decision document analysis and other bibliometric techniques.
Ad hoc inquiries to known product users.
Evaluation of web and social media metrics indicating attention and usage of publicly available scientific products.
Partner interviews.

None of the above methods were found to provide the consistent methodology or analysis that will be enabled by this survey.

A.3.b Public Notice Required Prior to ICR Submission to OMB

To comply with the 1995 Amendments to the Paperwork Reduction Act (PRA), EPA solicited public comment on this ICR for a 60-day period on December 19, 2022. Specifically, EPA published a notice in the Federal Register (87 FR 77602) requesting comment on the estimated respondent burden and other aspects of this ICR. This notice is included in Appendix A. EPA received one substantive comment. ORD’s response to the comments can be seen in Appendix B— Response to Comment. An additional Federal Register notice will be published when this ICR is submitted to OMB. The public comment period for this additional notice will be 30 days.

A.3.c Consultations

In the preparation of this ICR, EPA contacted respondents from the FY21 survey evaluation to comment on the burden associated with the survey. The respondents reported that on average it takes 15-20 minutes to complete the survey, which was not found to be overly burdensome.

Table 1: List of names and contact information for three individuals whom provided consultations
Name	Address	Telephone Number	Email Address
Philip Villanueva	Washington, DC	202-566-2183	[email protected]
Elise Jakabházy	Washington, DC	202-564-1734	[email protected]
Madeline Strum	Raleigh, NC	919-541-2383	[email protected]

A.3.d Effects of Less Frequent Collection

This information collected will occur on an annual voluntary basis. The survey is completed once a year to inform ORD’s process improvement and develop best practices. Additionally, summary data is published each year in the Annual Performance Report which is mandated under GPRA. Meeting partner needs is a priority goal for the Agency. Less frequent collection would reduce process improvement, as well as decrease responsiveness to state and local governments and other end users.

A.3.e General Guidelines

These reporting or recordkeeping requirements do not violate any of the regulations promulgated by OMB under 5 CFR Part 1320, Section 1320.5.

A.3.f Confidentiality Questions

Some information reported on the form is confidential, including respondent names and e-mail address. This information is required to be withheld from the public pursuant to Section 107(a) of the Ethics in Government Act of 1978. Agency personnel shall not publicly release the information contained therein.

A.3.g Sensitive Questions

None of the information collected by this action results in or requests sensitive information of any nature.

A.4 THE RESPONDENTS AND THE INFORMATION REQUESTED

A.4.a Respondents/NAICS Codes

NAICS Codes

The respondents are individuals working for state, tribal, and local government agencies, public health partners, and nonprofit organizations involved in environmental and natural resources management that are end users of ORD research products. The North American Industry Classification System (NAICS) codes for some of the state and local agencies EPA will survey are shown below in Table 2. Other respondents may include trade organizations and specific industry establishments through cooperative research and development agreements. These cannot be categorized in a single NAICS code.

Table 2: North American Industry Classification System (NAICS) Codes
Administration of Air and Water Resource Solid Waste Management Programs	924110
Administration of Public Health Programs	923120
American Indian and Alaska Native Tribal Governments	921150

Respondents

EPA will survey individuals in state, tribal, and local government agencies, public health partners, and nonprofit organizations, all of which are involved in environmental and natural resources management that are end users of ORD research products. Partner organizations are defined as entities which participated in the formulation of the scope of ORD research product and/or will subsequently use the ORD research products for policy, regulatory, financial/budget, and other decision needs related to environmental and natural resources management. Research products include but are not limited to journal articles, reports, databases, test results, methods, models, technical facts sheets, technical support, best practices, patents, decision tools, workshops, webinar series, trainings, and websites.

A.4.b Information Requested

EPA will collect information through a series of questions ascertaining the quality, usability, and timeliness of the research products. A copy of the survey is attached as Attachment A and is further described in Supporting Statement B. As described above, respondents will be asked to provide information on their opinion of a specific ORD product.

A.4.b.i Data Items

The respondent is asked to provide information to verify their role as an end user for the product (i.e., name, email address, confirmation of product request, time of product receipt, and product familiarity). This information is also requested to verify the respondent’s identity and provide additional context regarding the partner’s use of the product.

The respondent is requested to respond to a series of product specific questions regarding the respondent’s opinion of the research product’s quality, usability, and timeliness. Information collected includes the respondent’s opinion of the product’s:

Accuracy and scientific soundness
Transparency
Content
Product-specific support provided by EPA
Timeliness of development and delivery

The respondent is also asked to provide responses to questions related to the user’s experience coordinating with EPA on the product’s development and overall impressions of EPA’s research.

The Agency will maintain records of the surveys administered. The Project Manager will retain files of the completed form electronically in a combination of restricted-access network drives and Microsoft SharePoint sites. The Agency does not anticipate any record keeping activities on the part of the respondents under this ICR.

A.4.b.ii Respondent Activities

The respondent burden hours associated with these activities are described in Section 6a.

Read the survey notification email. Respondents will receive an email containing introductory information on the purpose and objective of the survey, identification of and access to the product(s) to be assessed, and instructions for completion of the survey.
Review the product information. A link to the product will be included in the survey email. If necessary, respondents will review the identified product to ensure they are familiar with the product content. This is expected to take approximately 5 minutes to complete.
Complete the survey. Respondents will access the 508 compliant surveys electronically through the provided link to Qualtrics. Each survey will require the applicant to read the instructions and complete the survey, which will take approximately 10 minutes to complete.

A.5 INFORMATION COLLECTED: AGENCY ACTIVITIES, COLLECTION METHODOLOGY AND INFORMATION MANAGEMENT

A.5.a Agency Activities

Up-Front Activities

The following pre-assessment activities will be conducted:

Select research products. As described in Part B, EPA will conduct a stratified random selection of sixty (60) research products developed by the Office of Research and Development (ORD) that were delivered in the prior fiscal year for survey. The products will be selected from a subset of ORD research products based upon pre-determined criteria to ensure the research product was intended for immediate, discrete use. This will allow the selected respondents to respond fully to the survey.
Select respondents. For each research product, the responsible ORD Centers and Offices will identify five to ten individuals to be surveyed and provide corresponding contact information. This respondent identification will be conducted based on pre-determined criteria that will ensure those identified are suitable respondents to provide feedback on EPA’s research products.
Send Surveys. This electronic survey will be prepopulated with selected product and respondent information. Survey invitations will be sent via e-mail to the selected respondents. Survey invitation emails will include a cover letter introducing the survey purpose, an EPA point of contact, identify the associated ORD research product, and include instructions and a link to complete the survey through Qualtrics. As mentioned in section A.1.b Short Characterization, the survey tool changed from SurveyGizmo to Qualtrics.

Data Collection Activities

EPA will conduct the following activities during the data collection phase. The Agency burden is described in Section 6c and further information on technical assistance, quality assurance and communication can be found in Part B:

Provide technical assistance: The Agency provides technical assistance to respondents through the provided EPA point of contact.
Review completed surveys: EPA will review the completed data and perform quality assurance/quality control checks of data entry.
Communicate results of review to the EPA Senior Management: EPA will communicate the results of the survey to EPA Senior Leadership through a summary report provided to the EPA Administrator, ORD Assistant Administrator, ORD Center Directors and National Research Program Directors.

A.5.b Collection Methodology and Management

This section discusses the steps that EPA has taken to ensure that the information being collected will be accurate, reliable and retrievable. This methodology was developed using experience gained in conducting the previous survey of federal agency product users, which is further described in Part B. EPA incorporated lessons learned into this methodology.

Survey

Attachment A contains the survey information. EPA developed the survey with the assistance of a workgroup staffed with EPA employees and managers. The workgroup included representatives from EPA Headquarters, EPA Regions, and EPA Research Laboratories and Centers.

Section B.2.c.ii describes the steps taken to ensure that the data collection instrument will be an effective tool for retrieving the information EPA needs to meet the 2020 survey objectives.

Selected survey respondents will include:

Individuals which have or may have used a product, or the information translated in a product
Individuals engaged by ORD to inform the development, delivery, or application of a product
Individuals who requested a product

The selected survey respondents will be familiar with the research product and will be able to provide accurate information on their opinion of the quality, usability, and timeliness of the research products. The experience of end users that participated in the previous surveys indicates that survey respondents could provide an opinion on the research products in question.

Data Quality

EPA will screen the submitted electronic surveys for completeness and check for self-exclusion. As further described in Part B.5.a, data from the survey will be automatically populated into a spreadsheet for analysis. For all Likert-score questions, responses are required in the survey instrument, so completeness is addressed for these questions. The responses are then screened manually to ensure non-scored questions are completed appropriately and there is no information in open-ended responses to indicate the responded is self-excluding from the survey. EPA will use commercial and open source programs such as Microsoft Excel, and R/RStudio/R Shiny to manage, analyze, and visualize the data. ORD is now primarily using R/RStudio/R Shiny to analyze the data collected from this ICR as the platform provides a free comprehensive library of open-source packages to conduct a robust statistical analysis.

Public Access to Data

The Agency’s policy is to make the fullest possible disclosure of information without unjustifiable expense or unnecessary delay to the requester. The public will have access to summaries of the survey data in EPA’s Annual Performance Report, which is included as a part of EPA’s Annual President’s Budget Request¹. The summary report will not include identifying information of survey respondents.

A.5.c Small Entity Flexibility

The respondents are small entities (i.e., scientists and program managers). The impact of small entities was taken into consideration during the development of the survey. This collection does not pose a substantial burden on them.

A.5.d Collection Schedule

The current schedule assumes EPA will receive OMB approval for data collection by July 2023. The schedule will be adjusted based on the final approval date. EPA will send surveys to end users as soon as possible after OMB approval. All survey participants will be asked to complete and return the survey within 2 weeks of receipt. Table 3. summarizes the major milestones.

Table 3: Data Collection Schedule
Task	Date
Information Collection Request approved by OMB	July 2023
EPA selects research products to be included in Survey	March 2023
EPA identifies end users for each research product to be included in Survey	April 2023
Electronic Delivery of Surveys to Survey respondents	July 2023
Date by which Respondents are to return the Survey to EPA	August 2023
All Information in the Data System Finalized	September 2023
Report to EPA Assistant Administrator for ORD Due	October 2023

A.6 Estimating the Burden and Cost of the Collection

A.6.a Respondent Burden

The mean hourly wage for life, physical and social science occupations is listed in the Bureau of Labor Statistics’ website: https://www.bls.gov/oes/current/oes_nat.htm#19-0000. The currently reported mean hourly wage for life, physical and social science occupations is $40.21.

The estimated completion time for the survey is 15 minutes. The estimated labor cost for life, physical and social science occupations is $40.21, which is multiplied by the standard overhead factor of 1.6, producing a loaded wage rate of $64.34. Approximately 270 individuals will respond to the survey annually. Previously, ORD estimated that 250 individuals would respond to its survey. Based on FY19-22 actuals and adjusting for a slightly increased sample size, ORD estimates the maximum number of respondents it may receive is 270.

A.6.b Respondent Costs

A.6.b.i Estimating Labor Costs Table 4: Estimating Labor Costs
Respondent Activities to Complete Form	Burden Hours Per Respondent	Burden for Respondent ($40.21 per labor hour multiplied by 1.6)	Number of Respondents Per Year	Total Respondent Cost Per Year
Read instructions and review product summary	5 minutes (.08 hours)	$5.15	270	$1,390.50
Respond to survey questions	10 minutes (.17 hours)	$10.94	270	$2,953.80
TOTAL	15 minutes (.25 hours)	$16.09	270	$4,344.30

A.6.b.ii Estimating Capital and Operations and Maintenance Costs

There are no capital or operations and maintenance costs.

A.6.b.iii Capital/Start-up Operating and Maintenance (O&M) Costs

There are no capital/startup operating and maintenance costs.

A.6.b.iv Annualizing Capital Costs

There are no capital costs.

A.6.c Agency Burden and Cost

Table 5: Annual Agency Burden and Cost
	Agency Hours Per Respondent	Total No. Respondents Per Year	Labor Cost Per Year Per Respondent	Overhead Cost Per Year (60% of labor cost) Per Respondent	Subtotal Labor & Overhead Cost Per Respondent	Total Cost Per Year (rounded)
Project Manager Activities²	225 minutes (3.75 hours)	270 respondents	$201.26	$120.76	$322.02	$86,945.40
Product Identification	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
Respondent Identification	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
Survey dissemination	5 min (.08 hours)	Same as Above	$4.29	$2.57	$6.86	$1,852.20
Technical support for respondents	5 min (.08 hours)	Same as Above	$4.29	$2.57	$6.86	$1,852.20
Survey response screening	5 min (.08 hours)	Same as Above	$4.29	$2.57	$6.86	$1,852.20
Data processing	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
Data analysis	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
Recordkeeping	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
Report writing	60 min (1 hours)	Same as Above	$53.67	$32.20	$85.87	$23,184.90
Center Managers Activities	60 minutes (1 hour)	270 respondents	$53.67	$32.20	$85.87	$23,184.90
Respondent Identification	15 min (.25 hours)	Same as Above	$13.42	$8.05	$21.47	$5,796.90
Product Identification	15 min (0.25 hours)	Same as Above	$13.42	$8.05	$21.47	$5,796.90
Report writing	30 min (.5 hours)	Same as Above	$26.84	$16.10	$42.94	$11,593.80
EPA Grand Total	285 minutes (4.75 hours)	270 respondents	$254.93	$152.96	$407.89	$110,130.30

A.6.d Estimating Respondent Universe and Total Burden and Costs

See 6(a) and (b).

A.6.e Bottom Line Burden Hours and Costs

A.6.e.i Respondent Tally

Table 6: Bottom-Line Annual Burden and Cost for Respondents

Respondent Activities to Complete the Survey

Burden Hours/Respondent

Burden for Respondent ($40.21 per labor hour multiplied by 1.6)

Number of Respondents Per Year

Total Cost Per Year

Total Per Year

15 minutes

(.25 hours)

$16.09

270

$4,344.30

A.6.e.ii Agency Tally

Table 7: Bottom-Line Annual Agency Burden and Cost

Agency Hours for Each Respondent

Number of Hours Per Year

Labor Cost Per Year

Overhead Cost Per Year

Total Agency Cost Per year

Agency Total (Program Manager & Reviewers) for 270 Respondents

285 minutes
(4.75 hours)

76,950 minutes (1282.50 hours)

$68,831.10 (254.93 per respondent)

$41,299.20

($152.96 per respondent)

$110,130.30

($407.89 per respondent)

A.6.e.iii Variations in the Annual Bottom Line

We do not anticipate major variations in the annual bottom line burden hours and costs for the respondents or the Agency.

A.6.f Reasons for Change in Burden

There is a decrease of .08 hours in the total estimated respondent burden compared to what was previously reported in the original ICR. This burden reduction is attributed to a decrease in time needed for survey completion as survey participants became more accustomed to completing the survey, ORD improved its partner identification process, and ORD now submits an initial outreach notifying participants they were selected for the survey before it is released. However, the estimated number of respondents increased by 20 individuals. The mean hourly wage increased since FY18, (an increase from $35.76/per hour to $40.21 per hour). The respondent burden slightly decreased since ORD submitted its original ICR in FY19 (a slight decrease from $4,785 to $4,344).

A.6.g Burden Statement

The annual public reporting and recordkeeping burden for this collection of information is estimated to be 15 minutes per response. Burden means the total time, effort, or financial resources expended by persons to generate, maintain, retain, or disclose or provide information to or for a federal agency. For the purposes of this request, EPA estimates that the time to complete the survey will be 15 minutes. This includes the time needed to review instructions; develop, acquire, install, and utilize technology and systems for the purpose of collecting, validating, and verifying information, processing and maintaining information, and disclosing and providing information; adjusting the existing ways to comply with any previously applicable applications and requirements; train personnel to be able to respond to a collection of information; search data sources; complete and review the collection of information; and transmit, or otherwise disclose the information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for EPA’s regulations are listed in 40 CFR part 9 and 48 CFR chapter 15.

To comment on the Agency’s need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including the use of automated collection techniques, EPA has established a public docket for this ICR under Docket Number EPA-HQ-2018-0774, which is available for online viewing at www.regulations.gov, or in person viewing at the Office of Environmental Information Docket in the EPA Docket Center (EPA/DC), EPA William Jefferson Clinton West Building, Room 3334, 1301 Constitution Avenue, NW, Washington, DC. The EPA Docket Center Public Reading Room is open from 8:30 a.m. – 4:30 p.m., Monday through Friday, excluding legal holidays. The telephone number for the Reading Room is (202) 566-1744, and the telephone number for the Office of Environmental Information Docket is (202) 566-9744. An electronic version of the public docket is available at www.regulations.gov. This site can be used to submit or view public comments, access the index listing of the contents of the public docket, and to access those documents in the public docket that are available electronically. Also, you can send comments to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17^th Street, NW, Washington, D.C. 20503, Attention: Desk Officer for EPA. Please include the EPA Docket ID Number EPA-HQ-2018-0774, OMB Control No. 2080-0085 in any correspondence.

1 The Annual President’s Budget Request is available at epa.gov/planandbudget3

2 The Project Manager and Center Manager roles are subject to change according to staffing circumstances. Pay rates are, therefore, averaged to that of a GS-13 Step 1, including the Washington D.C. locality adjustment ($53.67 per hour effective January 2023.)

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Sean Paul
File Modified	0000-00-00
File Created	2023-08-18