2593ss01

The U.S. Environmental Protection Agency’s (EPA) Office of Research and Development (ORD) is implementing a new framework to evaluate its scientific research products. One of the measures is designed to track the number of research products meeting customer needs, which is aligned with the Agency’s third Strategic Plan Goal: Rule of Law and Process. To prioritize robust science, ORD will conduct a survey of individuals currently using ORD’s products.

In determining an appropriate survey approach, EPA strives to achieve an acceptable level of statistical precision and avoid measurement errors while minimizing the burden placed on respondents and the Agency in conducting the survey effort. To do this, EPA is implementing a stratified random selection of products to evaluate using the satisfaction survey. Survey respondents, who are identified through an expert-led selection process, can participate voluntarily by completing and submitting the survey through the online tool SurveyGizmo. Information collected includes a series of questions ascertaining the quality, usability, and timeliness of the research products. EPA anticipates that the number of respondents completing the survey during the period of this ICR will average 250 per year. Information gathered will be collected and securely stored in a combination of restricted-access network drives and password-protected Microsoft SharePoint sites. The survey responses will be securely maintained by ORD and will be withheld from the public pursuant to Section 107(a) of the Ethics in Government Act of 1978.

Summary data will be used to inform the annual end of year performance reporting to Congress and the general public by being published each year in the Annual Performance Report (APR), which is part of the President’s Budget Request and fulfills the EPA’s performance reporting obligations under the Government Performance and Results Act (GPRA). The survey results will also be used to estimate the degree to which ORD research products meet customer needs, which will enable the improvement of the development and delivery of products.

The public will benefit from EPA’s survey because of the transparency and dialogue that this endeavor will offer. States, and other external end users will gain a valuable opportunity to offer input into the development of ORD research products. Participation will ensure that EPA’s research is not only timely, but also highly relevant to intended end users.

A.2 NEED FOR AND USE OF THE COLLECTION

A.2.a Authority and Need for the Collection

Summary data will be used to inform the annual end of year performance reporting to the Congress and the general public by being published each year in the Annual Performance Report (APR), which is part of the President’s Budget Request and fulfills the EPA’s performance reporting obligations under the Government Performance and Results Act (GPRA). The survey results will also be used to estimate the degree to which ORD research products meet customer needs, which is an EPA/ORD performance metric and will enable the improvement of the development and delivery of research products.

2(b) Practical Utility/Users of the Data

The EPA has a responsibility to ensure that its scientific research products meet customer needs. This aligns with the Agency’s third Strategic Goal: Rule of Law and Process. The results of the survey will be used to improve ORD’s scientific research products, particularly the aspects of quality, usability, and timeliness.

A.3 NONDUPLICATION, CONSULTATIONS AND OTHER COLLECTION CRITERIA

The following sections verify that this information collection satisfies the OMB’s non-duplication and consultation guidelines and does not duplicate another collection.

A.3.a Non-duplication

To the best of EPA’s knowledge, formal, direct feedback from non-federal product users on the quality, usability, and timeliness of ORD’s research products is not available through any other avenue. Other data collection efforts EPA considered include the following:

Inclusion of comment boxes on web sites where products are available for use or download.
Decision document analysis and other bibliometric techniques.
Ad hoc inquiries to known product users.
Evaluation of web and social media metrics indicating attention and usage of publicly available scientific products.
Partner interviews.

None of the above methods were found to provide the consistent methodology or analysis that will be enabled by this survey.

A.3.b Public Notice Required Prior to ICR Submission to OMB

To comply with the 1995 Amendments to the Paperwork Reduction Act (PRA), EPA solicited public comment on this ICR for a 60-day period on September 13, 2019. Specifically, EPA published a notice in the Federal Register (84 FR 48352) requesting comment on the estimated respondent burden and other aspects of this ICR. This notice is included in Appendix A. EPA received one non-substantive comment. Therefore, the burden and cost calculations were not adjusted. An additional Federal Register notice will be published when this ICR is submitted to OMB. The public comment period for this additional notice will be 30 days.

A.3.c Consultations

In the preparation of this ICR, the EPA provided respondents from the pilot study the opportunity to comment on the burden associated with the survey. The respondents reported that on average it takes 20 minutes to complete the survey, which was not found to be overly burdensome.

Table 1: List of names and contact information for three individuals whom provided consultations
Name	Address	Telephone Number	Email Address
Mark Hannant	Springfield, Illinois	217-720-0856	[email protected]
Victoria Kalkirtz	San Diego, California	N/A	[email protected]
Chad Loflen	San Diego, California	619-521-3370	[email protected]

A.3.d Effects of Less Frequent Collection

This information collection will occur annually on a voluntary basis. The survey is required to be completed once a year to inform ORD’s process improvement and develop best practices. Additionally, summary data will be published each year in the Annual Performance Report which is mandated under GPRA. Meeting customer needs is a priority goal for the Agency. Less frequent collection would reduce process improvement, as well as decrease responsiveness to state and local governments and other end users.

A.3.e General Guidelines

These reporting or recordkeeping requirements do not violate any of the regulations promulgated by OMB under 5 CFR Part 1320, Section 1320.5.

A.3.f Confidentiality Questions

Some information reported on the form is confidential, including respondent names and e-mail address. This information is required to be withheld from the public pursuant to Section 107(a) of the Ethics in Government Act of 1978. Agency personnel shall not publicly release the information contained therein.

A.3.g Sensitive Questions

None of the information collected by this action results in or requests sensitive information of any nature.

A.4 THE RESPONDENTS AND THE INFORMATION REQUESTED

A.4.a Respondents/NAICS Codes

NAICS Codes

The respondents are individuals working for state, tribal, and local government agencies, public health partners, and nonprofit organizations involved in environmental and natural resources management that are end users of ORD research products. The North American Industry Classification System (NAICS) codes for some of the state and local agencies EPA will survey are shown below in Table 2. Other respondents may include trade organizations and specific industry establishments through cooperative research and development agreements. These cannot be categorized in a single NAICS code.

Table 2: North American Industry Classification System (NAICS) Codes
Administration of Air and Water Resource Solid Waste Management Programs	924110
Administration of Public Health Programs	923120
American Indian and Alaska Native Tribal Governments	921150

Respondents

EPA will survey individuals in state, tribal, and local government agencies, public health partners, and nonprofit organizations, all of which are involved in environmental and natural resources management that are end users of ORD research products. Customers’ organizations are defined as entities which participated in the formulation of the scope of ORD research products and/or will subsequently use the ORD research products for policy, regulatory, financial/budget, and other decision needs related to environmental and natural resources management. Research products include, but are not limited to, journal articles, reports, databases, test results, methods, models, technical facts sheets, technical support, best practices, patents, decision tools, workshops, webinar series, trainings, and websites.

A.4.b Information Requested

EPA will collect information through a series of questions ascertaining the quality, usability, and timeliness of the research products. A copy of the survey is attached as Attachment A and is further described in Supporting Statement B. As described above, respondents will be asked to provide information on their opinion of a specific ORD product.

A.4.b.i Data Items

The respondent is asked to provide information to verify their role as an end user for the product or output (i.e. name, email address, confirmation of product request, time of product receipt, and product familiarity). This information is also requested to verify the respondent’s identity and provide additional context regarding the partner’s use of the product.

The respondent is then requested to respond to a series of product specific questions regarding the respondent’s opinion of the research product’s quality, usability, and timeliness. Information collected includes the respondent’s opinion of the product’s:

Accuracy and scientific soundness
Transparency
Content
Product-specific support provided by EPA
Timeliness of development and delivery

The respondent is also asked to provide responses to questions related to the user’s experience coordinating with EPA on the product’s development and overall impressions of EPA’s research.

The Agency will maintain records of the surveys administered. The Project Manager will retain files of the completed form electronically in a combination of restricted-access network drives and password-protected Microsoft SharePoint sites. The Agency does not anticipate any record keeping activities on the part of the respondents under this ICR.

A.4.b.ii Respondent Activities

The respondent burden hours associated with these activities are described in Section 6a.

Read the survey notification email. Respondents will receive an email containing introductory information on the purpose and objective of the survey, identification of and access to the product(s) to be assessed, and instructions for completion of the survey.
Review the product information. A link to the product will be included in the survey email. If necessary, respondents will review the identified product to ensure they are familiar with the product content. This is expected to take approximately 5 minutes to complete.
Complete the survey. Respondents will access the 508 compliant survey electronically through the provided link to SuveyGizmo. Each survey will require the applicant to read the instructions and complete the survey, which will take approximately 15 minutes to complete.

A.5 INFORMATION COLLECTED: AGENCY ACTIVITIES, COLLECTION METHODOLOGY AND INFORMATION MANAGEMENT

A.5.a Agency Activities

Up-Front Activities

The following pre-assessment activities will be conducted:

Select research products. As described in Part B, EPA will conduct a stratified random selection of fifty (50) research products developed by the Office of Research and Development (ORD) that were delivered in the prior fiscal year for survey. The products will be selected from a subset of ORD research products based upon pre-determined criteria to ensure the research product was intended for immediate, discrete use. This will allow the selected respondents to respond fully to the survey.
Select respondents. For each research product, the responsible ORD Centers will identify five to ten individuals to be surveyed, and provide corresponding contact information. This respondent identification will be conducted based on pre-determined criteria that will ensure those identified are suitable respondents to provide feedback on EPA’s research products.
Send Surveys. This electronic survey will be prepopulated with selected product and respondent information. Survey invitations will be sent via e-mail to the selected respondents. Survey invitation emails will include a cover letter introducing the survey purpose, an EPA point of contact, identify the associated ORD research product, and include instructions and a link to complete the survey through Survey Gizmo.

Data Collection Activities

EPA will conduct the following activities during the data collection phase. The agency burden is described in Section 6c and further information on technical assistance, quality assurance and communication can be found in Part B:

Provide technical assistance: The Agency will provide technical assistance to respondents through the provided EPA point of contact.
Review completed surveys: EPA will review the completed data and perform quality assurance/quality control checks of data entry.
Communicate results of review to the EPA Senior Management: EPA will communicate the results of the survey to EPA Senior Leadership through a summary report provided to the EPA Administrator, ORD Assistant Administrator and ORD Center Directors

A.5.b Collection Methodology and Management

This section discusses the steps that EPA has taken to ensure that the information being collected will be accurate, reliable and retrievable. This methodology was developed using experience gained in conducting the previous survey of federal agency product users, which is further described in Part B. EPA incorporated lessons learned into this methodology.

Survey

Attachment A contains the survey information. EPA developed the survey with the assistance of a workgroup staffed with EPA employees and managers. The workgroup included representatives from EPA Headquarters, EPA Regions, and EPA Research Laboratories and Centers. The 2020 survey is based on experience in distributing the survey in fiscal years 2018 and 2019 to product users within the federal government, as described in Part B.

Section B.2.c.ii describes the steps taken to ensure that the data collection instrument will be an effective tool for retrieving the information EPA needs to meet the 2020 survey objectives.

Selected survey respondents will include:

Individuals which have or may have used a product, or the information translated in a product
Individuals engaged by ORD to inform the development, delivery, or application of a product
Individuals who requested a product

The selected survey respondents will be familiar with the research product and will be able to provide accurate information on their opinion of the quality, usability, and timeliness of the research products. The experience of federal agency end users that participated in the previous surveys indicates that survey respondents could provide an opinion on the research products in question.

Data Quality

EPA will screen the submitted electronic surveys for completeness and check for self-exclusion. As further described in Part B.5.a, data from the survey will be automatically populated into a spreadsheet for analysis. For all Likert-score questions, responses are required in the survey instrument, so completeness is addressed for these questions. The responses are then screened manually to ensure non-scored questions are completed appropriately and there is no information in open-ended responses to indicate the responded is self-excluding from the survey. EPA will use commercial and open source “off the shelf” programs Microsoft Excel, Stata, and R/RStudio to manage and analyze the data.

Public Access to Data

The Agency’s policy is to make the fullest possible disclosure of information without unjustifiable expense or unnecessary delay to the requester. The public will have access to summaries of the survey data in EPA’s Annual Performance Report, which is included as a part of EPA’s Annual President’s Budget Request¹. The summary report will not include identifying information of survey respondents.

A.5.c Small Entity Flexibility

The respondents are small entities (i.e. scientists and program managers). The impact of small entities was taken into consideration during the development of the survey. This collection does not pose a substantial burden on them.

A.5.d Collection Schedule

The current schedule assumes EPA will receive OMB approval for data collection by January 2020. The schedule will be adjusted based on the final approval date. EPA will send surveys to end users as soon as possible after OMB approval. All survey participants will be asked to complete and return the survey within 3 weeks of receipt. Table 3. summarizes the major milestones.

Table 3: Data Collection Schedule
Task	Date
Information Collection Request approved by OMB	December 2019
EPA selects research products to be included in Survey	March 2020
EPA identifies end users for each research product to be included in Survey	April 2020
Electronic Delivery of Surveys to Survey respondents	May 2020
Date by which Respondents are to return the Survey to EPA	June 2020
All Information in the Data System Finalized	July 2020
Report to EPA Assistant Administrator for ORD Due	August 2020

A.6.a Respondent Burden

The mean hourly wage for life, physical and social science occupations is listed in the Bureau of Labor Statistics’ website: https://www.bls.gov/oes/current/oes_nat.htm#19-0000. The mean hourly wage for this occupation group is $35.76.

Based on the pilot study, it is estimated that each respondent will take 20 minutes to complete the survey. The estimated labor cost for life, physical and social science occupations is $35.76, which is multiplied by the standard overhead factor of 1.6, producing a loaded wage rate of $57.21 (rounded to $58). Approximately 250 individuals will respond to the survey annually. Multiplying the estimated number of respondents (250) by the estimated respondent labor cost ($58) by the number of hours (.33), we estimate the total dollar burden estimate for this ICR to be $4,785 annually. The grand total for all 3 years is estimated to be $14,355.

A.6.b Respondent Costs

A.6.b.i Estimating Labor Costs Table 4: Estimating Labor Costs
Respondent Activities to Complete Form	Burden Hours Per Respondent	Burden for Respondent ($35.76 per labor hour multiplied by 1.6 for overhead equals $58)	Number of Respondents Per Year	Total Respondent Cost Per Year
Read instructions and review product summary	5 minutes (.08 hours)	$4.64	250	$1,160
Respond to survey questions	15 minutes (.25 hours)	$14.50	250	$3,625
TOTAL	20 minutes (.33 hours)	$19.14	250	$4,785

A.6.b.ii Estimating Capital and Operations and Maintenance Costs

There are no capital or operations and maintenance costs.

A.6.b.iii Capital/Start-up Operating and Maintenance (O&M) Costs

There are no capital/startup operating and maintenance costs.

A.6.b.iv Annualizing Capital Costs

There are no capital costs.

A.6.c Agency Burden and Cost

Table 5: Annual Agency Burden and Cost
	Agency Hours Per Respondent	Total No. Respondents Per Year	Labor Cost Per Year Per Respondent	Overhead Cost Per Year (60% of labor cost) Per Respondent	Subtotal Labor & Overhead Cost Per Respondent	Total Cost Per Year (rounded)
Project Manager Activities²	219 minutes (3.65 hours)	250 respondents	$173.44	$104.06	$277.50	$69,379
Product Identification	30 min (.5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
Respondent Identification	30 min (5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
Survey dissemination	3 min (.05 hours)	Same as Above	$2.38	$1.43	$3.81	$950
Technical support for respondents	3 min (.05 hours)	Same as Above	$2.38	$1.43	$3.81	$950
Survey response screening	3 min (.05 hours)	Same as Above	$2.38	$1.43	$3.81	$950
Data processing	30 min (.5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
Data analysis	30 min (.5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
Recordkeeping	30 min (.5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
Report writing	60 min (1 hours)	Same as Above	$47.52	$28.51	$76.03	$19,008
Lab/Center Managers Activities	60 minutes (1 hour)	250 respondents	$47.52	$28.51	$76.03	$19,008
Respondent Identification	15 min (.25 hours)	Same as Above	$11.88	$7.13	$19.01	$4,752
Product Identification	15 min (0.25 hours)	Same as Above	$11.88	$7.13	$19.01	$4,752
Report writing	30 min (.5 hours)	Same as Above	$23.76	$14.27	$38.03	$9,504
EPA Grand Total	279 minutes (4.65 hours)	250 respondents	$220.96	$132.58	$353.54	$88,387

A.6.d Estimating Respondent Universe and Total Burden and Costs

See 6(a) and (b).

A.6.e Bottom Line Burden Hours and Costs

A.6.e.i Respondent Tally

Table 6: Bottom-Line Annual Burden and Cost for Respondents

Respondent Activities to Complete the Survey

Burden Hours/Respondent

Burden for Respondent ($23.60 per labor hour multiplied by 1.6 for overhead equals $38)

Number of Respondents Per Year

Total Cost Per Year

Total Per Year

20 minutes

(.33 hours)

$19.14

250

$4,785

A.6.e.ii Agency Tally

Table 7: Bottom-Line Annual Agency Burden and Cost

Agency Hours for Each Respondent

Number of Hours Per Year

Labor Cost Per Year

Overhead Cost Per Year

Total Agency Cost Per year

Agency Total (Program Manager & Reviewers) for 250 Respondents

279 minutes
(4.65 hours)

69,750 minutes (1162.50 hours)

$55,242

($220.96 per respondent)

$26,017

($132.58 per respondent)

$88,387 ($353.54 per respondent

A.6.e.iii Variations in the Annual Bottom Line

We do not anticipate major variations in the annual bottom line burden hours and costs for the respondents or the Agency.

A.6.f Reasons for Change in Burden

This ICR does not modify an existing ICR.

A.6.g Burden Statement

The annual public reporting and recordkeeping burden for this collection of information is estimated to average 20 minutes per response. Burden means the total time, effort, or financial resources expended by persons to generate, maintain, retain, or disclose or provide information to or for a Federal agency. This includes the time needed to review instructions; develop, acquire, install, and utilize technology and systems for the purpose of collecting, validating, and verifying information, processing and maintaining information, and disclosing and providing information; adjusting the existing ways to comply with any previously applicable applications and requirements; train personnel to be able to respond to a collection of information; search data sources; complete and review the collection of information; and transmit, or otherwise disclose the information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for EPA’s regulations are listed in 40 CFR part 9 and 48 CFR chapter 15.

To comment on the Agency’s need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including the use of automated collection techniques, EPA has established a public docket for this ICR under Docket Number EPA-HQ-2018-0774, which is available for online viewing at www.regulations.gov, or in person viewing at the Office of Environmental Information Docket in the EPA Docket Center (EPA/DC), EPA William Jefferson Clinton West Building, Room 3334, 1301 Constitution Avenue, NW, Washington, DC. The EPA Docket Center Public Reading Room is open from 8:30 a.m. – 4:30 p.m., Monday through Friday, excluding legal holidays. The telephone number for the Reading Room is (202) 566-1744, and the telephone number for the Office of Environmental Information Docket is (202) 566-9744. An electronic version of the public docket is available at www.regulations.gov. This site can be used to submit or view public comments, access the index listing of the contents of the public docket, and to access those documents in the public docket that are available electronically. Also, you can send comments to the Office of Information and Regulatory Affairs, Office of Management and Budget, 725 17^th Street, NW, Washington, D.C. 20503, Attention: Desk Officer for EPA. Please include the EPA Docket ID Number EPA-HQ-2018-0774, OMB Control No. 2080-NEW in any correspondence.

1 The Annual President’s Budget Request is available at epa.gov/planandbudget

2 The Project Manager and Lab/Center Manager roles are subject to change according to staffing circumstances. Pay rates are, therefore, averaged to that of a GS-13 Step 1, including the Washington D.C. locality adjustment ($47.52 per hour in calendar year 2019)

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Sean Paul
File Modified	0000-00-00
File Created	2021-01-14