Part B of supporting statement (24Sep'21)

Supporting statement PART b: Collection of Information Employing Statistical Methods

B.1. Respondent Universe and Selection Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The statistical method employed in the NRT monitoring task collection is a census that includes all active NRT awards. Since this collection will primarily be used for accountability and program management purposes, including responding to queries from Committee of Visitors¹ (COVs) and other scientific experts, a census rather than sampling design is necessary. This project has five respondent types (Principal Investigators (PIs), Co-PIs, Coordinators, Faculty, and Trainees) and is comprised of five survey instruments (Project, PI, Co-PI, Faculty, and Trainee surveys). Table B.1.1 below shows the survey types and the respondents charged with completing those surveys.

	Respondent Type
Survey Type	PI	Co-PI	Coordinator	Faculty	Trainee
Project Survey	X	X*	X
PI Survey	X
Co-PI Survey		X
Faculty Survey				X
Trainee Survey					X

Table B.1.1 NRT Instrument Surveys and Respondents Responsible for Survey Completion

*The PI and Coordinator both have the ability to respond to the Project Survey. In addition, the PI can grant permission to the Co-PI to access and complete the Project Survey if they wish.

Data collection for the project involves all PIs, Co-PIs, Coordinators, Faculty, and Trainees in the NRT program. The universe of respondents is estimated to be 2448 respondents. This estimate is based on 24 respondents per award with 102 awards total. A typical award is comprised of one PI, two Co-PIs, one Coordinator, ten Faculty, and ten Trainees. It should be noted that the Coordinator is tasked with assisting the PI with the completion of the project survey. They are not asked to complete a Coordinator survey as the only information collected about Coordinators consists of their contact details and this is entered by the PI.

The universe of respondents is expected to remain stable over time as new awards are added and other awards conclude..

B.2. Procedures for the Collection of Information

The NRT data collection will use Web-based instruments, and each respondent (PIs, Co-PIs, Coordinators, Faculty, and Trainees) will enter data annually to satisfy their usual

Research Performance Progress Report (RPPR) requirements. To ensure that data are entered in a timely manner, respondents will receive an initial email from the contractor with the link to their specific survey(s), directions for setting up their login credentials, and a note encouraging them to log in to the monitoring system and begin entering data. Requirements for login, data entry, and timelines will also be shared at annual program and new PI orientation meetings.

B.2.1 Statistical Methodology for Stratification and Sample Selection.

Not applicable.

B.2.2 Estimation Procedures.

Not applicable.

B.2.3. Degree of accuracy needed

Not applicable.

B.2.4. Unusual problems requiring specialized sampling procedures

Not applicable.

B.2.5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden

Not applicable.

B.3. Methods to Maximize Response Rates and the Issue of NonResponse

Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

The data collection methodology described above incorporates strategies intended to maximize the response rates for the target population of respondents. The expected response rate for the NRT program is 100%, as data reporting for the NRT program will satisfy awardee’s usual RPPR requirements. The following strategies will be used to help achieve the expected response rate:

• Convenience of Web survey instrument: A Web-based data collection system has been selected as the mode to reduce respondent burden and improve overall response. The proposed system of online surveys will allow respondents to complete the surveys at their convenience (within the annual timeframe of the system opening to the system close.) Respondents will automatically be routed to questions that apply to them through the use of skip patterns. Respondents will be able to save and resume the survey as needed within the allowed timeframe. To further reduce burden, the system will carryover any data that can be pre-populated after the initial award year.

• Targeted outreach to non-respondents: PIs and Coordinators will be responsible for ensuring that other individuals involved in the project submit all necessary data, and as such they will have access to status information via the Web-based system indicating whether the individual respondents in their projects have completed their data entry. In addition, NRT staff will have access to online monitoring tools to check the status of reporting by award. Furthermore, e-mail messages sent by the contractor will be used to follow up with respondents to ensure that all necessary data are reported in a timely manner. The contractor will implement the following outreach plan shown in Table B.3.1:

Table B.3.1 NRT Targeted Outreach Timeline

Notification Timeline	Notification Purpose
Initial survey invitation email	All PIs and Coordinators will receive an email invitation containing a unique link to the web survey, details on setting up their login credentials, and requesting that they begin the survey.
First follow-up email	Within one week after the initial invitation is sent, emails will be sent to individuals who have not yet logged in.
Second follow-up email2	Within two weeks, emails will be sent to individuals who have not yet logged in and individuals who have logged in but who have not entered and saved data.
Third follow-up email2	At the end of the first month, emails will be sent to individuals who have not yet logged in and individuals who have logged in but who have not entered and saved data.
Phone call	If an individual has not responded to any emails after the one-month mark, the contractor will reach out to that individual by phone.
NSF involvement	In the event that an individual has not responded to any emails or phone calls within six weeks from the initial email, the contractor will bring this to NSF’s attention.
Monthly status reports	Every month, the contractor will generate a report to send to each Coordinator and PI. This report will include updates on login statuses, data entry, etc., for their award, while encouraging the Coordinator and PI to check-in with members of their projects.
One month prior to deadline	The contractor will send a reminder of the approaching deadline to those individuals who have not submitted the survey.
One week prior to deadline	The contractor will send a deadline reminder to those individuals who have not submitted the survey.
One day prior to deadline	The contractor will send a final deadline reminder to those individuals who have not submitted the survey.
One day post deadline:	The contractor will send an email informing the individual of the missed deadline and will ask them to respond with their plans to complete the survey. Any requests for extensions will be forward to the COR for approval.

• Survey support (for technical issues and survey content questions): A helpdesk/technical support email and phone number will be provided to respondents on the survey and all outreach emails, so that they may contact a representative with questions about the survey or troubleshooting issues such as login requirements. The contractor will aim to respond to all initial questions within four hours of receipt and resolve those issues within 48 hours of receipt. It is our expectation that the vast majority of issues will be resolved on the same day they are received. The contractor staff will use a ticket management system to log all requests, to track completed tickets, partially completed tickets, and nonresponse tickets throughout the data collection period; the ticket management system will be used daily by the contractor to manage open tickets through to resolution. The ticket management system will be available to NSF staff, and the contractor will submit weekly status reports to the COR to report the number of open tickets, the number of tickets resolved, and the average time taken to resolve tickets.

B.4. Tests of Procedures or Methods

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The NRT data collection has been tested using a pilot test to obtain user feedback to ensure that the questions can be answered in a reasonable amount of time (i.e., that the burden estimates are accurate) and that the directions and question content are easy to understand and follow. The pilot test involved nine individuals who were either PIs or Coordinators. Members from the contracting team and NSF moderated video “Think Aloud” exercises with two of the nine pilot testers. Data obtained from pilot test revealed some areas needing clarification that have since been addressed. The system will continue to be tested throughout the data collection by allowing respondents to provide comments and feedback using the Web system. Any input on the system received from users will be shared with NSF, and any changes needed will be implemented as the system is upgraded in the future. Other opportunities to obtain feedback for improvement purposes include collecting feedback from PIs during meetings and conferences; comprehensive reviews by NSF staff; and testing performed by the system developers. In addition, the NRT system is based on data collection methods currently used by other NSF data collections, and many of the items and response categories follow formats that are already in use.

B.5. Consultants

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NSF has contracted with Creative Business Solutions, Inc. (CBS) to produce and conduct the data collection using the NRT monitoring system. Table B.5.1 identifies specific individuals who consulted on survey design and development and who will be responsible for collecting and analyzing the data. The Contracting Officer Representative (COR), Elizabeth Webber, will be responsible for receiving and approving all contract deliverables. CBS’s project manager, Sean Teehan, (see below for contact details) will work with the COR on any data analysis needs.

Table B.5.1

Individuals Responsible for Statistical Aspects, Data Collection, and Analysis

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Elizabeth Webber	NSF Contracting Officer Representative	National Science Foundation 2415 Eisenhower Ave. Alexandria, VA 22314	703-292-4316
Daniel Denecke	NSF-NRT Program Director	National Science Foundation 2415 Eisenhower Ave. Alexandria, VA 22314	703-292-8072
Sean Teehan	CBS-NRT Project Manager	Creative Business Solutions, Inc. 13003 Winding Creek Rd. Bowie, MD 20721	571-235-4111

1 https://www.nsf.gov/od/oia/activities/cov/

2 Respondents may opt out of these reminders by emailing technical support