Supporting Statement for Usability-testing of a Tracking System for Research Experiences for Undergraduate Site Students

Request for Approval under the “Generic Clearance for the Collection of
Routine Customer Feedback” (OMB Control Number: 3145-0215)
TITLE OF INFORMATION COLLECTION: Usability-testing of a Tracking System for
Research Experiences for Undergraduate Site Students
PURPOSE:
Mathematica Policy Research is developing an electronic data system for the National Science
Foundation (NSF) in support of its Research Experiences for Undergraduate (REU) program.
The data system (hereafter the “REU Data System”) will be used to pilot test different
approaches to collecting data required by Congress in the America COMPETES Reauthorization
Act of 2010, which states that students in the REU program must “be tracked, for employment
and continued matriculation in STEM fields, through receipt of the undergraduate degree and for
at least three years thereafter” (Section 514[a][6] of Public Law 111-358). The purpose of this
information collection is to conduct usability-testing of the REU Data System to a) ensure the
content of the system works as intended (that is, items are eliciting the responses we seek), b)
learn how long it takes to complete tasks, and c) identify potential areas to improve userexperience. Mathematica will use results from the usability testing to make revisions to the
system and finalize the OMB clearance package before launching the system’s pilot collection in
fall 2018.
DESCRIPTION OF RESPONDENTS:
The REU Data System will provide a web-platform to pilot test two approaches to obtain basic
student background and participation information—namely, a registration and a common
application. Respondents are college/university students and current or former REU principal
investigators who will volunteer to test the registration and the common application before the
web application is launched in fall 2018. Specifically:
• College students testing the Registration will be asked to register at the REU Data
System website to obtain a unique ID and submit basic demographic and contact
information.
• College students testing the Common Application (which includes the Registration)
will first navigate through the REU Registration as described in the prior bullet, and then
proceed to the common application through which they will submit information commonly
required by REU Sites as part of their applications, such as college GPA.
• Principal investigators testing the Registration or the Common Application. Principal
investigators will register at the REU Data System website and either use the IDs provided
by students (if testing the registration) or retrieve the applicants IDs (if testing the common
application) to record application decisions (whether student was admitted and accepted an
offer to participate) and the participation status of admitted applicants (whether they
actually participated at their Site or not).

1

TYPE OF COLLECTION: (Check one)
[ ] Customer Comment Card/Complaint Form
[X] Usability Testing (e.g., Website or Software)
[] Focus Group

[ ] Customer Satisfaction Survey
[ ] Small Discussion Group
[] Other: ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal
agencies.
4. The results are not intended to be disseminated to the public.
5. Information gathered will not be used for the purpose of substantially informing influential
policy decisions.
6. The collection is targeted to the solicitation of opinions from respondents who have
experience with the program or may have experience with the program in the future.
Name:__Suzanne H. Plimpton, NSF Reports Clearance Officer_______________________
To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Is personally identifiable information (PII) collected? [X] Yes [ ] No
2. If Yes, is the information that will be collected included in records that are subject to the
Privacy Act of 1974? [X] Yes [ ] No
3. If Applicable, has a System or Records Notice been published? [ ] Yes [ ] No N/A
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [X] Yes [ ] No
The students will be offered a gift of $40 for their participation. This amount is roughly
equivalent to four hours of time at minimum wage—including about 2 hours to test the system, 1
hour debrief to share experiences using the system, and 1 hour for logistics (coordination and
review of background information). 1 Principal investigators will not receive a gift for testing the
system as they will be beneficiaries of the REU program.

1

Source of minimum wage information: https://www.dol.gov/whd/minwage/america.htm

2

BURDEN HOURS
Category of Respondent
1. College students – Registration
2. College students – Common Application
3. REU Principal Investigators – Registration
or Common Application
Totals

No. of
Respondents
5
5
5

Participation
Time
2.5 hours
4 hours
2.5 hours

Burden

15

9

45 hours

12.5 hours
20 hours
12.5 hours

Explanation:
1. College students testing the Registration: We plan to reach out to 5 college students to
test the registration.
Participation time (number of burden hours per respondent): We estimate a total of
2.5 hours. The subset of 26 mostly multi-choice questions included in the registration
would take a respondent about .5 hours to complete, assuming about 1 minute per
question and 4 to 5 minutes to answer a handful questions needed to obtain a username
and create a password. In addition, we assume 1 hour debrief to share experiences using
the system and 1 hour for logistics (coordination and review of background information).
2. College students testing the Common Application: We plan to reach out to 5 college
students to test the common application.
Participation time (number of burden hours per respondent): Based on the NSF
GRFP common application, we estimate that the full REU common application will take
12 hours to complete. However, for the purpose of testing the system, respondents will be
told not to spend time writing a personal statement or preparing their resume (as they
would in a real application). Instead, they will test the functionality of uploading such
documents (using mock documents provided for testing). Respondents testing the
common application will therefore need to complete the registration, complete 15 multichoice common application items, and test functionalities for Site searching and
uploading materials. We estimate it will take respondents a total of 4 hours to
participate—including 2 hours to test the system, 1 hour debrief using the system, and 1
hour for logistics.
3. REU Principal Investigators (PIs) in Registration and Common Application: We
estimate 5 REU principal investigator’s respondents (at least one in each of the
disciplines included in the pilot.) Respondents in biology and earth sciences will be
piloting the registration and respondents in the engineering and mathematics the common
application.
Participation time (number of burden hours per respondent): Burden for principal
investigators is estimated to be 2.5 hours based on the time it takes each respondent to:
a) Provide information about their Site and submit additional application
requirements (24 items at a rate of 1.25 minutes per item = 0.5 hours)

3

b) Record applicant IDs (for those testing the registration), admissions decisions,
acceptances, and participation (for those testing the registration or common
application) (3 or 4 items per each of 4 hypothetical applicants= 15 minutes)
c) Test functionality to download data, view applicants’ profiles (for those testing
the common application), and upload admissions decisions through an Excel file
(15 minutes)
d) Debrief with users about their experiences with the system and logistics to
coordinate participation in the test (1.5 hours)
FEDERAL COST: The estimated annual cost to the Federal government is $15,000, which will
be accounted for in the forthcoming information collection request.
Explanation:
This estimate includes the cost to recruit and compensate respondents, conduct testing and
debrief with respondents, analyze data, and finalize the OMB package.
If you are conducting a focus group, survey, or plan to employ statistical methods, please
provide answers to the following questions:
The selection of your targeted respondents
1. Do you have a customer list or something similar that defines the universe of potential
respondents and do you have a sampling plan for selecting from this universe?
[X] Yes
[ ] No
If the answer is yes, please provide a description of both below (or attach the sampling plan)? If
the answer is no, please provide a description of how you plan to identify your potential group of
respondents and how you will select them?
College students will be identified through friends and relatives of Mathematica’s employees,
giving preference to those studying disciplines included in the pilot (mathematics, engineering,
geosciences, and biology) and ensuring representation of two- and four-year college students.
NSF REU Program Officers and REU leadership groups will be asked to identify current or
former REU principal investigators who might be interested in testing the system and provide
feedback.
Administration of the Instrument
1. How will you collect the information? (Check all that apply)
[X] Web-based or other forms of Social Media
[X] Telephone
[ ] In-person
[ ] Mail
[ ] Other, Explain
2. Will interviewers or facilitators be used? [X] Yes [ ] No
Please make sure that all instruments, instructions, and scripts are submitted with the
request.

4