Supporting Evidence on Burden

Attachment C ● Supporting evidence on burden

95

Estimated burden hours for applicants using the REU data system common application

Cecilia Speroni 2, Brittany Vas, and Maria Bartlett
On November 2, 2018, the Office of Management and Budget (OMB) approved the pilot testing of
a new data collection system for the National Science Foundation’s (NSF’s) Research Experiences
for Undergraduates (REU) program (OMB Control No. 3145-0248). The REU data system was
designed as a centralized online platform for the following purposes:
•

Student applicants to provide information required to participate in the REU program.

•

REU Site principal investigators to access applicant information and report admissions and
participation status.

•

Individuals providing references for a student applicant to upload letters of
recommendation.

•

NSF staff to access information about applicants, participants, and Sites.

Most of the system’s respondent burden in this information collection (about 85 percent) is
associated with students completing the common application to the program Sites. In the absence of
evidence from the program, the approved burden for the REU pilot was based on the estimate from
the information collection for the NSF Graduate Research Fellowship Program (GRFP) (12 hours
to complete the full application, OMB Control Number 3145-0023). To inform future information
requests and to ensure the reported burden is accurate, we used data from the pilot to estimate the
average amount of time applicants spent completing the common application. Specifically, we used
system’s paradata—usage data describing the data collection process—to calculate the time elapsed
between an applicant logging into the system and submitting an application to an REU Site.
We found that the average REU applicant in the pilot submitted applications to 2.3 Sites and spent
3.25 hours actively engaged in the system from log-in to submission. If we factor an additional 3.8
hours offline to prepare materials for the application, the total burden estimate is 7 hours—lower
than the 12 hours per applicant originally requested and approved by OMB.
Burden estimate
Because paradata track users’ activities (such as logging in, logging out, and navigating to another
page in the application form), we do not know whether periods of inactivity represent “burden” (for
example, the user could have been thinking about how to answer a question in the application or the
user could have disengaged with the application and the session ultimately expired). Given this, we
estimated burden hours by first calculating the time spent (actively engaged) online and then made
assumptions on the time spent offline preparing application materials.
Online time
We found that, on average, applicants spent about 3 hours, 2 minutes to complete the first
application submission, which included applying to two REU Sites, on average (Table 1). (This
2

Corresponding author ([email protected]).
96

translates into a burden of about 1.5 hours per Site application when applying for the first time.) A
small share of applicants came back to the system to submit applications to additional Sites after the
first submission. These subsequent applications took considerably less time to complete. For
example, among those who submitted a second round of applications, applicants spent 42 minutes
to complete the submission.
Table 1. Applicant burden in the REU common application system

Application
submission
session

Applicants
(N)

Average elapsed time
(hours: minutes:
Average number of
seconds)
Sites applied to (N)

Average elapsed
time per Site
application
(hours: minutes:
seconds)

1

2,892

3:02:16

1.96

1:33:00

2

414

0:42:13

1.3

0:32:28

3

190

0:43:51

1.2

0:36:33

4

94

0:29:23

1.17

0:25:07

5

49

0:34:46

1.1

0:31:36

6

21

0:37:54

1

0:37:54

7

7

0:48:14

1

0:48:14

8

3

0:20:11

1

0:20:11

2

0:05:41

1

9
Average (weighted)

a

2:32:07

0:05:41
1:20:11

Source:

2020 REU data system.

Notes:

Table is based on the 2,892 students who have applied to a 2020 REU Site through the REU data
system as of May 21, 2020 (a total of 6,286 applications).
A total of 125 students were excluded from the analysis of time elapsed due to missing data from
the Site application timestamps. This issue stems from updates to the database that were made
manually to accommodate help desk requests to (1) retract application submissions for students
who wanted to update their application materials and (2) enter award numbers after Sites began
collecting applications (for Sites that recruited while awaiting award funding from NSF).

Table reads: A total of 2,892 applicants submitted an average of two Site applications in one application
session and took, on average, 3 hours, 2 minutes to complete the submission. This translates into
an average time per Site application of 1 hour, 33 minutes. A total of 414 applicants submitted
their applications in two application sessions. In the second session, applicants submitted about 1
Site application, on average, and spent 42 minutes to complete the submission.
aAverage

weighted by the share of applicants in the application submission session.

Offline time
Burden estimates reported in Table 1 include time that applicants spent reviewing instructions,
completing and reviewing their responses, and transmitting their application. It does not necessarily
include time applicants might have spent offline preparing for the application. As part of their
common application, students must upload a curriculum vitae and an unofficial college transcript,
and type a short personal essay. In addition, students must respond to any Site-specific questions
Sites might have added to the common application.
To estimate time spent offline preparing application materials, we analyzed the length of applicants’
essays. We found that applicants typed, on average, a total of 4,585 characters (2,934 characters in
97

their personal statement and 1,651 characters in their responses to the Site-specific questions; Table
2). This amounts to approximately 680 to 700 words. Adults typically type at about 40 words per
minute (wpm) when writing for leisure and 5 wpm for in-depth essays or articles. 3 At the rate of 5
wpm, we estimate applicants spent 2 hours, 20 minutes to draft their responses in the REU
application (700 words divided by 5 wpm is 140 minutes).
Table 2. Length of applicants’ responses to the personal statement and Sites’ questions (number
of characters)
Site-specific questions
Personal
statement

Question 1

Question
2

Question 3

All questions

Average

2,934

561

727

363

1,651

Median

2,955

145

314

94

1,143

Minimum

27

0

0

0

0

Maximum

5,000

5,000

5,000

4,998

14,277

Number of applicants

2,891

2,890

2,891

2,890

2,891

Number of applications

2,891

6,776

6,774

6,772

6,778

Source:

2020 REU data system.

Notes:

Table is based on the 2,892 students who have applied to a 2020 REU Site through the REU data
system as of May 21, 2020. However, one applicant who withdrew the application left responses
missing. The personal statement is a required component of the common application. Sites can add
Site-specific questions to the common application. Because not all Sites required additional
questions, the minimum number of characters for responses to each of these questions is 0. The
system imposes a limit of 5,000 characters to all responses. Spaces and punctuation count as
characters.

Table reads:

Applicants typed, on average, 2,934 characters in their personal statement and 1,651 characters in
their responses to the Site-specific questions.

3

https://capitalizemytitle.com/writing-time/
98

In addition to the time spent drafting the personal statement and Site-specific questions, applicants
spend time offline preparing a resume and obtaining an unofficial college transcript. We assume
applicants would need an additional 1 hour, 30 minutes (offline) to fulfill these requirements. Most
applicants are expected to have a resume they can use with minor updates (because resumes are
often encouraged, if not required, for college admissions) and colleges facilitate transcript requests,
often enabling students to download unofficial copies directly from the registrar’s online system.
Total time
For the average REU applicant in the pilot, the estimated burden is about 7 hours (3.25
hours online and 3.8 hours offline), lower than the 12 hours approved by OMB. The
respondent burden to complete an application through the REU data system depends on the
number of REU Sites the student is applying to. Because REU system users can submit multiple Site
applications through the common application, and Sites can request up to three additional Sitespecific questions, the more Sites the student applies to, the higher the burden the respondent
incurs. The average applicant in the REU pilot submitted 2.3 Site applications through the system
(not shown). Applicants who submit applications to more than two Sites generally do it in two
application sessions (as indicated in Table 1). The average applicant would then submit two Site
applications in one application session, and 0.3 of an application in a subsequent session. Therefore,
for this average applicant, the estimated time to fill out the application online was 3 hours, 15
minutes (Table 3).
In addition, we estimated applicants spend about 2 hours, 20 minutes (offline) drafting the personal
statement and Site-specific questions. If we assume an additional 1 hour, 30 minutes (offline) to
prepare a resume and get an unofficial copy of their college transcript, the total burden estimate is 7
hours, 5 minutes (Table 3). This incurred burden is lower than the approved 12 burden hours for
the REU pilot, which were originally estimated based on the GRFP information collection.
Table 3. Estimated applicant burden for the average REU applicant
REU common application system
(hours: minutes: seconds)
Online time

3:15:44

Offline time
Personal statement and Site-specific questions

2:20:00

Other materials (resume and unofficial transcripts)

1:30:00

Average applicant burden (total)

7:05:44

Source: 2020 REU data system.
Note:

The average REU applicant submitted 2.3 Site applications. Online time for the average REU applicant in
the pilot is calculated using estimates from Table 1 as follows: 2 applications multiplied by time elapsed per
application in first submission session, plus 0.3 applications multiplied by time elapsed per application in
the second submission session. Offline time to draft the personal statement and Site-specific questions
assumes applicants write at the rate of 5 words per minute.

99

Methodological appendix
The REU data system’s paradata record timestamps for when users complete each of the following
activities:
1.
2.
3.
4.

Logging in and logging out
Saving a page
Navigating to a different page
Submitting an application

We extracted the system activity data for each user with at least one application submission for the
2020 REU program and arranged the activity records in chronological order. To calculate an
individual user’s elapsed time, we subtracted the difference between the user’s system activity
timestamps and valid log-in sessions, then summed the resulting differences. A log-in session was
considered valid until the user actively logged out or submitted the application, or until the last
recorded activity if the session expired due inactivity. Log-in errors that occurred before users
successfully logged into the system, if present, were not included in the analysis.
Table A.1 illustrates the system’s usage data for an applicant and provides an example of how we
calculated the online time the applicant spent completing the common application.
Table A.1. Example of system’s usage data for an applicant using the REU common application
Page visited

Running elapsed
time

User number

Date and time

Activity

100020473

2020-01-25 22:57:52

Log-in

100020473

2020-01-25 22:59:12

Personal Info page

0:01:20

100020473

2020-01-25 22:59:42

Demographic Info page

0:01:50

100020473

2020-01-25 23:01:11

Current Enrollment page

0:03:19

100020473

2020-01-25 23:01:54

Additional Info page

0:04:02

100020473

2020-02-15 01:01:54

Session expired

100020473

2020-02-15 04:12:11

Log-in

100020473

2020-02-15 04:12:19

100020473

2020-02-15 04:12:24

100020473

2020-02-15 04:12:24

0:00:00

0:04:02
Letter of Recommendation page

0:04:10
0:04:15

Application submitted

Total elapsed time

0:04:15
0:04:15

Source: REU data system, 2019–2020 REU application cycle.
Note:

Table shows an example of usage data for an applicant who completed the common application over
two days. This applicant never logged out from the system but had one session expire after two hours
of inactivity.
Total elapsed time indicates the time online for which the user was known to be actively engaged with
the website, excluding periods of no recorded system activity until the system automatically logged the
user out.

100