Report - Examination of Paradata Supporting Burden Reduction

Burden Report: Summary of Results for the Quarterly Survey of
Plant Capacity Utilization (QPC)
1 Total Time Spent in the Web Instrument
Total time spent in the survey is the accumulation of time from the initial encounter with the web
instrument to the last event. Last event doesn’t necessarily mean submission because not every
respondent submitted.
The total time spent is not inherently recorded within the paradata, so they need to be calculated
using the time stamps. Here are the steps needed to calculate the total time spent per respondent
using a dummy dataset table for illustration purposes only:
1. Sort by ‘RespondentId’ and ‘Time’ by ascending values (the paradata should already be
sorted but this is done just in case)
2. Create another column called “time_spent_calculation” that takes the difference between
the time of a certain row by the time of the row prior. For instance,
RespondentId

Time

PageValue

100000001

27OCT01:
16:50:00
27OCT01:
16:52:00
27OCT01:
16:52:20
28OCT01:
16:51:00
28OCT01:
16:52:00
28OCT01:
17:30:00

Screen_1

100000001
100000001
100000002
100000002
100000002

Screen_2
Screen_3

time_spent_calculation

time_spent
(seconds)

27OCT01:16:52:00
− 27OCT01:16:50:00
27OCT01:16:52:20
− 27OCT01:16:52:00

120

28OCT01:16:52:00
− 28OCT01:16:51:00
28OCT01:17:30:00
− 28OCT01:16:52:00

60

20

Screen_1
Screen_2
Screen_3

2280

3. Take the sum of the time spent calculation column per respondent to calculate the total
time spent per respondent. For instance,
RespondentId
100000001
100000002

time_spent_total_calculation
120 + 20
60 + 2280

time_spent_total (seconds)
140
2340

4. Summary statistics such as the average and median can then be calculated

1

In quarter three (Q3), the average time spent in the instrument was close to four hours, and the
median time spent was approximately 12 minutes. Up to 75% of respondents spent approximately
72 minutes or less. Table 1a shows the total time spent in the survey for all respondents who logged
in the survey in Q3.
In quarter four (Q4), the average time spent in the instrument was also close to four hours. The
median time sent was approximately 12 minutes. Up to 75% of respondents spent approximately
56 minutes or less. Table 1b shows the total time spent in the survey for all respondents who logged
in the survey in Q4.
Table 1a. Total Time Spent in the Survey for the Q3 QPC Data.
Average
03:58:51

Median
00:12:12

Min
Less than 5 seconds

Max
Less than 4 days

Source: U.S. Census Bureau, Q3 QPC Paradata

Table 1b. Total Time Spent in the Survey for the Q4 QPC Data.
Average
03:42:05

Median
00:11:59

Min
Less than 5 seconds

Max
Less than 5 days

Source: U.S. Census Bureau, Q4 QPC Paradata
Figures 1a and 1b show the total time spent in the survey for Q3 and Q4.

2

Figure 1a. Histogram of Total Time Spent in the Survey for the Q3 QPC Paradata.
100

Percent of Respondents

90
80
70
60
50
40
30

20
10
0
[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Total Time Spent in Hours
Source: U.S. Census Bureau, Q3 QPC Paradata
Figure 1b. Histogram of Total Time Spent in the Survey for the Q4 QPC Paradata.
100

Percent of Respondents

90
80
70
60
50
40
30
20
10
0
[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Total Time Spent in Hours
Source: U.S. Census Bureau, Q4 QPC Paradata

3

2 Completion Time
Completion occurs when the respondent selects the final ‘Submit Data’ button as shown in Figure
2. Completion time is defined as the time between a respondent’s initial contact with the survey
questionnaire and final submission. While completing the questionnaire or partially completing
the questionnaire, the respondent may have multiple contacts and (one or more) submissions over
the course of hours or days. Respondents may also answer some questions, leave others blank,
and fail to formally submit the data. Only respondents who formally submit the questionnaire will
have a completion time calculated. Establishments that filled out some questions but did not select
the submit button will not have a calculated completion time because the questionnaire is
considered incomplete.
Figure 2. Review Screen from the QPC.

Submission occurs in the paradata when the variables ‘PageValue’ and ‘Type’ are first equal to
‘welcome’ and ‘entry’ respectively. Same day completion excludes multiple submissions over the
course of several days. We analyzed the keystroke data that comes before the first instance of a
respondent selecting ‘Submit Data’ to calculate the completion time.
2.1 Completion Time for Those Who Completed the Survey on the Same Day
In quarter 3, of the respondents who submitted the survey regardless of duration, 76% of them
submitted the survey in one day. These respondents had an average completion time of 20 minutes,
and a median time of 8 minutes.
In quarter 4, of the respondents who submitted the survey regardless of duration, 78% of them
submitted the survey in one day. These respondents also had an average completion time of 20
minutes, and a median time of 8 minutes.
Table 2a and Table 2b show the elapsed time between the initial encounter with the instrument
and the first submission in Hours: Minutes: Seconds, for those submissions captured in one sitting.

4

Table 2a. Elapsed Time between Initial Contact with the Web Instrument and Submission for
One Day with the Q3 QPC Data.
Average
00:20:18

Median
00:08:22

Min
Less than 1 minute

Max
Less than 10 hours

Source: U.S. Census Bureau, Q3 QPC Paradata
Table 2b. Elapsed Time between Initial Contact with the Web Instrument and Submission for
One Day with the Q4 QPC Data.
Average
00:20:09

Median
00:08:06

Min
Less than 1 minute

Max
Less than 10 hours

Source: U.S. Census Bureau, Q4 QPC Paradata
Below are the steps needed to calculate the completion time:
1.
2.
3.
4.
5.

Subset the data by removing the respondents who never submitted the survey
Subset the data by removing the respondents that spent more than one day on the survey
Obtain the time of first login & time of first submission
Create a dataset with only rows of the first submission & first login per respondent
Add a column that provides the time difference between the first submission & the first
login
6. Take the sum of the time differences by respondent
7. Calculate the mean and median using the sum of the time differences
Figures 3a and 3b show the total time spent in the survey on one day for the Centurion
respondent IDs in Q3 and Q4.

5

Figure 3a. Histogram of Survey Completion Time for Those Who Submitted on the Same Day
for the Q3 Data.
100

Percent of Respondents

90
80

70
60
50
40
30
20
10
0

[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Completion Time in Hours
Source: U.S. Census Bureau, Q3 QPC Paradata
Figure 3b. Histogram of Survey Completion Time for Those Who Submitted on the Same Day
for the Q4 Data
100

Percent of Respondents

90
80
70
60
50
40
30
20
10
0
[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Completion Time in Hours
Source: U.S. Census Bureau, Q4 QPC Paradata
6

2.2 Completion Time for those Who Completed the Survey in More than One Day
In quarter 3, of the respondents who submitted the survey regardless of duration, 24% of them
finished the survey in more than one day. In quarter 4, of the respondents who submitted the
survey regardless of duration, 22% of them finished the survey in more than one day. Note that,
we included this scenario in our analysis (Table 2c and Table 2d); however, the results are not
reliable since we have no evidence to support the respondents worked on the survey during the
time they were off the survey instrument.
Table 2c. Elapsed Time between Initial Contact with the Web Instrument and First Submission
for More than One Day with Q3 QPC Data.
Average
17 days 17:17:55

Median
16 days 01:42:11

Min
Less than 2 minutes

Max
Less than 80 days

Source: U.S. Census Bureau, Q3 QPC Paradata
Table 2d. Elapsed Time between Initial Contact with the Web Instrument and First Submission
for More than One Day with Q4 QPC Data.
Average
15 days 09:12:00

Median
9 days 00:10:04

Min
Less than 1 minute

Max
Less than 75 days

Source: U.S. Census Bureau, Q4 QPC Paradata
These completion times are calculated as follows:
1.
2.
3.
4.

Subset the data by removing the respondents who submitted the survey in one day
Subset the data by removing the respondents who did not submit the survey
Create a dataset with only rows of the first submission & first login per respondent
Add a column that provides the time difference between the first submission & the first
login
5. Take the sum of the time differences by respondent
6. Calculate the mean and median using the sum of the time differences
Figure 4a and 4b shows the total time spent in the survey on any number of days for the
Centurion respondent IDs in Q3 and Q4.

7

Figure 4a. Histogram of Completion Time in the Survey on More than One Day for the Q3 QPC
Paradata.
100

Percent of Respondents

90
80
70
60
50
40
30
20
10
0
[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Completion Time in Hours
Source: U.S. Census Bureau, Q3 QPC Paradata
Figure 4b. Histogram of Completion Time in the Survey on More than One Day for the Q4 QPC
Paradata.
100

Percent of Respondents

90
80
70

60
50
40
30

20
10
0
[0-1]

(1-2]

(2-3]

(3-4]

(4-5]

(5-6]

>6

Completion Time in Hours
Source: U.S. Census Bureau, Q4 QPC Paradata
8

3 Conclusion
Table 3 below combines summary statistics for the analysis of time in the web instrument and
completion times for quarter 3 and quarter 4.
Table 3. Summary Statistics for all 3 Prior Sections
Quarter 3
Interacted with
the survey
instrument
Completed the
survey in one day
Completed the
survey in multiple
days
Quarter 4
Interacted with
the survey
instrument
Completed the
survey in one day
Completed the
survey in multiple
days

Mean
03:58:51

Median
00:12:12

Min
Less than 5
seconds

Max
Less than 4
days

00:20:18

00:08:22

17 days 17:17:55

16 days
01:42:11

Less than 1
minute
Less than 2
minutes

Less than 10
hours
Less than 80
days

Mean
03:42:05

Median
00:11:59

Min
Less than 5
seconds

Max
Less than 5
days

00:20:09

00:08:06

15 days 09:12:00

9 days 00:10:04

Less than 1
minute
Less than 1
minute

Less than 10
hours
Less than 75
days

To calculate the completion time for the QPC, we encountered a series of methodological issues.
This study requires more analysis to account for the outlier times. The min and max times are
unrealistically short or long (i.e., finishing the survey in a minute or less and taking more than an
entire day). The current method of taking the difference of an action with the action prior creates
large outliers when the respondent works on the survey in multiple sessions or days. As such, the
mean and median for section 2.2 (respondents who spent multiple days on the survey) are massive.
Adjusting for the outlier time difference values could solve this issue.
It is well established that some survey respondents take shortcuts when answering survey
questions, providing responses that are acceptable but not optimal. Thus, when survey respondents
cut corners, it has been called “survey satisficing”. Existing research suggests that speeding can
be detected by identifying respondents who complete the survey in less than 60% of the median
completion time. Further investigation is needed to establish an acceptable completion speed or
what constitutes a response that is “too fast.”
Overall, we feel confident in the case of those who completed the survey on the same day, the
average may be worth a more careful consideration compared to the other two sections. The
skewed nature of the data provides credibility to the median, but in the case of same-day
respondents, the average may factor in time necessary for completion of those who required
additional time for research and survey navigation. The average time, 20 minutes, is reasonable
9

unlike the averages in other sections that are above 3 hours or 15 days. Additionally, due to the
high same-day submission rate (over 76% of those who submitted the survey in one day in both
quarters), 20 minutes is our finding for the recommended completion time.

The Census Bureau has reviewed this data product to ensure appropriate access, use, and
disclosure avoidance protection of the confidential source data used to produce this product (Data
Management System (DMS) number: P-6000374, Disclosure Review Board (DRB) approval
number: CBDRB-FY22-ESMD011-002).

10