NCSES Response to 2015 NSCG OMB ToC

NCSES Response to OMB Terms of Clearance for the
2015 National Survey of College Graduates
June 11, 2015
On May 22nd, the Office of Management and Budget (OMB) approved the collection of the 2015
National Survey of College Graduates (NSCG) under the following terms:
Within four weeks, NCSES will submit as a nonsubstantive change request, a "decision tree" or other
descriptive rendering of the way that criteria will be applied to determine which non-responding sample
cases will be assigned which treatments when in the adaptive design experiment. It will be important for
the record to reflect how this experiment will be carried out. OMB approves NCSES starting production
activities in the interim.
In response to these terms of clearance, NCSES is submitting the following. First is a description and
graphic of the data collection flow for the 2015 NSCG. Second are the specific criteria that will be used
to determine which adaptive intervention will be implemented.
2015 NSCG Data Collection Flow
All sample cases will be monitored beginning at week 0. Adaptive interventions will be reviewed and
implemented as needed at weeks 4, 6, 8, 10, 12, 14, 16, 18, 20, 23, and 24 of the data collection period.
As part of the adaptive design experiment, we have identified certain adaptive interventions that might
be implemented depending upon the case monitoring results that could help the NSCG meet its data
collection goals. The decision to implement an adaptive intervention will be based on the evaluation of
specific criteria associated with the data collection metrics. The specific criteria are described generally
below and the specifics are provided in Table 1.
2015 NSCG Adaptive Intervention Schedule and Criteria
The interventions that are considered at a given week are designed to address specific data collection
goals. Early in the data collection, the adaptive interventions attempt to increase the
representativeness of the responding sample by reducing under-representativeness in certain
subgroups. During the middle of the data collection, some of the interventions attempt to address
under-representation concerns, for example with extra questionnaire mailings to the specific groups,
while others focus more on trying to increase representativeness by reducing over-representation
through the reduction of contacts to certain subgroups. Finally, near the end of the data collection,
using metrics such as the number of trips to locating, response propensities, and the number of call
attempts, the interventions attempt to control data collection costs. The list of potential interventions
for each week is shown in Table 1 which includes information about metrics and criteria used and
adaptive interventions by week.

1

2015 NSCG Data Collection Flow
Up-Front
Locating
CATI First Case OR
TIE Case

YES

TIE: Temporarily Ineligible

New
Cohort?

NO
YES

YES

Week 0
6/11

NO

Mail Pre-Notice Letter
(new)

TIE and no
previous
response?

YES

Week 1
Incentive
Case?

Week 1
Incentive
Case?

YES NO

No YES

Week 1
6/18

Mail
Web/CATI
Invite
w/$

Week 2
6/25

Mail
Reminder
Letter
(incentive)

Mail
Web/CATI
Invite

Mail
Reminder
Letter

Mail Pre-Notice Letter
(old)

NO

2013 nonrespondent?
YES

NO

Valid U.S.
Address?

Mail
Web
Invite

NO

Week 1
Incentive
Case?
YES

NO

Old Cohort,
Mail First?

NO

Mail
Web Invite
w/$

Mail
Web Invite
w/ $
(by Cohort)

Mail
Reminder
Letter
(incentive)

Mail
Reminder
Letter
(incentive)

Week 1
Incentive
Case?

YES

NO

Mail
Web Invite
(by Cohort)

Mail
Web Invite
w/ Qnaire

Week 6
7/23

Week 8
8/6

Adaptive Design Interventions (weeks 4-24)

Week 5
7/16

Wk 2.5:
Phone Tree
Reminder – 6/28

UAA w/o address
correction?

CATI Start
Week 4 Adaptive Interventions:
(1) Send cases to CATI early
(2) Send Questionnaire
(3) Do Nothing

Week 4
7/9

Mail
Web Invite
w/ Qnaire
w/$

Mail
Reminder
Letter
(incentive)

Mail
Reminder
Letter

Wk 2.5:
Phone Tree
Reminder – 6/28

Week 3.5
7/9

YES

YES

CATI Add
Continue
Weekly UAA
CATI adds
until week
12, CATI start

Early
(Bad Address)
CATI Case

NO

Mail Web Invite
(Brown Envelope)

Week 6 Adaptive Interventions:
(1) Send Cases to CATI early
(2) Send Questionnaire
(3) Put Cases on Hold in CATI
(4) Do Nothing

Reminder
Letter

CATI HOLD!
Mail
8/5

Week 8 Adaptive Interventions:
(1) Send Cases to CATI early
(2) Send Web Invite Only
(3) Send No Mailing
(4) Put Cases on Hold in CATI
(5) Take Cases off Hold in CATI
(6) Do Nothing

If a the case is
sent to
locating for a
bad phone
number and
the locator
finds an
address – the
locator should
send a remail.
If the case has
a number but
the locator
also finds an
address, the
case should
continue in
CATI but be
mailed the
next “all
group”
mailing.

Mail
Questionnaire
w/Web Invite

Continue Page 2

2

Continue Page 2

INTERACTIVE LOCATING

CATI First Cases: (2011 ACS CATI or
CAPI responder and 2013 NSCG
CATI Responder and mode pref not
web or mail) OR (2013 CATI mode
preference)

Continue Weekly UAA
CATI adds until week 12, CATI start

Continued

Week 9
8/13

Mail Reminder
Postcard

Old Cohort
and CATI
First Case?

Phone on file?

Email on file?
YES

YES
NO

Week 10
8/20

Phone Tree
Reminder

Email
Reminder

Week 12
9/3

Web/CATI
Invite

NRFU Start

NEW Cohort?

YES

Week 13
9/10

Week 12 Adaptive Interventions:
(1) Do Not Send Cases to CATI
(2) Put Cases on Hold in CATI
(3) Take Cases off Hold in CATI
(4)Only Send Web Invite
(5) Send No Invite
(6) Do Nothing

NO

Reminder Letter
Week 14 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in cATI
(3) Send Questionnaire
(4) Do Nothing

Week 14
9/17

Week 16
10/1

Week 18
10/15

Week 10 Adaptive Interventions:
(1) Send Cases to CATI Early
(2) Put Cases on Hold in CATI
(3) Take Cases off Hold in CATI
(4) Do Nothing

Week 16 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Send Questionnaire
(4) Withhold Email Reminder
(5) Do Nothing

Email Reminder

Week 18 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Send Web Invite Only
(4) Send No Invites
(5) Do Nothing

Priority Mail
Questionnaire w/Web
Invite

Old Cohort with
email on file?

YES

Week 20
10/29

NO
Week 20 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter
(4) Do Nothing

Reminder Letter

Week 23
11/19

Mail Web Invite

Week 24
11/25

Reminder
Letter

Week 23 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter
(4) Do Nothing

Week 24 Adaptive Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter
(4) Do Nothing

END PRODUCTION

12/13 Closeout

3

When an
address is
found,
case
moves to
web first
path

INTERACTIVE LOCATING – Closeout 11/04

YES

NRFU - CONTINUE CATI - Closeout 11/18

NO

Early
(Bad
Address)
CATI Case

Table 1. 2015 NSCG Adaptive Intervention Schedule and Criteria
Intervention
Adaptive Interventions
Week
Send Cases to CATI early

4

Send Questionnaire

Metric to Track
R-Indicators

Initial Intervention Criteria
- Review the 3-5 most under-represented subgroups for intervention.

Propensity to Respond by Mode

- If the probability to respond by mail > probability to respond by web

Do Nothing

6

- All other

Send Cases to CATI early

R-Indicators

Send Questionnaire

Propensity to Respond by Mode

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

Do Nothing

- Review the 3-5 most under-represented subgroups that haven't already
been sent to CATI for intervention.

- If these subgroups are low interest groups (e.g., non-S&E) we
will not use this intervention.
- If the subgroups are very large, consider moving all cases to
CATI or use response propensity for these cases and intervene
only for "higher" propensity cases.
'- If cases are in over-represented groups or if they are in low
- If the probability to respond by mail > probability to respond by web
interest groups (e.g., non-S&E), we will not use this
intervention.
- Review the 3-5 most over represented subgroups for intervention.
-If key estimates of interest have not stabilized in the
- If a case has been to locating 4+ times, put case on hold.
experimental group, we will not use this intervention.
- If the number of unsuccessful calls is high, put case on hold.
- If case with many trips/calls or low propensity is in a high
- If response propensity is in the lowest decile of the subgroup, put case on interest group, we will not use this intervention until later in
hold.
data collection.
- All other

Send Cases to CATI early

R-Indicators

- Review the 3-5 most under-represented subgroups for intervention.

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

- Review the 3-5 most under-represented subgroups for intervention.

Send No Mailing/Invite

R-Indicators
Overall Response Propensity

- Review the 1-2 most over-represented subgroups for intervention.

8

Do Nothing
Put Cases on Hold in CATI

Supplemental Intervention Criteria
- If the subgroups are very large, consider moving all cases to
CATI or use response propensity for these cases and intervene
only for "higher" propensity cases.
- If cases are in over-represented groups or if they are in low
interest groups (e.g., non-S&E), we will not use this
intervention.

- If these subgroups are low interest groups (e.g., non-S&E) we
will not use this intervention.
- If the subgroups are very large, consider moving all cases to
CATI or use response propensity for these cases and intervene
only for "higher" propensity cases.
- If probability of responding by mail > probability of
responding by web, we will not apply this intervention to all
cases in these subgroups.
- If the most over-represented subgroups are not much
different than other groups, we will not use this intervention.
- If key estimates of interest have not stabilized for the
experimental group, we will not use this intervention.
- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.

- All other
R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

4

-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention until later in
data collection.

Intervention
Adaptive Interventions
Week
Take Cases off Hold in CATI

Metric to Track
R-Indicators
Response Rate

Initial Intervention Criteria
- If previously over-represented cases are now under-represented or
approaching being under-represented, take cases off hold.
- If response rate for subgroup is 10% less than control, take cases off hold.

8 (Continued)

Do Nothing

Supplemental Intervention Criteria
- If estimates are not significantly different from the control
group, we will not use this intervention.
- If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.

- All other

Send Cases to CATI early

R-Indicators

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

Take Cases off Hold in CATI

R-Indicators
Response Rate

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- All other

Do Not Send Cases to CATI

See Next Row

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- This is effectively the same as putting cases on hold in CATI
for nonrespondents in Week 12
- Review the 3-5 most over represented subgroups for intervention.
-If key estimates of interest have not stabilized in the
- If a case has been to locating 4+ times, put case on hold.
experimental group, we will not use this intervention.
- If the number of unsuccessful calls is high, put case on hold.
- If case with many trips/calls or low propensity is in a high
- If response propensity is in the lowest decile of the subgroup, put case on interest group, we will not use this intervention until later in
hold.
data collection.

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

10

Do Nothing

12

- Review the 3-5 most under-represented subgroups that haven't already
been sent to CATI for intervention.

- If these subgroups are low interest groups (e.g., non-S&E) we
will not use this intervention.
- If the subgroups are very large, consider moving all cases to
CATI or use response propensity for these cases and intervene
only for "higher" propensity cases.
- Review the 3-5 most over represented subgroups for intervention.
-If key estimates of interest have not stabilized in the
- If a case has been to locating 4+ times, put case on hold.
experimental group, we will not use this intervention.
- If the number of unsuccessful calls is high, put case on hold.
- If case with many trips/calls or low propensity is in a high
- If response propensity is in the lowest decile of the subgroup, put case on interest group, we will not use this intervention until later in
hold.
data collection.

See Next Row

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- Review the 3-5 most over-represented subgroups for intervention.
- If probability of responding by mail > probability of
responding by web, we will not apply this intervention to all
cases in these subgroups.
- If the most over-represented subgroups are not much
different than other groups, we will not use this intervention.
- If key estimates of interest have not stabilized for the
experimental group, we will not use this intervention.

5

Intervention
Adaptive Interventions
Week
Send No Mailing/Invite
12 (continued)

Metric to Track
R-Indicators
Overall Response Propensity

Do Nothing

Initial Intervention Criteria
- Review the 1-2 most over-represented subgroups for intervention.

Supplemental Intervention Criteria
- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.

- All other

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Questionnaire

Propensity to Respond by Mode

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- If the probability to respond by mail > probability to respond by web,
- If these cases are in over-represented groups or if they are in
review for intervention.
low interest groups (e.g., non-S&E), we will not use this
intervene.
- If overall response propensity is among the lowest decile, we
will not use this intervention.
- All other
- Review the 3-5 most over represented subgroups for intervention.
-If key estimates of interest have not stabilized in the
- If a case has been to locating 4+ times, put case on hold.
experimental group, we will not use this intervention.
- If the number of unsuccessful calls is high, put case on hold.
- If case with many trips/calls or low propensity is in a high
- If response propensity is in the lowest decile of the subgroup, put case on interest group, we will not use this intervention until later in
hold.
data collection.

14

Do Nothing
Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Questionnaire

R-Indicators

Send No Email Reminder
(Old Cohort Only)

R-Indicators
Overall Response Propensity

16

Do Nothing

-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention until later in
data collection.

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
If the probability to respond by mail > probability to respond by web,
- If these cases are in over-represented groups or if they are in
review for intervention.
low interest groups (e.g., non-S&E), we will not use this
intervene.
- If overall response propensity is among the lowest decile, we
will not use this intervention.
Review the 1-2 most over-represented subgroups for intervention.
- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.
- All other

6

Intervention
Adaptive Interventions
Week
Put Cases on Hold in CATI

18

Metric to Track
R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

Initial Intervention Criteria
- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- Review the 3-5 most over-represented subgroups for intervention.
- If probability of responding by mail > probability of
responding by web, we will not apply this intervention to all
cases in these subgroups.
- If the most over-represented subgroups are not much
different than other groups, we will not use this intervention.
- If key estimates of interest have not stabilized for the
experimental group, we will not use this intervention.

Send No Mailing/Invite

R-Indicators
Overall Response Propensity

Do Nothing

- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.

- All other

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send No Reminder Letter

R-Indicators
Overall Response Propensity

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- Review the 1-2 most over-represented subgroups for intervention.
- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.
- All other

20

Do Nothing
Put Cases on Hold in CATI
23

- Review the 1-2 most over-represented subgroups for intervention.

Supplemental Intervention Criteria
-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention until later in
data collection.

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

7

-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention until later in
data collection.

-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention.

Intervention
Adaptive Interventions
Week
Take Cases off Hold in CATI

Metric to Track
R-Indicators
Response Rate
Benchmarking to Frame Totals

Initial Intervention Criteria
- If previously over-represented cases are now under-represented or
approaching being under-represented, take cases off hold.
- If response rate for subgroup is 10% less than control, take cases off hold.
- If benchmarking to frame totals shows we may be inducing bias by not
contacting individuals in a subgroup, take cases off hold.

R-Indicators
Overall Response Propensity

- Review the 1-2 most over-represented subgroups for intervention.

23 (continued)
Send No Web Invite

Do Nothing

24

Supplemental Intervention Criteria
- If estimates are not significantly different from the control
group, we will not use this intervention.
- If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- If the top two most-overrepresented subgroups are not much
more over-represented than the next groups, we will not use
this intervention.

- All other

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Total Number of Calls
Response Propensity

- Review the 3-5 most over represented subgroups for intervention.
- If a case has been to locating 4+ times, put case on hold.
- If the number of unsuccessful calls is high, put case on hold.
- If response propensity is in the lowest decile of the subgroup, put case on
hold.

Take Cases off Hold in CATI

R-Indicators
Response Rate

- If previously over-represented cases are now under-represented or
- If estimates are not significantly different from the control
approaching being under-represented, take cases off hold.
group, we will not use this intervention.
- If response rate for subgroup is 10% less than control, take cases off hold. - If estimate have stabilized, we will not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we will not intervene.
- Review cases that were previously on hold in CATI or previously did not
- If we have any subgroups that we would like to have one last
receive mailings for intervention.
attempt at conversion, we will intervene.
- All other

Send No Final Reminder Letter Existing Restrictions
Response Propensity
Do Nothing

8

-If key estimates of interest have not stabilized in the
experimental group, we will not use this intervention.
- If case with many trips/calls or low propensity is in a high
interest group, we will not use this intervention until late in
data collection.