2017 NSCG Adaptive Design Experiment Goals, Interventions, and Monitoring Metrics

Appendix H Design et al.pdf

2017 National Survey of College Graduates (NSCG)

2017 NSCG Adaptive Design Experiment Goals, Interventions, and Monitoring Metrics

OMB: 3145-0141

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 3145-0141 can be found here:

Document [pdf]

Download: pdf | pdf

APPENDIX H

2017 NSCG Adaptive Design
Experiment Goals, Interventions, and
Monitoring Metrics

2017 NSCG Adaptive Design Experiment Goals, Interventions,
and Monitoring Metrics
The 2017 NSCG Adaptive Design Experiment (“2017 Experiment”) will be structured largely
the same as the 2015 NSCG Adaptive Design Experiment. Just as in 2015, we will have
experimental samples for the new sample cases (8,000) and the returning sample cases (10,000)
with control groups identified for comparative purposes. Improvements will come from two
directions for the 2017 Experiment:
1) We will expand the data monitoring metrics that we implement during data collection to
include evaluating the stability of survey estimates.
2) We will automate much of the data analytic and business rule execution that was ad hoc
in nature in the 2013 and 2015 Experiments.
In 2015, NCSES and the Census Bureau worked to develop flow processing capabilities for the
entire survey, with editing, weighting, and imputation occurring at time points during the data
collection period as opposed to waiting until after data collection was over to perform the data
processing. For the 2017 Experiment, we will be implementing simplified versions of flow
processing to allow us to examine differences between the treatment and control not only with
respect to representativeness and response rate, but stability of estimates and the effect of our
nonresponse adjustment. These types of metrics will be considered as contributing factors in our
decisions to make interventions.
The second improvement will arise from automation of the data analytic and business rule
execution that was ad hoc in nature in the adaptive design experiments from previous cycles.
While some monitoring metrics, including R-indicators, were run on an automated basis, specific
decisions about when and where interventions should actually occur were the result of extended
conversations and incremental data analysis. While these steps were important in the early
stages of adaptive design, and for understanding how large interventions would be, adaptive
design cannot be implemented in a standardized, repeatable production setting while maintaining
such an extremely hands-on approach. Instead, for the 2017 Experiment, we will review the
analytical questions that arose during past adaptive design decision meetings, and attempt to
automate these types of analyses in conjunction with the data monitoring metrics.
In a general sense, the goal of the 2017 Experiment is to replicate the successful results we had
in the 2015 Experiment, expand usage of and access to data monitoring metrics, and develop a
baseline level of comfort with automated interventions for adaptive design in a production
setting.
The remainder of this appendix discusses adaptive design goals that NSCG will pursue as part of
the 2017 Experiment, what interventions would allow the NSCG to achieve those goals, and
what monitoring metrics would inform those interventions. As noted earlier, the 2017
Experiment will be structured largely the same as the 2015 Experiment. As a result, the
following goals are similar to the goals pursued as part of the 2015 Experiment. A major
difference for the 2017 Experiment is that these goals will be pursue through the use of increased
automated interventions.

Goal 1: Balance Sample / Reduce Nonresponse Bias
Sampling balancing and/or reducing nonresponse bias relate to maintaining data quality in the
face of shrinking budgets and falling response rates. Nonresponse bias arises when the outcomes
of interest (the survey estimates) for respondents are different from those of nonrespondents.
This difference results in a bias because the resulting estimates only represent a portion of the
total target population. Surveys often try to correct for this after data collection using weighting,
post-stratification, or other adjustments. Adaptive design interventions during data collection
attempt to correct for nonresponse bias during data collection by actually changing the
respondent population to be more balanced on frame characteristics related to response and
outcome measures.
While discussing R-indicators, Schouten et al., provides reasons why balancing on variables
related to response status and outcome variables is desirable. “In fact, we view the R-indicator
as a lack-of-association measure. The weaker the association the better, as this implies that there
is no evidence that non-response has affected the composition of the observed data.” [3] This
suggests that “selective forces…are absent in the selection of respondents” out of the sample
population [2], and so nonresponse approaches missing at random, reducing the risk of nonresponse bias.
Interventions: Interventions are used to change the type or quantity of contacts targeted at
specific subgroups or individuals. Interventions that will be considered for inclusion in the 2017
Experiment include:
•

Sending an unscheduled mailing to sample persons;

•

Sending cases to CATI prior to the start of production CATI non-response follow up
(NRFU), to target cases with an interviewer-assisted mode rather than limiting contacts to
self-response modes;

•

Putting CATI cases on hold, to reduce contacts in interviewer-assisted modes, while still
requesting response in self-response modes;

•

Withholding paper questionnaires while continuing to encourage response in the web
mode to reduce the operational and processing costs associated with certain groups of
cases;

•

Withholding web invites to discourage response in certain groups of cases, while still
allowing these cases to respond using previous invitations;

•

Sending paper questionnaires to web nonrespondents earlier than the scheduled mail date
to provide two modes of self-response rather than one; and

•

Changing the CATI call time prioritization to increase or decrease the probability a case
is called during a specific time.

Monitoring Methods:
•

R-indicators [2], [3], [4];

•

Mahalanobis Distance or other distance measure [5];

•

Response influence [6]; and

•

Uncertainty/influence of imputed y-values [7].

We used R-indicators in the 2013 and 2015 Experiments, and plan to continue using them in the
2017 effort. As a metric, R-indicators were useful for measuring response balance, and served
their purpose as a proof of concept for data monitoring. However, employing more metrics
during data collection allows us to assess the usefulness of each monitoring metric and provides
more confidence that data collection interventions were targeted in the most efficient way
possible. That is, if R-indicators identify subgroups that should be targeted to increase response
balance, and another metric (e.g., response influence, Mahalanobis distance, etc.) identifies
specific cases in those subgroups that also are likely to have an effect on nonresponse bias, then
we have more confidence that those identified cases are the optimal cases for intervention, both
from a response balance and non-response bias perspective.
Goal 2: Increase Timeliness of Data Collection
Analysts and other data users that need relevant, up-to-date information to build models,
investigate trends, and write policy statements rely on timely survey data. NCSES specifically
focused on timeliness as a goal for the 2013 NSCG [4], and reduced the length of time from the
beginning of data collection to the time of data release from 28 months to 12 months. This
required a reduction in the data collection from ten months to six months. In the future, NCSES
is interested in further reducing data collection, specifically, from six months to five months.
Interventions: Interventions will attempt to either encourage response to the NSCG earlier than
the standard data collection pathway or will be used to stop data collection if new respondents
are not changing key estimates. This could be achieved by introducing modes earlier than the
standard data collection pathway, sending reminders that elicit response more quickly, or
stopping data collection for all or a portion of cases and reallocating resources. Possible
interventions include:
•

Sending cases to CATI prior to the start of production CATI non-response follow up
(NRFU), to target cases with an interviewer-assisted mode rather than limiting contacts to
self-response modes;

•

Sending paper questionnaires to web nonrespondents earlier than the scheduled mail date
to provide two modes of self-response rather than one;

•

Sending email reminders earlier than the scheduled dates in data collection; and

•

Stopping data collection for the sample or for subgroups given a sufficient level of data
quality. For example, we could stop data collection if:
o key estimates have stabilized and standard errors fall within acceptable ranges, or
o the coverage ratio for a subgroup of interest reaches a pre-determined threshold.

Monitoring Methods:
•

Propensity to Respond by Modes [8];

•

Change Point Analysis [9];

•

Stability of Estimates [10]; and

•

Coverage Ratios.

Ongoing NSCG research conducted by Chandra Erdman and Stephanie Coffey [8] could inform
appropriate times to introduce new modes to cases ahead of the standard data collection
schedule. Another possibility involves exploring change point analysis. If respondents per day
as a metric changes over time, showing fewer responses in a given mode, there may be cause to
introduce a new mode ahead of schedule. In addition, we will be able to calculate key estimates
on a weekly or semi-weekly basis. As a result, we will be able to track stability of estimates
during data collection to identify times when the data collection strategy has peaked, resulting in
fewer responses or similar information that was already collected.
Goal 3: Reduce Cost
Controlling costs are always a survey management goal. More recently however, “the growing
reluctance of the household population to survey requests has increased the effort that is required
to obtain interviews and, thereby, the costs of data collection…[which] has threatened survey
field budgets with increased risk of cost overruns” [10]. As a result, controlling cost is an
important part of adaptive design. By allowing survey practitioners to reallocate resources
during the data collection period, surveys can make tradeoffs to prioritize cost savings over other
goals.
Interventions: Interventions will be used to encourage survey response via the web while
discouraging response in more expensive modes (mail, CATI), or to eliminate contacts that may
be ineffective. Possible interventions include:
•

Putting CATI cases on hold, to reduce contacts in interviewer-assisted modes, while still
requesting response in self-response modes;

•

Withholding paper questionnaires while continuing to encourage response by web to
reduce the operational and processing costs associated with certain groups of cases;

•

Withholding web invites to discourage response from certain groups of cases, while still
allowing these cases to respond using previous invitations;

•

Prioritizing or deprioritizing cases in CATI during certain call times to increase or
decrease the probability a case is called during a specific time frame without having to
stop calling the case entirely; and

•

Stopping data collection for the sample or for subgroups if key estimates and their
standard errors have stabilized.

Monitoring Methods:
•

R-indicators;

•

Mahalanobis Distance or other distance measure;

•

Response influence;

•

Uncertainty/influence of imputed y-values;

•

Stability of estimates; and

•

Numbers of trips to locating.

The same indicators that are valuable for monitoring data quality also could measure survey cost
reduction. If cases are in over-represented subgroups, or have low response influence, we may
want to reduce or eliminate contacts on those cases.
In addition, the key estimates valuable to increasing timeliness, are also valuable for controlling
cost. When estimates stabilize and their standard errors fall within acceptable limits for
subgroups or the entire survey, new respondents are providing similar information to that which
we have already collected. If continuing data collection would have little effect on estimates and
their standard errors, stopping data collection to all or subgroups of cases would be an efficient
way to control costs.
Another potential cost-saving intervention would be to limit the number of times a case could be
sent to locating. If we have no contact information for a case, or previously attempted contact
information has not been useful for obtaining contact, a case is sent to locating where researchers
attempt to identify new, more up-to-date contact information. This operation can be time
intensive, especially for cases repeatedly sent to locating. We could track the number of times a
case is sent to interactive locating, or the length of time it spends in locating. Cases repeatedly
sent to locating and cases that spend a large amount of time being researched may not be
ultimately productive cases. Reallocating effort spent on these cases to those in locating for a
fewer number of times may be a sensible cost-saving measure that allows us to attempt contact
on more cases, rather than spending large amounts of time (money) on the same cases.
Adaptive Design Data Collection Flow, Intervention Schedule, and Intervention Criteria
To provide insight on the way that adaptive design criteria will be applied in the determination of
interventions for the 2017 NSCG adaptive design experiment, NCSES is submitting an adaptive
design data collection flowchart (Figure H.1.) and a table documenting the adaptive design
intervention schedule and criteria (Table H.1.).
All sample cases will be monitored beginning at week 0. Adaptive interventions will be reviewed
and implemented as needed at weeks 4, 6, 8, 10, 12, 14, 16, 18, 20, 23, and 24 of the data
collection period. As part of the adaptive design experiment, we have identified certain adaptive
interventions that might be implemented depending upon the case monitoring results that could
help the NSCG meet its data collection goals. The decision to implement an adaptive
intervention will be based on the evaluation of specific criteria associated with the data collection

metrics. The specific criteria are described generally below and the specifics are provided in
Table H.1.
The interventions that are considered at a given week are designed to address specific data
collection goals. Early in the data collection, the adaptive interventions attempt to increase the
representativeness of the responding sample by reducing under-representativeness in certain
subgroups. During the middle of the data collection, some of the interventions attempt to address
under-representation concerns, for example with extra questionnaire mailings to the specific
groups, while others focus more on trying to increase representativeness by reducing overrepresentation through the reduction of contacts to certain subgroups. Finally, near the end of the
data collection, using metrics such as the number of trips to locating, response propensities, and
the number of call attempts, the interventions attempt to control data collection costs. The list of
potential interventions for each week is shown in Table H.1. which includes information about
metrics and criteria used and adaptive interventions by week.

Figure H.1. Adaptive Design Experiment: Data Collection Flow
Up-Front
Locating
CATI First Case OR
TIE Case

YES
New
Cohort?

NO
YES

YES

Week 0

TIE and no
previous
response?

Week 1

Week 2

Mail Pre-Notice Letter
(new)

YES

Week 1
Incentive
Case?

YES NO

No YES

Mail
Web/CATI
Invite
w/$

Mail
Web/CATI
Invite

Mail
Reminder
Letter
(incentive)

Mail
Reminder
Letter

Mail Pre-Notice Letter
(old)

2013 nonrespondent?
YES

Valid U.S.
Address?

Mail
Web
Invite

Week 1
Incentive
Case?
YES

Old Cohort,
Mail First?

Mail
Web Invite
w/$

Mail
Web Invite
w/ $
(by Cohort)

Mail
Reminder
Letter
(incentive)

Week 1
Incentive
Case?

YES

Mail
Web Invite
(by Cohort)

Mail
Web Invite
w/ Qnaire

Week 6

Week 8

Adaptive Design Interventions (weeks 4-24)

Week 5

Wk 2.5:
Phone Tree
Reminder – 6/28

UAA w/o address
correction?

CATI Start

Week 4 Possible Interventions:
(1) Send cases to CATI early
(2) Send Questionnaire

Week 4

Mail
Web Invite
w/ Qnaire
w/$

Mail
Reminder
Letter
(incentive)

Mail
Reminder
Letter

Wk 2.5:
Phone Tree
Reminder – 6/28

Week 3.5

YES

CATI Add
Continue
Weekly UAA
CATI adds
until week
12, CATI start

Early
(Bad Address)
CATI Case

Mail Web Invite
(Brown Envelope)

Week 6 Possible Interventions:
(1) Send Cases to CATI early
(2) Send Questionnaire
(3) Put Cases on Hold in CATI

Reminder
Letter

Week 8 Possible Interventions:
(1) Send Cases to CATI early
(2) Send Web Invite Only
(3) Send No Mailing
(4) Put Cases on Hold in CATI
(5) Take Cases off Hold in CATI

Mail
Questionnaire
w/Web Invite

CATI HOLD!
Mail
8/5

Continue Page 2

If a the case is
sent to
locating for a
bad phone
number and
the locator
finds an
address – the
locator should
send a remail.
If the case has
a number but
the locator
also finds an
address, the
case should
continue in
CATI but be
mailed the
next “all
group”
mailing.

Continue Page 2

INTERACTIVE LOCATING

CATI First Cases: (2011 or 2013 ACS
CATI or CAPI responder and 2015
NSCG CATI Responder and mode
pref not web or mail) OR (2015 CATI
mode preference)

Continue Weekly UAA
CATI adds until week 12, CATI start

Continued

Week 9

Mail Reminder
Postcard

Old Cohort
and CATI
First Case?
YES

Phone on file?

Email on file?
YES

YES
NO

Week 10

Phone Tree
Reminder

Week 10 Possible Interventions:
(1) Send Cases to CATI Early
(2) Put Cases on Hold in CATI
(3) Take Cases off Hold in CATI

Email
Reminder

Week 12

Web/CATI
Invite

NRFU Start

When an
address is
found,
case
moves to
web first
path

Week 12 Possible Interventions:
(1) Do Not Send Cases to CATI
(2) Put Cases on Hold in CATI
(3) Take Cases off Hold in CATI
(4)Only Send Web Invite
(5) Send No Invite

YES

Week 13

Reminder Letter

Week 14 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in cATI
(3) Send Questionnaire

Week 14

Week 16

Week 18

Week 16 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Send Questionnaire
(4) Withhold Email Reminder

Email Reminder

Priority Mail
Questionnaire w/Web
Invite

Week 18 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Send Web Invite Only
(4) Send No Invites

Mail Web Invite

Old Cohort with
email on file?

YES

Week 20

Week 23

Week 24

NO
Week 20 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter

Reminder Letter

Mail Web Invite

Week 23 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter

Reminder
Letter

Week 24 Possible Interventions:
(1) Put Cases on Hold in CATI
(2) Take Cases off Hold in CATI
(3) Withhold Reminder Letter

END PRODUCTION

Closeout
8

NRFU - CONTINUE CATI - Closeout 11/18

NEW Cohort?

INTERACTIVE LOCATING – Closeout 11/04

Early
(Bad
Address)
CATI Case

Table H.1. 2017 NSCG Adaptive Design Experiment: Intervention Schedule and Criteria
Intervention
Week
Adaptive Interventions
Metric to Track
Eligibility for Intervention
Send Cases to CATI early

R-Indicators
Overall Response Propensity

- If a threshold of a metric of interest is met. For example, if the - If these subgroups are low interest groups (e.g., non-S&E) we
unconditional partial R-indicator is less than -0.01.
may not intervene.
- If the subgroups are very large and we do not want to move all
cases to CATI, use response propensity for these cases, and
move over "higher" propensity cases.

Send Questionnaire

Propensity to Respond by Mode

- If the probability to respond by mail > probability to respond
by web, consider for intervention.

Do Nothing

Other Contributing Factors

Send Cases to CATI early

R-Indicators

Send Questionnaire

Propensity to Respond by Mode

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

Do Nothing

- If these cases are in over-represented groups or if they are in
low interest groups (e.g., non-S&E), we may not intervene.

If criteria to intervene are not met or contributing factors
outweigh interventions.
- If a threshold of a metric of interest is met. For example, if the - If these subgroups are low interest groups (e.g., non-S&E) we
unconditional partial R-indicator is less than -0.01.
may not intervene.
- If the subgroups are very large and we do not want to move all
cases to CATI, use response propensity for these cases, and
move over "higher" propensity cases.
- If the probability to respond by mail > probability to respond - If these cases are in over-represented groups or if they are in
by web, consider for intervention
low interest groups (e.g., non-S&E), we may not intervene.
- If a threshold of a metric of interest is met. For example, if the - If key estimates of interest have not stabilized in the
unconditional partial R-indicator is greater than +0.01.
experimental group, we may not use this intervention.
- If a case has been to locating 4+ times, put case on hold.
- If response propensity is in the lowest decile of the subgroup,
put case on hold.

If criteria to intervene are not met or contributing factors
outweigh interventions.

Intervention
Week
Adaptive Interventions

Metric to Track

Eligibility for Intervention

Send Cases to CATI early

R-Indicators

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

- If a threshold of a metric of interest is met. For example, if the - If probability of responding by mail > probability of responding
unconditional partial R-indicator is greater than +0.01.
by web, we may apply this intervention to all cases in these
subgroups.
- If the most over-represented subgroups are not much different
than other groups, we may not use this intervention.
- If key estimates of interest have not stabilized in the
experimental group, we may not use this intervention.

Send No Mailing/Invite

R-Indicators
Overall Response Propensity

- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

- If the most over-represented subgroups are not much different
than other groups, we may not use this intervention.
- If key estimates of interest have not stabilized in the
experimental group, we may not use this intervention.

Put Cases on Hold in CATI

R-Indicators

- If a threshold of a metric of interest is met. For example, if the
unconditional partial R-indicator is greater than +0.01.
- If a case has been to locating 4+ times, put case on hold.
- If response propensity is in the lowest decile of the subgroup,
put case on hold.

- If key estimates of interest have not stabilized in the
experimental group, we may not use this intervention.
- If a case with many trips/calls or low propensity is in a high
interest group, we may not use this intervention until late in
data collection.

- If previously over-represented cases are now underrepresented or have an unconditional partial R-indicator less
than +0.002, take cases off hold.
- If response rate for subgroup is 10% less than control, take
cases off hold.

Trips to Locating
Response Propensity

Take Cases off Hold in CATI

Do Nothing

R-Indicators
Response Rate

If criteria to intervene are not met or contributing factors
outweigh interventions.
10

Other Contributing Factors

Intervention
Week
Adaptive Interventions

Metric to Track

Eligibility for Intervention

Send Cases to CATI early

R-Indicators

Put Cases on Hold in CATI

R-Indicators

Trips to Locating
Response Propensity

Other Contributing Factors

10
Take Cases off Hold in CATI

R-Indicators
Response Rate

Do Nothing
Do Not Send Cases to CATI

See Next Row

Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

If criteria to intervene are not met or contributing factors
outweigh interventions.
See Next Row

- This is effectively the same as putting cases on hold in CATI for
nonrespondents in Week 12
- If a threshold of a metric of interest is met. For example, if the - If key estimates of interest have not stabilized in the
unconditional partial R-indicator is greater than +0.01.
experimental group, we may not use this intervention.
- If a case has been to locating 4+ times, put case on hold.
- If a case with many trips/calls or low propensity is in a high
- If response propensity is in the lowest decile of the subgroup, interest group, we may not use this intervention until late in
put case on hold.
data collection.

12
Take Cases off Hold in CATI

R-Indicators
Response Rate

Intervention
Week
Adaptive Interventions

12
(continued)

Metric to Track

Eligibility for Intervention

Other Contributing Factors

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

- If probability of responding by mail > probability of responding
by web, we may apply this intervention to all cases in these
subgroups.
- If the most over-represented subgroups are not much different
than other groups, we may not use this intervention.
- If key estimates of interest have not stabilized in the
experimental group, we may not use this intervention.

Send No Mailing/Invite

R-Indicators
Overall Response Propensity

- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

Do Nothing
Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Questionnaire

Propensity to Respond by Mode

Do Nothing

If criteria to intervene are not met or contributing factors
outweigh interventions.
- If a threshold of a metric of interest is met. For example, if the
unconditional partial R-indicator is greater than +0.01.
- If a case has been locating 4+ times, put case on hold.
- If response propensity is in the lowest decile of the subgroup,
put case on hold.
- If previously over-represented cases are now underrepresented or have an unconditional partial R-indicator less
than +0.002, take cases off hold.
- If response rate for subgroup is 10% less than control, take
cases off hold.

Intervention
Week
Adaptive Interventions
Put Cases on Hold in CATI

Metric to Track

Eligibility for Intervention

Other Contributing Factors

R-Indicators

Trips to Locating
Response Propensity

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Questionnaire

R-Indicators

Send No Email Reminder
(Old Cohort Only)

R-Indicators
Overall Response Propensity

Do Nothing
Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

- If estimates are not significantly different from the control
group, we may not intervene.
- If estimate have stabilized, we may not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we may not intervene.
If the probability to respond by mail > probability to respond by - If these cases are in over-represented groups or if they are in
web, consider for intervention.
low interest groups (e.g., non-S&E), we may not intervene.
- If overall response propensity is in lowest decile (or possibly
quintile) we may not intervene.
- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

Intervention
Week
Adaptive Interventions

Metric to Track

Eligibility for Intervention

Take Cases off Hold in CATI

R-Indicators
Response Rate

Send Web Invite Only

R-Indicators
Propensity to Respond by Mode
Stability of Estimates

Send No Mailing/Invite

R-Indicators
Overall Response Propensity

18
(continued)

Do Nothing
Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

Other Contributing Factors

- If estimates are not significantly different from the control
group, we may not intervene.
- If estimate have stabilized, we may not intervene.
- If benchmarking to frame totals shows that we are accounting
for nonresponse of both controlled for and uncontrolled for
variables, we may not intervene.
- If a threshold of a metric of interest is met. For example, if the - If probability of responding by web > probability of responding
unconditional partial R-indicator is greater than +0.01.
by mail, we may apply this intervention to all cases in these
- If a case has been to locating 4+ times, put case on hold.
subgroups.
- If response propensity is in the lowest decile of the subgroup, - If the most over-represented subgroups are not much different
put case on hold.
than other groups, we may not use this intervention.
- If key estimates of interest have not stabilized in the
experimental group, we may not use this intervention.
- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

20
Take Cases off Hold in CATI

R-Indicators
Response Rate

Intervention
Week
Adaptive Interventions
Send No Reminder Letter

Metric to Track

Eligibility for Intervention

Other Contributing Factors

R-Indicators
Overall Response Propensity

- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

20
(continued)
Do Nothing
Put Cases on Hold in CATI

R-Indicators
Trips to Locating
Response Propensity

Take Cases off Hold in CATI

R-Indicators
Response Rate
Benchmarking to Frame Totals

Send No Web Invite

R-Indicators
Overall Response Propensity

Do Nothing

- If previously over-represented cases are now underrepresented or approaching being under-represented, take
cases off hold.
- If response rate for subgroup is 10% less than control, take
cases off hold.
- If benchmarking to frame totals shows we may be inducing
bias by not contacting individuals in a subgroup, take cases off
hold.
- If a (higher than above) threshold of a metric of interest is
met. For example, if the unconditional partial R-indicator is
greater than +0.20.

If criteria to intervene are not met or contributing factors
outweigh interventions.

Intervention
Week
Adaptive Interventions
Put Cases on Hold in CATI

Metric to Track

Eligibility for Intervention

Other Contributing Factors

R-Indicators

Trips to Locating
Response Propensity

Take Cases off Hold in CATI
24

R-Indicators
Response Rate

Send No Final Reminder Letter Existing Restrictions
Response Propensity
Do Nothing

- Consider cases that were previously on hold in CATI or
previously did not receive mailings for intervention.
If criteria to intervene are not met or contributing factors
outweigh interventions.

References:
[1] Coffey, S. “Report for the 2013 National Survey of College Graduates Methodological
Research Adaptive Design Experiment”. Census Bureau Memorandum for NCSES. April,
2014.
[2] Schouten, B. Cobben, F. Bethlehem, J. “Indicators for representativeness of survey
response.” Survey Methodology. 35.1 (June 2009): pp 101 – 113.
[3] Schouten, B. Shlomo, N. Skinner, C. “Indicators for monitoring and improving
representativeness of response.” Journal of Official Statistics. 27.2 (2011): pp 231 – 253.
[4] Coffey, S. Reist, B. White, M. “Monitoring Methods for Adaptive Design in the National
Survey of College Graduates (NSCG).” 2013 Joint Statistical Meeting Proceedings, Survey
Research Methods Section. Alexandria, VA: American Statistical Association.
[5] de Leon A.R., Carriere K.C. “A generalized Mahalanobis distance for mixed data.” Journal
of Multivariate Analysis. 92 (2005). 174-185.
[6] Särndal, C., Lundström, S. (2008). Assessing auxiliary vectors for control of nonresponse
bias in the calibration estimator. Journal of Official Statistics. 24, 167-191.
[7] Wagner, J. (2014). “Limiting the Risk of Nonresponse Bias by Using Regression
Diagnostics as a Guide to Data Collection.” Presentation at the 2014 Joint Statistical
Meetings. August, 2014
[8] Erdman C., Coffey S. (2014). “Predicting Response Mode During Data Collection in the
NSCG.” Presentation at the 2014 Joint Statistical Meetings. August, 2014
[9] Killick, R. Eckley, I. “Changepoint: An R Package for Changepoint Analysis”.
Downloaded from http://www.lancs.ac.uk/~killick/Pub/KillickEckley2011.pdf on August 8,
2014.
[10] Groves, Robert M., and Steven Heeringa. (2006). “Responsive design for household
surveys: tools for actively controlling survey errors and costs.” Journal of the Royal
Statistical Society Series A: Statistics in Society, 169, 439-457.

File Type	application/pdf
File Title	1999 OMB Supporting Statement Draft
Author	Demographic LAN Branch
File Modified	2016-12-09
File Created	2016-12-09