HSLS 2009 2nd Follow-up MS 1st Incentive Boost Change Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: June 8, 2016

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, OMB Liaison, NCES

FROM: Elise Christopher, HSLS:09 Project Officer, NCES

SUBJECT: High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study First Incentive Boost Change Request (OMB# 1850-0852 v.21) - Calibration Sample Results: Phase 3 and “Thank you” Email

The High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study and 2018 Panel Maintenance request was approved by OMB in December 2015 (OMB# 1850-0852 v.17) with updates in March and May 2016 (OMB# 1850-0852 v.18-20). This submission provides results from the third phase of calibration sample recruitment experiment, and requests approval for recommended first incentive boost amount for phase 3 of the main sample data collection, as well as approval to send a “thank you” email to respondents once they complete the survey. No change to estimated response burden or the total cost to the federal government is associated with this request.

1. “Thank you” email

We propose to send a “thank you” email to respondents when they complete the survey. The goal of this message is to thank them for their time and also provide a convenient way for sample members to contact the project team to report any issues or with questions that may arise related to processing time for incentives. Sample members will be able to reach the Help Desk by replying to the email, rather than having to search for the Help Desk email address or telephone number in the materials they received previously. There are separate versions for respondents who chose to receive their incentives through PayPal, respondents receiving their incentives by check, and for non-incentivized respondents.

PayPal Incentive:

Dear [NAME]:

On behalf of the National Center for Education Statistics in the U.S. Department of Education's Institute of Education Sciences and the staff of the High School Longitudinal Study of 2009 (HSLS:09), we would like to thank you for your participation in the HSLS:09 survey. Your participation in this study is very important to its success! Your payment of $[XX] is a token of our appreciation for your time and participation. Our records indicate that you chose to receive your incentive via PayPal. Please note it can take up to a full business day to process. Please look for an email from PayPal indicating that you have received $[XX] from RTI International, or check your Paypal account. Please contact us with any concerns about your incentive.

If you have additional questions, please do not hesitate to call the HSLS:09 Help Desk at 1-877-282-4757.

Again, thank you for your time and willingness to participate.

HSLS:09 Data Collection Team

1-877-282-4757

Check Incentive:

Dear [NAME]:

On behalf of the National Center for Education Statistics in the U.S. Department of Education's Institute of Education Sciences and the staff of the High School Longitudinal Study of 2009 (HSLS:09), we would like to thank you for your participation in the HSLS:09 survey. Your participation in this study is very important to its success! Your payment of $[XX] is a token of our appreciation for your time and participation. Our records indicate that you selected a check payment. Your check is being processed and should arrive in a few weeks. Please contact us with any concerns about your incentive.

If you have additional questions, please do not hesitate to call the HSLS:09 Help Desk at 1-877-282-4757.

Again, thank you for your time and willingness to participate.

HSLS:09 Data Collection Team

1-877-282-4757

No Incentive:

Dear [NAME]:

On behalf of the National Center for Education Statistics in the U.S. Department of Education's Institute of Education Sciences and the staff of the High School Longitudinal Study of 2009 (HSLS:09), we would like to thank you for your participation in the HSLS:09 survey. Your participation in this study is very important to its success!

If you have additional questions, please do not hesitate to call the HSLS:09 Help Desk at 1-877-282-4757.

Again, thank you for your time and willingness to participate.

HSLS:09 Data Collection Team

1-877-282-4757

2. Phase 3 incentive boost amount

A change request was submitted and approved in April to establish baseline incentive plans for the main sample. This memorandum provides data collection results from the calibration sample third phase (first incentive boost experiment), and requests approval for recommended incentive boost plans for the main sample.

For reference, an excerpt from Part B of the second follow-up main study submission that describes main study responsive design plans is presented in Attachment 1 below. In the HSLS:09 second follow-up main study, there are three subgroups of special interest.

1. Subgroup A¹ (high school late/alternative/non-completers) is the subset of sample members who, as of the 2013 Update, had not completed high school, were still enrolled in high school, received an alternative credential, completed high school late, or experienced a dropout episode with unknown completion status.

2. Subgroup B (ultra-cooperative respondents) includes sample members who participated in the base year, first follow-up, and 2013 Update without an incentive offer. These cases were also early web respondents in the 2013 Update and, by definition, are high school completers.

3. Subgroup C (high school completers and unknown high school completion status) includes cases that, as of the 2013 Update, were known to be on-time or early regular diploma completers (and not identified as ultra-cooperative) and cases with unknown high school completion status who were not previously identified as ever having a dropout episode.

To determine optimal incentive amounts, a calibration subsample has been selected from each of the aforementioned subgroups to begin data collection ahead of the main sample. The experimental subsamples are fielded about six weeks prior to the main sample to allow time to analyze the results and consult with OMB to determine the baseline incentive amounts to be implemented for each subgroup in the main sample.

Calibration sample results from phase 3 (boost 1). Phase 3 of the calibration study introduced an incentive boost that was offered to a subset of pending nonrespondents in addition to the baseline amount offered in the prior phases. The purpose of this memo is to present the results from phase 3 for the calibration sample and to recommend an incentive boost amount for each subgroup to be implemented in the main sample.

• Subgroup A (High School Late/Alternative/ Non-Completers.) Among all remaining nonrespondents, cases were randomized to a boost 1 incentive amount of $15 or $25.

• Subgroup B (Ultra-Cooperative Respondents.) Among the remaining nonrespondents, cases that were identified for targeting by the bias-likelihood model were randomized to a boost 1 incentive amount of $10 or $20.

• Subgroup C (All Other High School Completers and Unknown Cases.) Among the remaining nonrespondents, cases that were identified for targeting by the bias-likelihood model were randomized to a boost 1 incentive amount of $10 or $20.

Subgroup A (High School Late/Alternative/ Non-Completers). Exhibit 1 displays response rates during phase 3 by incentive boost level. One-way chi-square tests were used to perform pairwise contrasts between the boost amounts offered to all remaining nonrespondents in Subgroup A calibration sample. Among cases in Subgroup A, analyses of phases 1 and 2 showed that there were no significant differences in response rates for the different baseline incentive amounts offered (for more details, see HSLS:09 calibration study analysis: Phase 1 (baseline incentive) and 2 (CATI)), therefore all baseline amounts were collapsed together for the comparison of response rates by boost amount.

No significant difference was detected between the response rates of sample members who were offered the $15 (13.5 percent) and $25 (15.6 percent) boost 1 incentive (χ² (1, N = 364) = .33, p = .56). Our recommendation, therefore, is that a boost of $15 be offered to all cases in the Subgroup A main sample.

Exhibit 1. Subgroup A response rates in phase 3, by boost 1 incentive amount

Boost 1	Sample members (n)	Boost 1 response		Boost 1 response rate (%)
		Yes	No
$15	185	25	160	13.5
$25	179	28	151	15.6
Total	364	53	311	14.6

NOTE: Excludes partially completed cases.

SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Subgroup B (Ultra-Cooperative Respondents). Exhibit 2 displays Subgroup B response rates during phase 3 by incentive boost level for those targeted by the bias-likelihood model for intervention. Note that most of the ultra-cooperative sample members had responded in phases 1 and 2, leaving very few that were targeted for an incentive intervention in phase 3 (only 18 remaining cases). Given the small number of cases within Subgroup B, statistical analysis of the boost 1 incentive was not conducted. However, we recommend that the minimum incentive ($10) be offered to all the Subgroup B main sample cases.

Exhibit 2. Subgroup B response rates in phase 3, by boost 1 incentive amount

Boost 1	Sample members (n)	Boost 1 response		Boost 1 response rate (%)
		Yes	No
$10	9	3	6	33.3
$20	9	2	7	22.2
Total	18	5	13	27.8

NOTE: Excludes partially completed cases.
SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

Subgroup C (All Other High School Completers and Unknown Cases). Exhibit 3 displays Subgroup C response rates during phase 3 by incentive level, among the 661 cases selected for an incentive boost based on the bias-likelihood model. No significant difference was detected between the phase 3 response rates of sample members offered $10 (13.0 percent) and $20 (15.5 percent) boost 1 (χ² (1, N = 661) = 0.88, p = .35). As such we recommend a boost 1 of $10 be offered to all cases in the Subgroup C main sample.

Exhibit 3. Subgroup C response rates in phase 3, by boost 1 incentive amount

Boost 1	Sample members (n)	Boost 1 response		Boost 1 response rate (%)
		Yes	No
$10	332	43	289	13.0
$20	329	51	278	15.5
Total	661	94	567	14.2

NOTE: Excludes partially completed cases.
SOURCE: U.S. Department of Education, National Center for Education Statistics. High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Main Study.

NCES thanks OMB for considering these changes. Data collection for the main sample is scheduled to begin on June 20, 2016.

Attachment 1 – Description of main study responsive design plans (excerpt from Supporting Statement Part B approved on 12/15/2015)

B.4.d Main Study Plans

NCES and RTI are working closely together to design a data collection approach that makes use of evaluations from prior interventions that were used to improve sample representativeness by ensuring that the responding sample is as similar as possible to the total sample. In previous rounds of HSLS:09 and in other NCES studies (such as BPS:12/14, B&B:08/12, and ELS:2002 third follow-up), responsive designs have been used to improve sample representativeness in key survey variables. The proposed main study data collection plan has been designed to maximize data quality through a responsive design approach in which variance between the responding sample and the overall sample is estimated at several points during data collection. An advantage of the proposed responsive design is that it allows us to determine, during data collection, how representative the responding sample is of the total sample, so that we can focus efforts and resources on bringing in the cases that are most needed to achieve balance in the responding sample.

Plans for the HSLS:09 second follow-up main study are based upon 1) results of incentive experiments and responsive design modeling simulations from the HSLS:09 second follow-up field test, 2) results from related longitudinal studies, and 3) prior experience with the HSLS:09 cohort. This section describes plans for responsive design in the main study data collection. In particular, there are three subgroups of interest that will be handled differently. This section describes the phases of data collection and how and when interventions will be implemented and evaluated. Finally, we discuss the development of the response likelihood and bias likelihood models that will be used to identify cases for targeted treatments.

Sample subgroup classification. In the HSLS:09 second follow-up main study, there will be three subgroups of special interest.

1. Subgroup 1 (high school late/alternative/non-completers) will be the subset of sample members who, as of the 2013 Update, had not completed high school, were still enrolled in high school, received an alternative credential, completed high school late, or experienced a dropout episode with unknown completion status.

2. Subgroup 2 (ultra-cooperative respondents) includes sample members who participated in the base year, first follow-up, and 2013 Update without an incentive offer. These cases were also early web respondents in 2013 Update and, by definition, are high school completers.

3. Subgroup 3 (high school completers and unknown high school completion status) will include cases that, as of the 2013 Update, were known on-time or early regular diploma completers (and not identified as ultra-cooperative) and cases with unknown high school completion status who were not previously identified as ever having a dropout episode.

Calibration subsamples. To determine optimal incentive amounts, a calibration subsample will be selected from each of the aforementioned subgroups to begin data collection ahead of the main sample. A similar approach was used successfully in BPS:12/14, where approximately 10 percent of that sample (3,700 cases) was selected and fielded seven weeks prior to the rest of the BPS:12/14 sample. The experimental subsample was treated in advance of the remaining cases, and after analyzing the results for the experimental sample and consultation with OMB, the successful treatment was implemented with the remaining sample. In the HSLS:09 second follow-up main study, a similar approach is proposed with the HSLS:09 calibration subsamples fielded six weeks prior to the rest of the HSLS:09 sample. Exhibit B-7 shows the estimated size of each subgroup, the percentage of cases to be selected for the calibration subsample, and the estimated number of cases in the calibration sample.

Exhibit B-7. Calibration Sample Sizes, by Subgroup

Subgroup Number	Subgroup Description	Main Sample	Calibration Sample	Calibration Percent
1	High School Late/Alternative/ Non-Completers Non-completers, late completers, still enrolled, and alternative credential as of the 2013 Update as well as ever dropouts with no completion status,	2,545	509	20%
2	Ultra-Cooperative Respondents High school completers who participated in base year and the first follow-up, and completed the 2013 Update in early web period, with no incentive	1,027	154	15%
3	All Other High School Completers and Unknown Cases HS Diploma completed early/on-time unknown or unknown completion status with no known dropout episode	19,747	1,975	10%

Data collection phases, treatments, and evaluations. For the second follow-up main study, the data collection plan includes a phased responsive design strategy specifically aimed at improving sample representativeness in the final survey participants. Exhibit B-8 presents the schedule for the planned phases of data collection for both the calibration samples and the main samples. Exhibit B-9 summarizes the baseline and boost incentives to be tested for each subgroup. The phases will proceed as follows:

Baseline incentive (phase 1). During this beginning phase of data collection, the survey will be open exclusively for self-administered interviews via the web. Web response will remain open throughout the entire data collection. As described above, the calibration samples will allow for testing of incentive amounts on a subset of cases, and the results will inform the implementation plan for the main samples. Prior to the start of the main sample data collection for phase 1, calibration sample response rates will be evaluated. An ANOVA-based model will be used to perform pairwise contrasts between the different incentive amounts offered to the treatment and control groups in each phase. NCES and OMB will meet to review the results of the calibration experiment and determine the optimal incentive amount for each of the subgroups.

• Subgroup 1 (high school late/alternative/non-completers) will be offered 3 different baseline incentive amounts ($30, $40, or $50). The optimal amount (to be determined in consultation with OMB) will be offered to all cases in the subgroup 1 main sample.

• Subgroup 2 (ultra-cooperative respondents) will not be offered a baseline incentive. The subgroup 2 calibration sample response rate will be evaluated against early response rates for other cohorts (such as BPS:12/14 and ELS:2002 third follow-up) to estimate a “successful” response benchmark for HSLS:09. If it is determined that the subgroup 2 calibration sample response rate is not successful, we will discuss with OMB the possibility of offering a baseline incentive (amount to be determined in consultation with OMB) to the subgroup 2 main sample.

• Subgroup 3 (high school completers and unknowns) will be offered 6 different incentive amounts, ranging from $15 to $40 ($15, $20, $25, $30, $35, or $40). The $15 starting point for this baseline incentive calibration experiment is based on the results of the HSLS:09 second follow-up field test experiment. The optimal amount (to be determined in consultation with OMB) will be offered to all cases in the subgroup 3 main sample.

Exhibit B-8. Data Collection Schedule and Phases.

Outbound CATI prompting (phase 2). After phase 1 data collection which is self-administered via the web (except for instances when sample members call in to the help desk), phase 2 will initiate another mode of data collection. Telephone interviewers will begin making outbound calls to prompt for self-administration or to conduct telephone interviews. No additional incentives will be offered during phase 2.

• Subgroup 1 will begin outbound CATI earlier than the other subgroups, to allow additional time for telephone interviewers to work these high priority cases.

Incentive boosts (phases 3 and 4). Phases 3 and 4 introduce the use of responsive design with the bias likelihood model. Targeted cases will be offered an incentive boost in addition to the baseline incentive offer. The calibration samples will allow for testing of incentive boost amounts on a subset of the remaining nonrespondents in phases 3 and 4, and the results will inform the incentive boost implementation plan for the main samples. Prior to the start of the main sample data collection for phases 3 and 4, calibration sample response rates will be evaluated. An ANOVA-based model will be used to perform pairwise contrasts between the different incentive boost amounts offered to the treatment and control groups in each phase. NCES and OMB will meet to review the results of the calibration experiment and determine the optimal incentive boost amount for each of the subgroups.

• Subgroup 1 (high school late/alternative/non-completers) will be offered an incentive boost of either $15 or $25, on top of the baseline incentive they were offered in phase 1. The optimal amount (to be determined in consultation with OMB) based on the calibration sample results, will be offered to all remaining nonrespondents in subgroup 1.

• The subset of subgroup 2 (ultra-cooperative respondents) cases that are targeted for intervention, based on bias likelihood modeling, will be offered an incentive boost of either $10 or $20, and the optimal amount (to be determined in consultation with OMB) will be offered only to targeted cases among the remaining subgroup 2 nonrespondents.

• The subset of subgroup 3 (high school completers and unknowns) cases that are targeted for intervention, based on bias likelihood modeling, will be offered an incentive boost of either $10 or $20, and the optimal amount (to be determined in consultation with OMB) will be offered only to targeted cases among the remaining subgroup 3 nonrespondents.

Exhibit B-9. Main study baseline and incentive boost experiments

	Incentive Phase	Amount	Total Cumulative Incentives Offered	Estimated Number of Cases to be Worked
High School Late/Alternative/Non-Completers	Base Incentive	$30	$30 to $50	170
	(all calibration sample cases)	$40		170
		$50		169
	Boost 1 (all remaining calibration sample nonrespondents)	$15	$45 to $75	158
		$25		158
	Boost 2 (all remaining calibration sample nonrespondents)	$10	$55 to $95	102
		$20		102
Ultra-Cooperative Respondents	Base Incentive (all calibration sample cases)	$0	$0	154
	Boost 1 (for targeted cases only: combined with subsample 3)	$10	$10 to $20 targeted; $0 otherwise	(very few if any cases expected to be selected)
		$20
	Boost 2 (for targeted cases only: combined with subsample 3)	$10	$10 to $40 targeted; $0 to $20 otherwise	(very few if any cases expected to be selected)
		$20
High School Completers and Unknowns	Base Incentive	$15	$15 to $40	330
	(all calibration sample cases)	$20		329
		$25		329
		$30		329
		$35		329
		$40		329
	Boost 1 (for targeted cases: 1/2 of non-respondents)	$10	$25 to $60 targeted; $15 to $40 otherwise	250
		$20		250
	Boost 2 (for targeted cases: 1/2 of non-respondents)	$10	$25 to $80 targeted; $15 to $60 otherwise	175
		$20		175

Additional treatments for targeted cases. In addition to the monetary interventions described above, the HSLS:09 second follow-up main study design includes non-monetary treatments to be used with targeted cases.

Field interviewing (phase 5). Field interviewing will be conducted for all targeted nonrespondents at the same time; there will be no time lag between the calibration and main samples. Cases identified for targeted treatment (all high school late/alternative/non-completers, and sample members with high bias likelihood scores) will be considered for field interviewing. The decision to conduct field interviewing for a case may also be determined by other factors, such as the location of a case and its proximity to other likely field cases. Nontargeted cases may potentially be included in field interviewing if it is cost effective to do so. Conversely, given the expense of field interviewing, cases with a very low response likelihood may not be pursued.

Extended data collection (phase 6). Cases identified for targeted treatment (all high school late/alternative/non-completers, and sample members with high bias likelihood scores) will be part of an extended data collection period. During this period (the last month of data collection), only targeted cases will be actively prompted to participate. Data collection will remain open for all other cases if they choose to participate, but effort to pursue those cases will be suspended.

Model development. A critical element of any responsive design is the method used to identify cases that will receive special treatment. As described above, the primary goal of this approach is to improve sample representativeness. The bias likelihood model will help determine which cases are most needed to balance the responding sample, and the response likelihood model will help determine which cases may not be optimal for pursuing with targeted interventions so that project resources can be most effectively allocated. In this section, we describe our modeling approach and the variables to be considered for use as predictor variables for both the bias likelihood and the response likelihood models. Variables will be drawn from data obtained in prior waves of data collection with this cohort (base-year, first follow-up, and 2013 Update survey data; high school transcripts; school characteristics; sampling frame information; and paradata). The models for the HSLS:09 second follow-up main study have been developed and will be refined from models for previous rounds of HSLS:09, ELS:2002, and other NCES studies, including BPS:12/14.

Response Likelihood Model. The response likelihood model will be run only once, before data collection begins. Using data obtained in prior waves that are correlated with response outcome (primarily paradata variables), we will fit a model predicting response outcome in the 2013 Update. We will then use the coefficients associated with the significant predictors to estimate the likelihood of response in the second follow-up main study, and each sample member will be assigned a likelihood score prior to the start of data collection. Exhibit B-10 lists the universe of predictor variables that will be considered for the response likelihood model.

During data collection, the response likelihood scores will be used as a “filter” to assist in determining intervention resource allocation. For example, cases that have a very high likelihood of participation may not be offered an incentive boost, since they are likely to participate without it. The response likelihood score can also be used to exclude cases with very low likelihood from the field interviewing intervention. We will also consider using the response likelihood score to adjust the classification of cases in the subgroups. For example, cases with very high response likelihood scores could potentially be treated as “ultra-cooperative” cases. The primary objective of the response likelihood model is to provide information that will inform decisions about inclusion or exclusion of targeted cases for interventions, thereby controlling costs.

Bias Likelihood Model. The bias likelihood model will be used to identify cases that are most unlike the set of sample members that have responded. As was done in the responsive design approach for the 2013 Update, the bias likelihood model will use only key survey and frame variables as predictors to identify nonrespondents most likely to reduce bias in key survey variables if converted to respondents. To calculate bias likelihood, we will run a logistic regression with the second follow-up response outcome as the dependent variable. The bias likelihood model will be run at the beginning of phases 3, 4, 5, and 6 for the calibration samples and at the beginning of phases 3, 4, 5, and 6 for the rest of the cases. (Modeling will be done on the combined sample [calibration cases and rest of cases] prior to phases 5 and 6.) We will then use the coefficients associated with the significant predictors to assign a bias likelihood score for each case. Because the set of respondents and nonrespondents is dynamic, the bias likelihood score for an individual case may change across the phases. The universe of candidate predictor variables have been selected due to their analytic importance for the study, and are presented in Exhibit B-11.

Exhibit B-10. Candidate Variables for the Main Study Response Likelihood Model

Data collection wave	Variables
Base year	Response outcome Response mode Early phase response status
First follow-up	Response outcome Response mode Early phase response status
Panel Maintenance and Address Updates	Panel maintenance response status Address update response status
2013 Update Survey	2013 Update response by student (not parent) Early phase response status Response mode Incentive amount (to control for the effect of incentives on response outcomes) Ever called in to the help desk Ever refused (sample member) Ever refused (other contact) Ever agreed to complete web interview Dual language speaker HS completion status indicator Gender
	Count of email addresses Count of phone numbers Count of addresses

Exhibit B-11. Candidate Variables for the Main Study Bias Likelihood Model

Data collection wave	Variables
Sampling frame	Race
	Gender
	School type
	Metropolitan area
	Geographic region
Base Year	Student’s educational expectations Parent’s education expectations Taking a Fall 09 math course Taking a Fall 09 science course Base year math assessment score
First Follow up	When Algebra 1 was taken Grade in Algebra 1 Student’s educational expectations Parent’s education expectations Grade in 2011-12 Location Dual language indicator Socioeconomic status indicator Repeated a grade? F1 math assessment score Attended a job fair? Toured a college? Taken a college class? Completed an internship? Performed work in job related to career goals? Searched internet/college guides for college options? Talked to HS counselor about after high school options? Talked with college admission counselor? Taken a college entrance exam prep course? Taking math classes in spring 2012?
2013 Update Survey and High School Transcript Collection	Sample member has high school credential Date of high school credential School characteristics of last-attended high school Dual-enrollment status/information
	Taking postsecondary classes as of Nov 1 2013 Sector of postsecondary institution as of Nov 1 2013
	Apprenticing as of Nov 1 2013
	Working for pay as of Nov 1 2013
	Serving in the military as of Nov 1 2013
	Starting family/taking care of children as of Nov 1 2013
	Attending high school or homeschool as of Nov 1 2013
	In a course to prepare for GED as of Nov 1 2013 Number of postsecondary institutions applied to
	Completed a FAFSA for teenager's education
	Did not complete FAFSA because did not want to go into debt
	Did not complete FAFSA because can afford college without financial aid
	Did not complete FAFSA because thought ineligible or unqualified
	Did not complete FAFSA because did not know how
	Did not complete FAFSA because forms were too time-consuming/too much work
	Did not complete FAFSA because did not know could
	Did not complete FAFSA because teen does not plan to continue education
	Currently working for pay
	Number of high schools attended
	Attended CTE center (flag)
	English language learner status
	GPA: overall GPA: English
	GPA: mathematics
	GPA: science
	Total credits earned
	Credits earned in academic courses Ever had a dropout episode

The goal of the bias likelihood model is not to accurately predict response, but to classify sample members’ current response rates along the dimensions represented by the predictor variables. As such, statistical significance should not be a determining factor in which variables are included in the model, rather the criterion should be that variable’s importance for HSLS:09. The threshold for identifying cases for targeted treatment will be based on an assessment of the bias likelihood score, the response likelihood score, and available project resources.

Evaluation of responsive design approach.

There are three elements to be evaluated in the proposed responsive design approach: (1) that sample cases that contribute to sample representativeness can be identified at the beginning of the third and subsequent data collection phases, (2) that interventions used during each phase of the data collection design are effective in increasing participation, and (3) that increasing response rates among the targeted cases will improve sample representativeness. We intend to examine these three aspects of the responsive design and its implementation for the HSLS:09 second follow up as follows:

1. Evaluate the bias likelihood model used to identify targeted cases. To assess whether the bias likelihood model successfully identifies nonresponding cases that are underrepresented on key survey variables, we will compare estimates within the categories of each model variable for respondents and nonrespondents at each phase. This comparison will highlight the model variables that exhibit bias at each phase and the relative size of the imbalance that remains to be reduced through the intervention.

2. Evaluate the effectiveness of each intervention in increasing survey participation. The second key component of this responsive design is the effectiveness of the targeted treatments in increasing participation. Experiments conducted with the calibration samples will allow us to assess the efficacy of the various treatments.

3. Evaluate the ability to increase sample representativeness, by identifying cases for targeted treatment. We will measure sample representativeness by comparing estimates on key variables for respondents and nonrespondents, at each phase of data collection and at the end of data collection. We will then be able to assess whether sample representativeness is improved over the course of data collection through the use of the targeted interventions for cases identified with the bias likelihood model.

References

Rosen, J. A., Murphy, J. J., Peytchev, A., Holder, T. E., Dever, J. A., Herget, D. R., & Pratt, D. J. (2014). Prioritizing low-propensity sample members in a survey: Implications for nonresponse bias. Survey Practice, 7(1), 1–8.

Pratt, D. J. (Invited Speaker). (2014, March). What is adaptive design in practice? Approaches, experiences, and perspectives. Presented at FedCASIC 2014 Workshop Plenary Panel Session, Washington, DC.

Pratt, D. J. (2013, December). Modeling, prioritization, and phased interventions to reduce potential nonresponse bias. Presented at Workshop on Advances in Adaptive and Responsive Survey Design, Heerlen, Netherlands.

1 Note that in Attachment 1, the subgroups are labeled as 1, 2, and 3 rather than A, B, and C.