Follow-up to Full Scale Recruitment Change Memo

MEMORANDUM OMB # 1850-0852 v.8

DATE: December 10, 2010

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Laura LoGerfo

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: HSLS:09 First Follow-up Full-scale School Recruitment Procedures and Materials

               The High School Longitudinal Study (HSLS:09/12) is a nationally representative, longitudinal study of more than 21,000 9th graders as they proceed through secondary and postsecondary education. The first round of data collection occurred in the fall of 2009, and the next round of data collection is scheduled for the spring of 2012 when most students will be 11th graders. The full-scale data collection begins in January 2012, with the school recruitment activities beginning in January 2011 to allow a full year for securing cooperation and arranging logistics with each of the base year schools. The full-scale study will comprise all 944 schools that participated in the base year study. The full-scale study for the first follow-up (FFU) data collection will closely resemble the base year full-scale study in approach and burden, and will resemble the first follow-up field test in materials used.

This submission requests approval for an addition of a small “thank you” token for the field test schools and the recruitment materials, incentive amounts, and the respondent burden associated with recruitment for the full scale study.

“Thank You” Token for the Field Test Schools

In addition to the field test recruitment materials approved in August 2010 (OMB# 1850-0852 v.6), NCES is also proposing to send an educational $0.50 "thank you" token to each of the 24 field test schools for providing enrollment updates and testing the survey CD (as part of the survey administration). The token consists of a set of 3 physics-related comic books published by the American Physics Society about Lasers (coinciding with the 50th anniversary of when the laser was first demonstrated; released copies can be accessed at http://www.physicscentral.org/experiment/physicsquest/upload/spectra.pdf and http://www.physicscentral.org/experiment/physicsquest/past/upload/spectra2.pdf). The comic books feature a female superhero Spectra, whose super powers are laser based. The story line explains the physics behind lasers (the superhero’s parents are laser scientists and the villain, Miss Alignment, poses as the superhero’s substitute teacher for her high school physics class). The cost to the U.S. Department of Education is $0.50 per 3 comic books set, for a total of $12 for 24 sets. In January 2011, one set will be sent to each of the 24 schools recruited for the HSLS 2011 field test along with the test CD, essentially thanking the schools for their repeated collaboration, their efforts in testing the survey CD on their computer systems, and their upcoming participation in the field test.

This will be followed, a few months later, by mailing schools the consent materials and initiating data collection activities. The schools will receive the larger token (subscription or other selection; as approved under OMB# 1850-0852 v.6) after they will have completed their participation. Based on the experience in the field test, we will evaluate whether it would be beneficial to propose including the 3 Spectra comic books set in the full scale.

Recruitment Materials for the Full Scale Study

Minimal changes were made to the approved field test materials to reflect the main study follow-up schedule. The approach to student data collection will be the same: computer administration of both math assessment and survey in students’ schools. Schools will be asked to provide enrollment status information, a task already approved by OMB. This change memo requests clearance of the first follow-up full-scale recruitment procedures and materials (included as an appendix), as well as an approval of an incentive plan because incentive amounts are included in recruitment materials.

Explanation of Gifts/Payments

Because the incentives NCES would like to offer to the school, school coordinator, and participants are included in the recruitment materials, inserted below is an excerpt (the Supporting Statement Part A section 9) from the clearance package that is currently in the 60-day Federal Register public comment period, awaiting OMB’s approval in early 2011 (OMB# 1850-0852 v.7; FR 75 No. 197 p. 62806).

A.9 Explanation of Payment or Gift to Respondents

Incentives are proposed to maximize school and student participation within schools and to encourage students and parents to participate outside of school. Incentives are also intended to help improve the chances of study participation by non-responding sample members. The use of incentives provides significant advantages to the government in terms of increased overall response rates, timely data collection, and reduction of nonresponse bias. In turn, increased response rates result in decreased data collection costs.

The incentive structure requested for the HSLS:09 first follow-up is presented by respondent type in Exhibit A‑1 for the field test and Exhibit A‑2 for the main study. A break out of incentives by respondent type is provided in Exhibit A-3 for the field test and Exhibit A-4 for the main study. A description and rationale for each incentive is provided below.

Exhibit A-1. Incentives by respondent type proposed for field test

Respondent	Incentive/Honorarium
School	Subscription or equivalent ~$50
School coordinator	$100 plus $25 for ≥ 85% or $50 for ≥ 92% student participation
IT coordinator	$50
School reimbursement for costs incurred	Up to $100 as required by schools
In-school student	$10
Out Of School student (OOS)	$15 for completing questionnaire plus $10 for completing assessment. An additional $25 to pre-identified “low propensity to respond” cases after the early web data collection period expires.
Parent	none
School administrator	none
School counselor	none

Note: Student cases would be categorized as “low propensity to respond” prospectively based on their contact and response history in addition to their current enrollment status. The additional incentive for low propensity to respond cases would only be implemented after the three-week early web data collection period had expired, during which time the outbound calling efforts commence.

Exhibit A-2. Incentives by respondent type proposed for full scale

Respondent	Incentive/Honorarium
School	Subscription equivalent; list of choices ~$50
School coordinator	$100 plus $25 for ≥ 85% or $50 for ≥ 92% student participation
IT coordinator	$50
School reimbursement for costs incurred	Up to $100 as required by schools
In-school student	$10
Out of school student (OOS)	$15 for completing questionnaire plus $10 for completing assessment. An additional $25 to pre-identified “low propensity to respond” cases after the early web data collection period expires.
Parent	$20 for ”difficult cases” only
School administrator	none
School counselor	none

NOTE: Student incentives for out-of-school data collection would be applied as described in the Note for Exhibit A-1. In the case of parents, incentives are only offered to the subset of the population who become “difficult cases” (estimated to be 20% of parents at $20) for the main study. No field test parent cases will receive incentives given that the necessary yield for the field test does not justify such an incentive. Parent response rate requirements for the main study, though, combined with the positive HSLS base year experience with the incentive experiment justifies the $20 incentive for “difficult cases.”

Exhibit A-3. Incentives by type of case and data collection phase for field test

Type of case and phase	% of sample	Response rate	% of respondents by phase / of all student respondents	Number of respondents	Survey incentive amount	Additional assessment incentive	Total incentive amount
In-school student	83%	80%	100% / 71.2%	398	$10	NA	$10
Out-of-school student*	17%	80%
Early Web			30% / 8.6%	48	$15	$10	$25
Production – high prop.			50% / 14.5%	81	$15	$10	$25
Low Propensity.			20% / 5.7%	32	$40	$10	$50
Parent	100%	48%
Early Web			30%	90	NA	NA	NA
Production			50%	150	NA	NA	NA
Difficult			20%	60	NA	NA	NA

Note: In-school nonrespondents will be contacted out-of-school and are included in the number of respondents, but are not reflected in the percent of sample to avoid double counting of sample members. High Prop. refers to high propensity of response cases and low prop. refers to low propensity of response cases. Percent of sample refers to the percent of the overall sample for each category (i.e., in-school student, out-of-school student, and parent), and the percentages associated with the data collection periods are the percent of responding sample members to participate within each data collection period.

Exhibit A-4. Incentives by type of case and data collection phase for main study

Type of case and phase	% of sample	Response rate	% of respondents by phase / of all student respondents	Number of respondents	Survey incentive amount	Additional assessment incentive	Total incentive amount
In-School Student	83%	90%	100% / 78.7%	18,829	$10	NA	$10
Out-of-school student*	17%	80%
Early Web			30% / 6.4%	1,531	$15	$10	$25
Production – high prop.			50% / 10.7%	2,551	$15	$10	$25
Production - low prop.			20% / 4.3%	1020	$40	$10	$50
Parent	100%	80%
Early Web			30%	2,748	NA	NA	NA
Production			50%	4,580	NA	NA	NA
Difficult			20%	1,832	$20	NA	$20

Note: In-school nonrespondents will be contacted out-of-school and are included in the number of respondents, but are not reflected in the percent of sample to avoid double counting of sample members. “High prop.” refers to high propensity of response cases and “low prop.” refers to low propensity of response cases. Percent of sample refers to the percent of the overall sample for each category (i.e., in-school student, dropout, etc) and the percentages associated with the data collection periods are the percent of responding sample members to participate within each data collection period.

A few of the incentives presented in the tables above were approved as part of the HSLS:09 First Follow-up School Recruitment Procedures and Materials change request (1850-0852 v.6) in August 2010. As in the base year, school coordinators will be offered an honorarium of $100 with the opportunity to earn an additional $25 for achieving at least an 85% student participation rate or an additional $50 for achieving a student response rate of 92% or better at the school. A modest token of appreciation to the schools, with an estimated value of $50 per school, in the form of a choice of 1-year science- or math-related magazine subscriptions for the school media center, was approved for the field test, with the understanding that during the field test recruitment effort, we will ask schools to suggest additional, alternative low-cost options that are meaningful to the schools, in order to develop a list of five options to present to schools during the main study to assist with the school recruitment/retention effort for the first follow-up. Lastly, as in the base year field test, a $10 incentive was approved for first follow-up field test in-school student respondents.

Incentives for students. The use of a $10 monetary student incentive was approved by OMB for students participating in in-school sessions for the base year and first follow-up field test (OMB# 1850-0852 v.2 and v.6). We request that the same incentive be offered to students participating in-school during the main study data collection. Most participants in the HSLS:09 first follow-up will be nearing the end of their junior year of high school, making them similar to high school seniors for whom research has demonstrated the importance of incentivizing to participate in voluntary research studies (National Commission on NAEP 12^th Grade Assessment and Reporting, 2004; National Research Council 2003). In the base year main study, 9th graders were given a goody bag filled with education supplies worth an estimated $5. We propose to give students a $10 cash incentive since it is anticipated that a cash incentive will be more positively received by upperclassmen than a token incentive. To support this point, an experiment conducted during the ELS first follow-up field test found that high school seniors were more likely to participate when receiving a $20 cash incentive (95.2% student response rate) than a token incentive (86.8% response rate). In addition, the cash incentive responds to the increased student reluctance to leave class for 90 minutes to participate in voluntary research that we encountered in the HSLS:09 base year data collection and offsets the perceived stress of missing class to take another assessment. Finally, the $10 incentive should help increase response rates for the in-school session, thus reducing the number of students requiring the costlier Web, CATI, or Field follow-up.

It is anticipated that approximately 75% of students will be available to participate in the in-school data collection for the HSLS:09 first follow-up. An estimated 8% of students will be enrolled in the base-year school but will be absent or unable to participate in the in-school session and will need to be contacted for an out-of-school administration. The remaining 17% of students will no longer be enrolled in the base-year school and will need to be contacted out of school for the study. Our experience on the ELS:2002/04 first follow-up demonstrated that additional incentives were necessary to gain cooperation from students participating outside of school. We propose to offer a $15 base incentive for students completing the questionnaire outside of school. For the first time in the series of high school longitudinal studies, NCES also will be administering the student assessment outside of school in addition to the questionnaire. We propose to offer students an additional $10 for completing the assessment, for a total of $25 to students who complete both components of the study.

Some students will be more reluctant than others and will be classified as having a “low propensity to respond” to the HSLS:09 FFU. Among the most serious problems created by nonresponse is the bias that can lead to inaccurate estimates and can compromise data quality. It is common for survey organizations to address nonresponse bias by attempting to increase the survey response rate, which is usually accomplished by pursuing the nonresponse cases most likely to be interviewed. However, this approach may not be successful in reducing nonresponse bias even if higher response rates are achieved—in fact, nonresponse bias could even be increased by adding more cases that are similar to those that have already responded (Merkle and Edelman 2009). If low propensity (i.e., difficult to complete) cases are brought into the response pool, we anticipate that this will not only increase the weighted response rate and result in less biased survey estimates.

RTI is currently undertaking an initiative, modeled on the Responsive Design methodologies developed by Groves (Groves and Heeringa, 2006), to develop new approaches to improve survey outcomes that incorporate different responsive and adaptive features. Although still in the development phase, RTI has implemented several of these procedures on recent studies and have published preliminary results (Rosen et al., in press; Peytchev et al., 2010). RTI’s approach aims to reduce nonresponse bias by using multiple sources of data to produce models that estimate a sample member’s response propensity prior to the commencement of data collection. After empirically identifying sample members with the lowest response propensities, the field team targets those cases with interventions (such as a higher incentive, prompting, use of a select group of interviewers that are specially trained in refusal conversion techniques, whatever may be appropriate for the sample) in an attempt to maximize the average response propensity.

The ultimate goal of the approach is to minimize bias by targeting the cases that, based on the available data, are expected to have a low likelihood of response and a high likelihood of contributing to nonresponse bias. Because the propensity-modeling plan considers respondent information (including survey response behaviors and socio-demographic characteristics) more inclusively and broadly, it is expected that it will also be able to determine which cases would potentially contribute most to minimization of bias in estimates, and ensure that these cases receive priority, via an effective treatment.

Criteria reviewed to determine response propensity classifications will include participation in the base year study, enrollment status (e.g., dropout, transfer), existence of contact information (e.g., mailing address, telephone number, email address), parent participation in base year, type of school, and school locale. Other variables may be added as the model is finalized, though race/ethnicity, gender, income and socioeconomic status will not be included in the model as the model focuses more on history of participation. A low propensity to respond will be determined initially at the start of data collection. The propensity model will be refined during the early web period based on the actual early web response. The identified “low propensity to respond” cases would be offered an increased incentive, but only after the three-week early web data collection period has expired and outbound telephone contacts have commenced. After the three week early web data collection period, “low propensity to respond” students would be offered $40 to complete the questionnaire and $10 to complete the mathematics assessment for a total of $50. We propose to implement this incentive immediately following the early web data collection period to ensure the opportunity for all sample members to respond in the early phase and to better determine which cases will have a lower propensity to respond. The $50 targeted for low propensity cases (with an additional $10 for assessment completion) provides a strong incentive level to encourage cooperation among the set that would otherwise potentially increase bias through nonresponse. These incentive amounts are comparable to those offered to difficult cases in the ELS first follow-up study, when challenges experienced with obtaining their participation resulted in the need to request additional incentives during the data collection period to achieve target response rates. At that time, OMB approved an increased incentive for the difficult cases from $40 to $60 that resulted in a final response rate of 78%, as compared with an overall response rate of 87%. The increased incentive level helped to generate a 20% increase in the overall out-of-school response rate in the last 8 weeks of ELS data collection.

Prior high school longitudinal studies, such as the ELS:2002/04 FFU, have shown that low propensity cases, which include large percentages of dropout students (an extremely important policy-relevant group), are particularly difficult to reach and to convince to participate. They require added effort to secure their participation, and respond positively to an increased incentive and more intensive and focused outreach efforts. The HSLS:09/12 plan is designed to minimize nonresponse bias, achieve sufficient yield for analytic needs, limit the number of cases requiring more costly follow-up steps, and accommodate the additional assessment component for the out of school cases among low propensity cases, in particular.

All other students participating outside of school will be offered the $15 incentive for completing the questionnaire with the additional $10 for completing the mathematics assessment for a total of $25.

IT Coordinator. During the base-year field test, it was determined that an IT Coordinator was necessary at each school to facilitate the use of the school computer labs and to ensure compatibility between the school’s computers and network connectivity and the Sojourn CD which provides a secure connection between the school’s computer and the NCES website for data collection. In the base-year main study, OMB approved a $50 honorarium for IT Coordinators who facilitated the in-school data collection. This honorarium proved extremely effective to enlist the assistance of an IT Coordinator in the schools. We propose to continue to offer the $50 honorarium to IT Coordinators for the First Follow-Up Study.

Incentive for counselors. No incentive is proposed for the counselors to complete their questionnaires. This precedent was set in the base-year study which realized high counselor response rates without the use of monetary incentives. Counselors would typically provide the information requested in the questionnaire as well as the administrative records as part of their normal duties. Because of the nature of the study, NCES suspects that many school principals will designate a counselor to perform the school coordinator duties, in which case the counselor will receive the coordinator honorarium as was previously approved by OMB.

Incentive for school administrators. NCES has achieved high response rates for the school administrator questionnaire on the HSLS:09 base year and on ELS:2002 and the ELS:2002 follow-up conducted in 2004. Based on past experience, no incentive will be offered for the school administrator questionnaire on HSLS:09.

Incentives for parents. For the parent data collection, we do not request an incentive for the field test, but propose that one be offered to a subset of parents for the full-scale study. The field test parent data collection will consist of a small set of parents with a low response rate expectation to test the questionnaire and procedures while containing costs. The full-scale study, however, will comprise a subsample of parents for which achieving high response rates is critical. In the base year, we experienced challenges achieving high parent response and used an incentive experiment to determine the most effective incentive threshold. Based on the results of the experiment (submitted to OMB earlier this year), we propose to offer a $20 incentive for nonresponse follow-up for the most challenging cases. The decision to offer an incentive for parents will be determined by rules similar to those implemented in the base-year incentive experiment, consisting of sample members who have not responded after receiving a high number of calls from RTI, refusals, and sample members for whom we have a good address but no good phone number. Given the two year lapse of time between data collections and the effectiveness of the experiment, using these conditions to dictate timing for offering incentives to parents should be effective for the first follow-up study.

Reimbursement of reasonable school expenses. In some cases there may be requests from schools for reimbursement of expenses associated with the testing session. For example, a number of base-year schools requested reimbursement for the production of enrollment lists and three others asked for reimbursement to keep the school open for testing sessions that occurred outside of normal school hours. Such cases will be reviewed by project staff on an individual basis and will be approved if the request is deemed reasonable.

Burden

For the recruitment effort, the respondent burden is for school district staff and the school coordinator. School district staff will review the notification letter, and a subset of the school districts will review a research application. Additional burden for the school coordinator will be realized at the data collection stage, which is included in the complete full-scale OMB package along with burden estimates for survey respondents (students, parents, school administrators, school counselors). Estimated burden for district personnel and the school coordinator for school recruitment activities is shown in Exhibit A-5. The cost to district/school staff to coordinate the data collection effort is estimated at $20 per hour. The total estimated cost is $80,980 for the full-scale recruitment activities.

Exhibit A-5. Estimated burden for first follow-up full-scale recruitment

Respondent	Number of respondents	Average burden/response	Range of response times	Total burden (hours)
School district: review notification letter	662	3 minutes	2-4 minutes	33
School district: application review	60	240 minutes	210-270 minutes	240
School coordinator: pre-data-collection logistics	944	240 minutes	120-360 minutes	3,776
Total	1,666			4,049

Methods to Maximize Participation

Procedures for maximizing response rates at the institution level are based on successful strategies and experiences from the base year of HSLS:09, predecessor studies (e.g., ELS:2002), and other similar studies. In this section, methods for maximizing response rates for school recruitment are discussed. Methods for maximizing response rates among students, parents, and school staff are discussed in the complete full-scale submission to OMB.

Achieving high participation rates on voluntary school–based research studies has proven increasingly difficult in recent years. Recent experience has shown that many schools already feel burdened by mandated “high stakes” testing and, at the same time, are hampered by fiscal and staffing constraints. Moreover, there are roadblocks not only at the school level, but also at the district level, where research studies must comply with stringent requirements and must submit formal detailed applications (similar to IRB applications) before schools can even be contacted.

Additionally, in the 2009 base year data collection of HSLS:09, a large number of schools rescinded their agreement to participate during the recruitment and data collection periods due to various factors. The keystone of the plan to work with school districts and schools in this round is to build upon relationships developed with school and district personnel in the base year, and to demonstrate the importance of the study while maintaining flexibility in negotiations with school districts and schools.

Sample materials to be sent to districts and schools are provided in appendix A. Recruitment will commence with a notification letter to school districts. The letter will thank the district for their support in the base year and inform them that base year participating schools will be contacted for the first follow-up activities. Where a district requires a formal application to conduct research in the schools and the existing HSLS:09 application has expired, a renewal application or new application will be submitted. Once approved by the district, schools will receive a letter reintroducing the study, inviting the schools to participate, and initiating the enrollment status update.

Within a few days of receiving the materials, a trained recruiter will contact the school district or school to discuss their participation in the study. Recruiters are hired for their knowledge, skill, and articulation and proven ability to develop relationships with district and school staff that will foster participation throughout the in-school follow-ups for the longitudinal study.

As much as possible, burden will be shifted from the school to HSLS:09 contractor staff. Possible ways of shifting the burden include scheduling survey administrations to fit the school calendar, mailing consent forms directly to parents, providing compensation for time and/or help in completing forms, offering a session administrator to come to the school to compile sampling information, and having a session administrator coordinate all aspects of survey day (e.g., posting reminders, processing consents, and gathering students). These options have proven helpful in the HSLS:09 base year and in similar studies to gain cooperation in schools that express scheduling, burden, or staffing concerns.

In the full scale study, students from schools that decline to participate in the in-school component of the study will be contacted outside of school to complete a questionnaire and assessment via web, telephone, or field interview. RTI will also attempt to collect a school administrator questionnaire and a school counselor questionnaire at schools that decline an in-school session for students.

Based on information obtained as part of the Enrollment Status Update, it is possible that a small set of schools may be added to the sample as “new” or “convenience” schools. New schools are defined as schools that have newly opened due to a change in the composition of the base year school (e.g., a base year school has been split into multiple schools) where a significant percentage of the students from the base year school have been moved en masse to the new school. New schools will be asked to participate following the same protocol as base year schools. Convenience schools are schools that may be contacted if four or more students from one or more base year schools have transferred to a particular school that was not in the base year sample. When it is determined that a transfer school has enough base year students enrolled to warrant contact, RTI will invite the school to conduct an in-school session with base year students who have transferred to the school. School staff will not be asked to complete a questionnaire at convenience schools.

References

Groves, R. M., & Heeringa, S. (2006). Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169(Part 3), 439-457.

Merkle, D. M., & Edelman, M. (2009). An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error. Survey Practice, (March)

Peytchev, A., S. Riley, J.A. Rosen, J.J. Murphy, and M. Lindblad. (2010). Reduction of Nonresponse Bias in Surveys through Case Prioritization. Survey Research Methods, 4(1), 21-29

Rosen, J., Murphy, J. J, A. Peytchev, Riley, S., & Lindblad, M;. The Effects of Differential Interviewer Incentives on a Field Data Collection Effort. Forthcoming in Field Methods.