Response to OMB Passback

MEMORANDUM OMB # 1850-0852 v.11 & 12

DATE: May 7, 2013

TO: Rochelle W. Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Elise Christopher

HSLS:09 Project Officer, National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: High School Longitudinal Study of 2009 (HSLS:09) 2013 Update and Transcript Main Study (2013) Response to OMB Passback

This memorandum provides responses to the questions posed by OMB on February 27, 2013 on the HSLS:09 OMB submission for the 2013 Update and high school transcript main study activities.

***Note: In April 2013, OMB and NCES agreed to remove all description of the responsive design and incentive approach from package materials so as to allow the clearance of the first panel maintenance while NCES and OMB continue to work on the details of responsive design and incentives. In May 2013, details of the responsive design and incentive plan were mutually agreed upon with OMB, and NCES revised all of the package materials to reflect the changes. As such, the clearance materials approved in May 2013 reflect the final study design and incentive structure, and they override the responses provided below. This document is included in the clearance package to reflect the passback and response history between OMB and NCES regarding this clearance.

1. Both BPS field test and B&B full scale are conducting experiments to determine the best way to use incentives in a responsive design context. OMB has been willing to allow ELS to use some of these techniques without experimentation given the small sample size and other justifications provided by NCES for the 3rd follow up. However, a priori, we would expect the HSLS proposal to look nearly identical to the BPS and B&B proposals, and/or to build on them. Rather, it looks like HSLS is operating as if these techniques were ready to be used in a production environment without experiments. Given the inconclusive evidence, at best, from the field test, and NCES’s posture on the other studies, we need to be convinced why the HSLS proposal shouldn’t be conducting experiments and building off of the postsecondary studies. Unless NCES can agree to develop an experiment and adapt the package quickly, we suspect that clearance by early March is infeasible. However, we are willing to clear the March panel maintenance activity soon (after items below are addressed) and have NCES resubmit the remainder of the proposal when ready, while these details are worked out.

NCES response: We have revisited our proposed plan in response to OMB’s passback and have revised Supporting Statement Parts A and B. A summary of the responsive design and associated experiments of other NCES studies and prior rounds of HSLS have been added in Part B.4, and a revised data collection plan including an experiment to determine the effectiveness of incentives is also described.

This response to the passback includes information on both the revised responsive design as well as the adaptations to the panel maintenance activities (Part A.9).

2. In general, it would be useful to have a succinct summary of the Field Test results. Right now, there are scattered implicit references to it, an assertion that “the materials” are not substantively changed from it, but then several deviations mentioned here and there, with no rationale for such changes. Results and a systematic treatment of changes would be very useful.

NCES response: A summary of the field test results has been provided in Appendix 5 of the OMB package, including outcomes from data collection, the responsive design activities, and the validation study to examine the reliability of responses provided by both students and parents.

Materials used to contact sample members in the 2012 Update field test were modified slightly for use in the 2013 Update main study. Changes were made to minimize redundancies in the letters. Transcript contact materials were modified to change the period of reference associated with the main study.

Revisions to the HSLS 2012 Update field test survey items are delineated in Part C.

3. Please provide more information about the response design model being proposed. As you may know, very few variables were significant in the models for the postsecondary studies, and few if any of those are likely to be relevant to this study’s participants. Therefore, we are unclear as to which variables are included in the model, which are significant, and how well the model overall is working.

NCES response: Table 1 below (Exhibit B-2 in the revised Part B) lists the candidate model covariates we propose for use in calculating the Mahalanobis distance scores, including survey variables (e.g., student’s expected education level), frame variables (e.g., school type: public, private), and some paradata (e.g., student’s base year and first follow-up response status). The same set of frame and paradata variables used in the field test will be considered for the 2013 model, given the encouraging results from the field test analyses.

In addition, survey variables with some level of imputation will be considered for the main study model. The field test could not include certain survey variables due to large numbers of missing values across key variables. Analytic results from the field test – presented in Appendix 5 – suggest that cases identified by the responsive design model for targeted outreach who eventually participated were different on several survey variables. Those variables are in the list of candidate model covariates for the main study responsive design model.

Table 1. Proposed variables to include in model to identify target cases

Source	Potential Variables
Survey variables	Student expected education level Timing of Algebra I coursetaking Base-year and first follow-up math assessment theta scores Enrollment status Parent level of education Race Gender Whether sample member remained at base year school as of first follow-up
Sample frame data	School type Metro area
Paradata	Whether sample member contacted the help desk Whether sample member logged in but did not complete the 2013 Update questionnaire Number of contact attempts in the early data collection period Whether sample member made an appointment to complete the interview Whether sample member told interviewer they would do the web interview Student base year and first follow-up response outcomes Parent base year and first follow-up response outcomes Response in the panel maintenance update(s) Student enrollment status at first follow-up Reason for prior student nonresponse (refusal, absent) if applicable Call counts in base year and first follow-up

Ultimately, the decision on which of these candidate variables to include in the final responsive design model for the HSLS:09 2013 update will be based upon careful examination of each variable distribution, and correlational analyses to exclude redundant information.

4. What were first follow up unconditional response rates for students and parents (separately)?

NCES response: The first follow-up unconditional response rates were 81.8 percent unweighted (82.0 percent weighted) for students and 72.4 percent unweighted (72.5 percent weighted) for parents. (Note that a subsample of 11,961 was selected for inclusion in the first follow-up parent data collection.)

5. What is the final unconditional response rate that NCES is projecting for this “update” activity?

NCES response: An 82 percent projected unconditional response rate for this “update” activity is projected with either the student or parent responding.

6. Taking these questions forward, does NCES believe that it will retain enough sample cases to meet its original targets?

NCES response: The required sample size established in the base year to meet the analytic objectives for HSLS:09, accounting for attrition across rounds of the longitudinal study, was 21,000 cases. A total of 23,415 sample cases are eligible for the 2013 Update, exceeding the required sample size by more than 2,000 cases.

An estimated 19,203 respondents (82 percent response rate) are anticipated from the HSLS:09 2013 Update. This decrease from the HSLS:09 base-year set of respondents (n=21,444) and first follow-up respondents (n=20,594) is not expected to have an appreciable influence on precision of the estimates. Additionally, the projected number of 2013 Update respondents exceeds, for example, the number of respondents in the Education Longitudinal Study of 2002 (ELS:2002) first and second follow-ups (approximately 15,000 and 14,200 respectively)¹.

7. On the “Address Update Paper Form” the sentence “Please remember that all information you provide will be protected” seems a little stilted and incomplete.

NCES Response: This sentence has been revised in the materials to match the language used in the Baccalaureate and Beyond Longitudinal Study address update forms and is included below.

8. On the “disclosure notice” that appears to accompany the transcript request form to schools, the follow statement appears: “Data will be combined to produce statistical reports for Congress and others. No individual data (e.g., names or addresses) will be reported.” I don’t recall seeing this specific formulation before. Has NCES used it successfully in other transcript studies or in other similar contexts? For various reasons, I’m not sure a “report to Congress” strikes the right tone in a confidentiality pledge especially when followed by a statement about “individual data” that seems more limited or weak than it could be.

NCES Response: The disclosure notice has been modified to the following:

DISCLOSURE NOTICE

Student record data for this student have been provided to RTI International, a nonprofit research organization and agent for the U.S. Department of Education, National Center for Education Statistics (NCES). This disclosure statement fulfills the requirements of 34 CFR 99.32(a) pursuant to the Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. 1232g).

The data were requested for the High School Longitudinal Study of 2009. NCES is authorized by federal law (20 U.S.C., § 9543) to conduct this voluntary study. Responses and student record information may be used only for statistical purposes and will not be disclosed, or used, in identifiable form for any other purpose, except as required by law (ESRA 2002, 20 U.S.C., § 9573). Data from these records will be combined with others into statistical summaries and tables. No individual data (e.g., names or addresses) will be reported.

OMB # 1850-0852
Expiration Date: <insert date>

9. Please provide more rationale for the $10 level for panel maintenance. As noted above, we have not been making ELS conduct experiments much of the time given so many operational issues and the age of the study. However, HSLS should not assume that it can offer the same incentives. If the $10 is the result of an experiment, please provide those results. If not, please propose an experiment or alternative justification.

NCES response: Two panel maintenance activities were proposed in the January 16, 2013 submission. One such activity is proposed to take place in March/April of 2013 in advance of the 2013 Update data collection. Sample members completed the first follow-up data collection less than a year ago, and this panel maintenance mailing will give parents an opportunity to update contact information while simultaneously informing sample members when the next data collection will commence. No incentive is proposed for the spring 2013 panel maintenance activity.

A second panel maintenance activity is proposed to occur between the “update” round and the second follow-up. The second follow-up is scheduled to occur in 2016, which would be 3 years after the “update” round, and at a time when the sample population is transient and mobile. A panel maintenance mailing between these two rounds is intended to enable the HSLS:09 study team to stay connected with sample members, especially those who left the base-year school before the first follow-up study and who we expect to be difficult to find. It is this activity for which a $10 incentive is proposed.

The incentive request is based on the success of an experiment conducted as part of the ELS:2002 third follow-up field test. In this experiment, half of the students in the field test sample were offered a $10 check if they or their parents confirmed or updated their contact information. No incentive was offered to the other half of the sample. A cost-benefit analysis was also conducted to evaluate the difference between the cost of the incentive offer and the difficulty of cases that responded. The impetus behind this analysis was determining if information was received from more difficult cases, as the benefit would be reduced if the “easy-to-track” cases were the ones to respond.

Overall, the $10 treatment group had a higher participation rate (25 percent) than the control group (20 percent, t = 1.90, p < .05). Further evaluation of the data indicates that the contact information provided largely new information not already in the study database; for 82 percent of the responding cases, at least one new address, phone number, or email address was provided for the student, parent, or both. Being able to make direct contact with the sample student during data collection saves time and costs, and is likely to increase interview participation. This panel maintenance was implemented for the main study sample for ELS:2002/12 based on the field test results. Forty percent of the ELS:2002/12 third follow-up sample participated in panel maintenance at some point. Among that group, 97 percent responded to the third follow-up survey compared to a 74 percent response rate among those that did not participate in panel maintenance.

Address Update Paper Form

High School Longitudinal Study of 2009 (HSLS:09)

Address Update Information

Thank you very much for your participation in the High School Longitudinal Study of 2009 (HSLS:09).  Please take a moment to confirm or update [your/StudentName’s] contact information. We are also requesting that you confirm or update contact information for [your parent/yourself] and provide contact information for one additional person. By updating our records, we will be better able to notify [you/him/her] when it is time for the next round of HSLS:09.  Please be assured that both NCES and RTI follow strict confidentiality procedures to protect the privacy of study participants. If you would like more information about the HSLS study, please visit http://surveys.nces.ed.gov/hsls/.

After you complete this form, please mail it back to RTI International using the enclosed Business Reply Envelope.

This update may also be completed on our secure website at [Website_Fil] by entering your study id: [ID_Fill] and password: [PW_Fill]. You may go online to indicate that the information provided in the form below is correct.

We greatly appreciate your help to make HSLS:09 a continued success!

1. [STUDENTS ONLY] May we send you a text message reminder when data collection begins?²

• Yes  Please enter your cell provider here: ___________________

• No

2. If [your/StudentName’s] current contact information is not correct below, please provide the updated contacting information on the right. If the contact information is correct, please check the box.

	[Your/StudentName’s] contacting information			Corrections and additions
	Name:	[StudentName]
	Address:	[StudentAddress]
	City	[StudentCity]
	State	[StudentState]
	ZIP	[StudentZIP]
	Home phone:	[StudentHome]
	Work phone:	[StudentWork]
	Cell phone:	[StudentCell]
	Primary e-mail address:	[Studentemail1]
	Alternate e-mail address:	[Studentemail2]
Check here if all information is correct

2. Please also provide the contact information for the parent most knowledgeable about [your/StudentName’s] education or career. Please also provide contact information for an additional person who will always know how to get in touch with [you/StudentName].

	Parent information			Corrections and additions
	Name:	[ParentName]
	Address:	[ParentAddress]
	City:	[ParentCity]
	State:	[ParentState]
	ZIP:	[ParentZIP]
	Home phone:	[Parenthome]
	Work phone:	[ParentWork]
	Cell phone:	[ParentCell]
	Primary e-mail address:	[Parentemail1]
	Alternate e-mail address:	[Parentemail2]
Check here if all information is correct

	Other contact information
	Name:
	Address:
	City:
	State:
	ZIP:
	Home phone:
	Work phone:
	Cell phone:
	Primary e-mail address:
	Alternate e-mail address:

4. [Are you/Is StudentName] currently enrolled in a college, trade school, or other higher education institution?

• Yes  Please continue with question 5

• No  You are finished with the address update. Thank you.

5. Please provide the name and address of the college, trade school, or other higher education institution that [you/StudentName] attend. Please do not use abbreviated school names (e.g. use Arizona State University, rather than ASU).

School Name:

Thank you in advance for making HSLS:09 a success!

Panel Maintenance Card

This panel maintenance update is planned to be sent in the spring of 2013, prior to the 2013 Update. We plan to send a greeting card-type mailing and enclose a hard copy update option. A Business Reply Envelope will accompany the hard copy form to facilitate returns.

Card text:

Thank you for your past participation in the High School Longitudinal Study of 2009 (HSLS:09). The next data collection update begins in June 2013. We look forward to speaking with you then!

To confirm or update your contact information, please log in to our secure website:

https://surveys.nces.ed.gov/hsls

Study ID = «userid»

Password = «password»<PW_ind>

Or you may return the enclosed form with your updated contact information.

Thank you!

RTI is collecting data as part of this study by the National Center for Education Statistics (NCES) of the U.S. Department of Education’s Institute for Education Sciences.

1 http://nces.ed.gov/pubs2008/2008347.pdf

2 Standard text messaging rates apply.

8