Part B-C HSLS-09 First Follow-up 2012

This section describes the target universe for this study and the sampling and statistical methodologies proposed for the HSLS:09 first follow-up 2012 main study. We will also address suggested methods for maximizing response rates and for tests of procedures and methods, and we will introduce the statisticians and other technical staff responsible for design and administration of the study.

B.1 Target Universe and Sampling Frames

The base-year target population for the HSLS:09 full-scale study consisted of 9th-grade students in public and private U.S. schools that include 9th and 11th grades. The target population for the first follow-up is the same 9th-grade cohort studied in the base year (2009-10).

B.2 Statistical Procedures for Collecting Information

B.2.a School Sample

We will return to all 944 participating base year schools with the goal of securing cooperation for the first follow-up from as many as possible.

B.2.b Student Sample

In the main study, the same students who were sampled for the 2009-10 data collection, regardless of their base year response status, will be recruited to participate in the 2012 data collection.

B.2.c High School Counselor and Administrator Samples

The principal and lead counselor at each of the 944 schools will be asked to complete a first follow-up questionnaire.

For HSLS:09 student respondents that transfer from the base-year school to a different school (that is not in the sample), the principal or appropriate designee will be asked to complete an abbreviated administrator questionnaire. The transfer schools will be identified from information provided by schools (enrollment status updates), students (on the student questionnaire), and parents (on the parent questionnaire). All identified transfer schools of respondents will be approached – i.e., there will not be a subsampling of transfer schools.

Collecting this information will allow comparisons between sampled schools and transfer schools, considerably enriching the analytic utility of the data, which, without this transfer school administrator survey, would have relied on very basic CCD and PSS data. The administrator data will be appended to the student-level data for the associated students, so that the school characteristics as of the first follow-up will be available for analysis. Having such information for transfer students will fill a gap of missing data for more than 10 percent of the student sample.

Given that some schools will not be identified until the end of student-/parent- data collection in August 2012, the administrator survey collection will continue until October 2012. As noted in Part A of this submission, an estimated 1,875 transfer school administrators will be identified and contacted and an estimated 70% (1,312) will complete the abbreviated administrator questionnaire. Attempts will be made with all transfer schools identified

The subset of administrator questionnaire items that will be asked on the abbreviated version are noted with double-asterisks (**) in Appendix 4 (Administrator Questionnaire) of this submission.

B.2.d Parent Samples

Contextual information was obtained from one knowledgeable parent or guardian for many but not all of the HSLS:09 base-year sample students. For the first follow-up main study, we will select a random subsample of HSLS:09 base-year parents as a cost-saving measure. The subsample will be selected from all base-year parents, regardless of their base-year response status, within subsampling strata defined by the cross of the first and second-stage design strata:

school type (public, private-Catholic, private-other);
10 augmented-sample state (public schools only);
the school’s geographic region in the United States (Northeast, Midwest, South, West);
the school’s geographic urbanicity in the U.S. (city, suburban, town, rural); and
student race/ethnicity (Hispanic, Asian, Black, Other).

Selecting parents regardless of their base-year response status avoids the potential nonresponse bias effect introduced into the estimates if only the base-year respondents were subsampled. The parent subsampling rates will be equivalent to those developed for the base-year main sample design and verified to meet the following objectives:

overall parent sample size of approximately 11,450 to yield at least 7,500 interviews;
minimum sample size for key reporting domains (e.g., Asian students, students in private schools) given desired levels of precision;
minimum inflation to the unequal weighting effects (i.e., design effect of the weights) within larger domains; and
maximize coverage of the contextual information for the student target population.

In addition to the primary sample of 11,450 parents, approximately 1,700 parent records¹ will be randomly selected as a precaution and released only if, for example, student eligibility rates are lower than expected or if preliminary analyses conducted during data collection indicate significant levels of nonresponse bias and/or low precision.

B.2.e Weighting

Currently we are planning two sets of longitudinal weights for the HSLS:09—one set for analyzing gross change in the student data such as algebra assessment scores from the base year to the first follow-up (student weight), and one set for analyzing change that includes contextual information on the student’s home life obtained from the student’s parent/guardian (home-life weight).

Three weight adjustments are proposed for the longitudinal weights. These adjustments are similar to those implemented in the base-year study and other NCES longitudinal studies such as the Education Longitudinal Study of 2002 (ELS:2002). First, the home-life contextual weights will be adjusted for the parent subsampling in the first follow-up. Second, the weights will be adjusted using non-linear models to account for differential unit nonresponse to minimize the potential biasing effects associated with less than perfect response. Variables used in the nonresponse weight adjustment will be identified from a larger set as having an association either with a set of key estimates from the study or with the pattern of nonresponse through a CHAID (Chi-square automatic interaction detection) methodology. Third, the weights will be calibrated to base-year counts to minimize coverage error and excessive variation in the weight values. Note that the nonresponse and calibration adjustments will be implemented with SUDAAN’s PROC WTADJUST (2008) to include main effects and interaction terms in the associated models and to control for extreme weight adjustments. The utility of the adjusted weights will be examined through evaluation of the design effect (precision) for important study variables and nonresponse bias analyses. The bias in , an estimate calculated with the respondent data, is estimated by , where is the estimate calculated with all sample data. The weight adjustments will be revisited if the analysis results are not comparable to the base-year analysis.

B.2.f Variance Estimation

For variance estimation, sets of 200 balanced repeated replication (BRR) weights will again be created for the HSLS:09 first follow-up study. The BRR weighting process will replicate the full weighting process and will use procedures developed for a number of other studies, including HSLS:09 base year, Education Longitudinal Study of 2002 (ELS:2002), and National Postsecondary Student Aid Studies (NPSAS). In addition, analysis strata and primary sampling units (PSUs) created from the sampling PSUs (i.e., schools) during the base-year study will be included on the HSLS:09 restricted-use data files to accommodate analysts wanting to use Taylor series rather than BRR variance estimation.

B.2.g Imputation of Missing Data

Missing values due to item nonresponse will be imputed for a small set of variables only after the data have been cleaned to ensure high quality values for this procedure. Imputation will be performed for items commonly used to define analysis domain (such as SES, which will be determined only after the instruments have been finalized) items that are frequently used in cross-tabulations, and items needed for weighting. Missing values for HSLS:09 questionnaire items identified for this task will be imputed first (if applicable) using logical imputation (e.g., gender imputed based on name of the respondent) and second through a weighted sequential hot deck procedure implemented in SUDAAN’s PROC HOTDECK. By incorporating the sampling weights, this method of imputation takes into account the unequal probabilities of selection in the original sample while controlling the expected number of times a particular respondent’s answer will be used as a donor. As with the base-year study, multiple imputation will be reserved for important continuous variables such as the math achievement score calculated from an adaptive test administered to the sampled students.

B.3 Methods for Maximizing Response Rates

Procedures for maximizing response rates at the institution and respondent levels are based on successful experiences in the base year study, first follow-up field test, and in predecessor and other similar studies. In this section, methods for maximizing response rates for students, parents, and school staff are discussed. Plans for maximizing response rates for school recruitment were presented and approved by OMB in an earlier submission (OMB# 1850-0852 v.6).

Student. The majority of students will participate in HSLS:09 during an in-school session. A subset of the students selected to participate in the base-year study will no longer be enrolled in the base-year school at the time of the first follow-up and will therefore be contacted to complete the questionnaire and assessment via Web, CATI, or CAPI. Methods for maximizing response rates among students participating in-school will be discussed, followed by methods for maximizing response rates for students participating out-of-school.

Ensuring a high student response at each school begins several weeks prior to the student session. Session administrators will work closely with the school coordinators to coordinate the logistics of the sessions and notify students about the sessions. Because the sampled students are dispersed across multiple classes, there is a heavy burden on the school coordinator to inform students about the session, distribute parental consent materials, and ensure that students arrive at the prescribed location at the scheduled date and time. Session administrators will assume as much of this burden as is possible and permissible by the school.

From past experience, ensuring that students are made aware of the session is the most critical aspect of making sure they arrive at the session at the scheduled time. Despite receiving the consent form to take home, students do not necessarily distinguish the form from other materials they take home, and they often forget about the session without frequent reminders. To help remind students about the sessions, the study will implement options such as handing out postcard reminders to the HSLS:09 students a day or two prior to the session, notifying the teachers of selected students from whose classes students will be pulled out for HSLS:09, asking the school coordinator to make an announcement on the PA system, and having the Session Administrator visit a few days prior to the session and convene a brief meeting of the student sample members to encourage participation. Parent contact information will be collected from each school from which the parent survey will be conducted. If phone numbers are provided, the Session Administrator will contact parents a day or two prior to the session to remind the students when they should arrive.

Each week, project staff will conduct group strategy calls with the Session Administrators to discuss the status of the schools with test dates scheduled for the coming 2 weeks. The purpose of these conference calls is to learn about the preparedness of each school for the student session, identify any concerns about anticipated response rate or computer capabilities at the school, provide a forum for brainstorming solutions to anticipated problems, and share success stories and lessons learned from other schools. Project staff will follow up frequently with Session Administrators who report problems or concerns with the preparations for student sessions at particular schools.

Plans for student incentives in the first follow-up main study were described in Part A. Each student participating in school will receive $10, as was done in the first follow-up field test. Offering an incentive to students participating in-school will help to maximize student participation in-school, which is a lower cost option than following students outside of school to participate in the follow-up study.

Students who have left the base-year school or who are absent for the in-school session will be contacted outside of school to participate in the study via Web or CATI administration. Parental consent is required before the student takes part in the study. We plan to collect parental consent in one of three ways. Parents will receive a letter asking for permission to have their student participate. Parents will be encouraged to hand students an enclosed envelope which has the letter to the student and credentials to log into student survey. Parents will also have the option to log into the study website and provide consent online for the student to complete the questionnaire. Finally, parents may provide consent verbally to a telephone interviewer or field interviewer for their teenager’s participation.

Student invitation materials will be sent through the parents until parental permission is provided online or by phone. Once parent permission is obtained, subsequent reminder notices will be sent directly to the student. Telephone interviewers will prompt students by phone to complete the questionnaire and assessment, with the opportunity to complete the questionnaire by phone. A small number of students will be contacted for CAPI administration if we are unable to reach them for a Web or phone interview.

Our experience on the HSLS:09 first follow-up field test and the ELS:2002/04 first follow-up showed that it is more challenging to obtain high response rates among students participating outside of school than their in-school counterparts. We also learned in the HSLS:09 first follow-up field test that students who have left their base year school by the time of the first follow-up study were the most challenging students from which to elicit a response. We further expect that students with the lowest likelihood to respond would be students who have dropped out of school. Gaining the cooperation of this group is critical, as dropouts are a group of particular interest in the research and policy communities. For that reason, we propose to target students who have left their base year school. We anticipate that this will increase the response rate for this group as well as increase the weighted response rate overall. In addition, we predict that this will result in less biased survey estimates because the focus will be on cases that are dissimilar from those that have already responded (Merkle and Edelman 2009). Because this information will be derived from enrollment status information provided by the school in advance of the data collection period, we plan to use this information to determine the set of students to target in lieu of developing a formal propensity model for the first follow-up main study.

At the start of the data collection, all students contacted to participate outside of school would be offered $15 to complete the questionnaire and $10 for completing the mathematics assessment, for a total of $25. After the three-week early web data collection period expires, those students identified as having left the base year school would be offered $40 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $50. All other students (including those who missed the in-school session) would continue to be offered the original incentive of $15 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $25.

Parent. Parents of base-year sampled students received a panel maintenance letter in the fall of 2011 indicating that they would be contacted to participate in the first follow-up in the spring of 2012. There will be several additional opportunities to interact with parents to encourage their participation in the study. The parental consent form will be sent home with the students several weeks before the student session, and the letter will mention that the parent interview is forthcoming for a subsample of parents. A letter will be sent to the parent via e-mail and U.S. mail to initiate the parent interview, providing a URL and credentials for the web instrument and a telephone number that can be used for a telephone interview. If a telephone number is available, the SA will contact the parent to remind him or her of the student session, and will take the opportunity to build a relationship with the parent and encourage participation from both the student and parent. Parents who do not complete the web instrument will be followed up via CATI (telephone), with CAPI (in-person) data collection conducted as a last resort. An abbreviated paper-and-pencil version of the questionnaire will be available for parents who do not have a telephone or internet access and/or as a last resort. The parent interview will be translated into Spanish to accommodate limited English proficient and nonproficient parents. Less than one percent of parents were excluded in the base year due to other language needs, thus it is not proposed to translate the interview into other languages.

In the first follow-up field test, we used propensity modeling to identify the parent cases with a low propensity to respond and implemented CAPI data collection for these cases immediately after the three-week web data collection period. We observed that this resulted in a higher parent response than those that went to CATI, but the sample size was too small to determine the effect on nonresponse bias. In the analysis of field test response rates, it was also determined that it was most challenging to gain participation from the parents of students who no longer attend the base-year school. Despite the inability to determine the effect on nonresponse bias, we have demonstrated success developing propensity models for longitudinal studies. Results from the ELS:2002 third follow-up field test study are still in progress, but preliminary analyses from that field test showed that the propensity model had a greater predictive power (with a higher R² value) than models run for the cross-sectional postsecondary studies.

For the main study, we propose to implement a propensity model that builds upon lessons learned from the field test. We plan for all parent cases to go through the three-week web data collection period and then to move to outbound CATI data collection. The propensity model would be applied at the end of the first three weeks of CATI production, after which the lowest propensity to respond cases will be worked via CAPI. It is expected that the parents of students who have left the base-year school would be prioritized in the set that is worked in CAPI. More information about the proposed propensity modeling can be found in Part G.

In the base year parent data collection, an experiment demonstrated that a $20 incentive proved effective in significantly increasing parent response among the most challenging cases. We propose to implement a similar incentive for the first follow-up study. The decision to offer an incentive for parents will be determined by rules similar to those implemented in the base-year incentive experiment (OMB # 1850-0852 v.4), consisting of sample members who have not responded after receiving a high number of calls from RTI, refusals, and sample members for whom we have a good address but no good phone number. This method of determining timing of offering an incentive was effective in the base year experiment and is proposed for the follow-up study as well. Toward the end of the data collection period, a mailing will be sent to nonresponding parent sample members who meet the incentive criteria to offer the $20 incentive. In the last month of data collection, we propose to send, via overnight-express delivery, a hard-copy abbreviated questionnaire to parents to whom we have a good address but no good phone number, along with a $5 pre-paid incentive. The letter will offer the parent an additional $20 upon completion of the full questionnaire, or indicate that the hard-copy abbreviated questionnaire could be completed without receiving an additional incentive.

School Staff (School Administrators and School Counselors). School staff will receive a letter to initiate their questionnaire at the start of the data collection period. The Session Administrator will work with the school coordinator to prompt school staff to complete their interview. While at the school, the Session Administrator will prompt for any outstanding staff questionnaires. If the questionnaires still have not been completed by 1 week after the session(s) in the school, CATI follow-up will commence. Schools that decline to participate in the student component of the first follow-up will still be contacted to complete the school staff questionnaires.

As noted above, transfer school administrators will be contacted after they are identified with student respondents. Procedures will be similar to those described for base-year school administrators. To minimize burden while maximizing analytic utility, the transfer school administrators will be asked only to complete the abbreviated form of the administrator survey, questions that may be answered by any knowledgeable school official, not necessarily the principal. To allow sufficient time for participation by schools identified late in the data collection period, the school administrator data collection will be conducted through October 2012.

The approach for gaining cooperation from transfer school administrators includes a series of mailings and e-mails with a single follow-up telephone call to prompt a subset of cases deemed of particular importance. Particular attention, including the more-intensive telephone prompting effort, will be provided to transfer school administrators who represent multiple sampled student respondents (i.e., schools to which more than one student respondent has transferred) since the administrator data will be linked to each associated student at the given school. Additionally, we will monitor response rates and precision estimates during data collection – and adjust case-prioritization if needed – with the intent of achieving stable estimates and low bias.

A non-response bias analysis for the administrator questionnaire generally (not just for transfer school administrators) will be conducted should the contextual response rate fall below 85 percent using the methodology detailed in part B, section 2.e. This contextual response rate is defined as the (weighted) proportion of responding students linked to the questionnaire responses from their school administrator. By including administrator data from transfer schools as well as sampled schools, the HSLS:09 data will contain a more comprehensive representation of the school environment for the student cohort as of 2012. Weighting and imputation procedures will be investigated in the event that item non-response rates for key administrator questionnaire variables exceed 15 percent.

B.4 Study Contacts

Laura LoGerfo and Jeff Owings are the primary contacts for the HSLS:09 study at NCES. Exhibit B-1 provides the names of contractor-affiliated consultants on statistical aspects of HSLS:09.

Exhibit B-1. Consultants on Statistical Aspects of HSLS:09

Name	Affiliation	Telephone
James Chromy	RTI	(919) 541-7019
Steven J. Ingels	RTI	(202) 974-7834
Jill A. Dever	RTI	(202) 974-7846
Daniel J. Pratt	RTI	(919) 541-6615
John Riccobono	RTI	(919) 541-7006
David Wilson	RTI	(919) 541-6990
Gary Phillips	AIR	(202) 403-6916
Steve Leinwand	AIR	(202) 403-6926

References

Merkle, D. M., & Edelman, M. (2009). An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error. Survey Practice, (March)

Research Triangle Institute (2008). SUDAAN Language Manual, Release 10.0. Research Triangle Park, NC: Research Triangle Institute (RTI).

September 2011

High School Longitudinal Study of 2009 (HSLS:09), First Follow-up

Main Study 2012

Supporting Statement

Part C

Request for OMB Review
OMB# 1850-0852 v.9

Submitted by

National Center for Education Statistics

U.S. Department of Education

Parent Response Propensity Modeling Design for HSLS:09 First Follow-up Study

Based on HSLS:09 field test results, we propose to implement a propensity based responsive design in the main study parent data collection so as to allocate nonresponse follow-up resources more efficiently. As stated, all cases will begin as web-surveys, then after a three-week production period, cases will move to computer-based telephone interviews (CATI). Finally, a select number of cases will be followed up via computer-based personal interviews (CAPI). The propensity model will be used to inform when and how quickly to move cases from CATI to CAPI . We propose to develop a propensity model that incorporates paradata^² and substantive survey variables collected during the first follow-up parent main study as well as data from the base year study and estimates a case’s likelihood of response. The plan for the parent propensity model is being based on the successful development of propensity models across NCES projects. Specifically, though results are still in progress, initial analyses from the ELS:2002 third follow-up field test showed that the propensity model had a strong predictive power of participant response (with a high R2 value). This proposed plan could be adapted according to lessons learned from other NCES projects as well as discussions during the currently scheduled November 2011 planning meeting to review findings across all NCES studies.

Results of the response propensity model and responsive design implemented in the parent first follow-up field test produced encouraging results. A propensity model (predicting early web period response outcome) was implemented immediately following the three-week early web period and incorporated information from the base year study and the panel maintenance activity. Paradata from the early web period was also considered but none of those variables contributed significantly to the model. Significant predictors of early response period outcome were whether or not the parent case completed the base year study, whether or not the parent completed the web address update activity, whether or not the student was enrolled in the same school as they were in the base-year study, whether or not the student was absent during the base-year study, and whether or not the parent has ever refused to participate. Using those variables, predicted probabilities were used to assign cases to low or high propensity groups. The 135 lowest propensity cases in the sample were assigned to the low propensity group, with half of those cases randomly assigned to the treatment group. The field test model was ultimately effective in predicting the final response outcome for low and high propensity cases; the overall response rate was 26.7% for all low propensity cases and 59.7% for high propensity cases (χ2 = 49.0206, p <.0001).

An observed difference was seen in response rate between the experimental and control groups. Low propensity experimental cases had a response rate of 33.3%, while low propensity control cases had a response rate of 21.3%. This difference did not turn out to be statistically significant (t = 1.64, p = .1028) likely due to small sample sizes, however, the observed difference suggests CAPI might be effective as an intervention for low propensity cases in the parent main study.

Implementing a “Responsive Design” in the Parent Main Study

The propensity model developed for the HSLS:09 first follow-up field test successfully distinguished between high and low propensity cases and the CAPI intervention produced an observed difference between low propensity treatment and control cases. Based on field test results, we believe we can produce an effective propensity model that distinguishes between high and low propensity cases and encourages more participation among low propensity cases in the parent main study.

In the parent main study, response propensities for all parent cases will be calculated six weeks into data collection – immediately following the three-week early response period and a subsequent three-week CATI period in order to capture paradata collected during the main study and, more importantly, to allow time for all cases for the less-costly early phases of the data collection period. The dependent variable on which response propensities are to be based will be the case’s response outcome during these first six weeks of data collection. Response propensities for parent cases therefore will be based on the most recent data captured on sample members and will also reflect cases that are low propensity in both web and CATI. As predictors, a range of paradata, student, parent, and school characteristics and panel maintenance results will be considered, similar to what was done in the HSLS:09 first follow-up field fest. Field test data revealed variables that had relatively large differences in estimates between high and low propensity cases and identified several variables that were associated with the predicted response propensity. The following variables are among those which showed these differences: 1) if the parent had attended a college night with their teenager, 2) if the parent had visited a college campus with their teenager, 3) how many colleges the parent has information about, 4) whether or not the parent encourages/discourages a job after high school, 5) whether or not the teenager participated in a religious group outside of high school, 6) whether or not the parent has attended career day or job fair with their teenager, 7) whether or not the parent arranged for teenager to sit in on or take a college class, 8) whether or not the parent has attended career day or job fair with their teenager, 9) whether or not the parent searched internet for college options with their teenager, 10) whether or not the parent talked with other parents about options for after high school, and 11) whether or not the teenager participated in organized sports outside of school in last year.

As an example, consider analyses of two of these parent variables asked in the first follow-up field test. Parents were asked: 1) if they had attended a college night with their teenager and 2) if they had visited a college campus with their teenager. Responses across those two variables were significantly correlated (r = .2014, p <.0001 and r = .1948, p <.0001 respectively) with those cases’ predicted response propensities. Furthermore, high and low propensity cases answered these two questions very differently. When asked if they had attended a college night, 39% of high propensity cases answered they had attended a college night while only 19% of low propensity cases answered positively. When asked if they have visited a college campus with their teenager, 44% of high propensity cases answered they had done so while only 25% of low propensity cases answered positively. Targeting and including more low-propensity cases in the response pool may help improve final estimates of key survey variables by including more cases that may not be typically interviewed. We expect the lowest propensity cases to include a disproportionately large number of parents of students who have left the base year school, since these were the cases that were the most challenging from which to gain participation in the field test. If the CAPI and additional tracing intervention is successful in bringing more low-propensity cases into the response pool, less bias is likely to be found in those survey variables.

The lowest quartile of response propensities will be identified as the study’s low propensity cases. The parent sample size is 11,450. We anticipate the early web period to result in responses from about 20% of cases. We estimate the three-week CATI period to result in responses from about an additional 15% of parent cases. We therefore expect 7,443 cases to remain after this six-week period. The cases will be ordered according to their response propensity and the lowest quartile of these cases (approximately 1,861 cases) will be sent for additional tracing and then to CAPI, if needed.

1 This approach was used in the base year as well as in many other studies. The size of the subsample was based on an estimated low participation rate that experience with other longitudinal studies suggests.

2 Paradata, in this context, refers to information related to locating, contacting and interviewing sample members (e.g., results of call attempts, status of mailings, and whether refused initially).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Chapter 2
Author	spowell
File Modified	0000-00-00
File Created	2021-01-31