PartB HSLS-09 First Follow-up Field Test 2011

This section describes the target universe for this study and the sampling and statistical methodologies proposed for the field test and the full-scale study. We will also address suggested methods for maximizing response rates and for tests of procedures and methods, and we will introduce the statisticians and other technical staff responsible for design and administration of the study.

B.1 Target Universe and Sampling Frames

The base-year target population for the HSLS:09 full-scale study consisted of 9th-grade students in public and private schools that include 9th and 11th grades. The target population for the first follow-up is this same 9th-grade cohort in 2012 (main study) or 2011 (field test).

B.2 Statistical Procedures for Collecting Information

B.2.a Field Test School Sample

The 41 base-year field test schools will be returned to in the first follow-up field test. The target will be to secure the cooperation of 24 of these base-year schools.

B.2.b Field Test and Full-Scale Student Sample

The same students who were sampled in the 2008 field test will be recruited for participation. In the main study, the same students who were sampled for the 2009 data collection will be recruited to participate in the 2012 data collection.

B.2.c Field Test and Full-Scale High School Counselor and Administrator Samples

At the returned-to base year schools in the first follow-up, the principal and lead counselor will be asked to complete a questionnaire.

B.2.d Parent Samples

In the field test, the parent of each participating student will be eligible for the field test. In the full-scale study a subsample to support a yield of 7,500 parent participants will be drawn.

The approach to parent subsampling is as follows. Contextual information was obtained from one knowledgeable parent or guardian for many of the HSLS:09 base-year sample students. In the HSLS:09 full-scale first follow-up (FFU), RTI will select a random subsample of HSLS:09 base-year parents, regardless of their base-year response status, as a cost-saving measure. Subsampling will not occur for the FFU field test due to the small field test sample.

The HSLS:09 FFU parent subsampling rates will be developed to meet the following objectives:

specified overall parent sample size of approximately 11,500;
minimum sample size for key reporting domains (e.g., Asian students) given specified detectable differences or levels of precision for key estimates;
minimum inflation to the unequal weighting effects (i.e., design effect of the weights) within larger domains; and
maximize coverage of the contextual information for the student target population.

A non-linear optimization algorithm¹ will be used to identify a solution that meets most if not all of the specified objectives. Particular attention will be paid to the cost implications associated with interviewing difficult-to-reach sample members, as well as the potential biasing effects of having an FFU subsample with a high proportion of base-year participants.

The subsampling rates will be reflected in the FFU contextual analysis weights. Additionally, imputation of key contextual items from the FFU parent questionnaire will be investigated as an option to address the loss of data associated with subsampling and nonresponse.

B.2.e Weighting

Virtually all survey data are weighted before they can be used to produce reliable estimates of population parameters. While reflecting the selection probabilities of sampled units, weighting also attempts to compensate for practical limitations of a sample survey, such as differential nonresponse and undercoverage. Furthermore, by taking advantage of auxiliary information about the target population, weighting can reduce the variability of estimates. The weighting process essentially entails four major steps. The first step consists of the computation of design or base weights. In the second step, base weights will be adjusted for nonresponse, while in the third step nonresponse-adjusted weights will be further adjusted so that aggregate counts can match reported estimates for the target population. Finally, adjusted weights will go through a series of quality control checks to detect extreme outliers and to prevent any computational or procedural errors.

The HSLS:09 multilevel and multicomponent design introduces significant complexity to the task of weighting. Every effort will be made to keep the resulting weights as simple and intuitive as possible. A minimum of two sets of weights will be required for the analysis of the HSLS:09 data: school weights and student weights. Although the expectation is to secure the stated rates of response, when response rates fall below the accepted limit (both at unit and item levels), detailed nonresponse bias analysis will be conducted to measure the extent of the incurred bias and to identify effective methods for nonresponse adjustment.

Several methods have been suggested for measuring nonresponse bias. In the simplest form, this bias can be approximated temporally by comparing responses obtained from those who respond earlier in the data collection period against late respondents. The incurred bias due to nonresponse can be measured more systematically, however, because the difference between survey estimates and their respective target parameters—the values that would result if a complete census were conducted and all units responded. For instance, when estimating a population mean () based on respondents only nonresponse bias can be expressed as

However, for variables that are available from the sampling frame,  can be estimated by Shape1 without sampling error, in which case the bias in can then be estimated by

Moreover, an estimate of the population mean based on respondents and nonrespondents can be obtained by

where is the weighted unit nonresponse rate, based on design weights prior to nonresponse adjustment. Consequently, the bias in can then be estimated by

That is, the estimate of the nonresponse bias is the difference between the mean for respondents and the mean for nonrespondents, multiplied by the weighted nonresponse rate, using the design weights prior to nonresponse adjustment. This basic approach will be used to measure bias in key survey estimates by relying on data that will be available for both respondents and nonrespondents.

As an attempt to reduce some of the bias due to nonresponse, when appreciable bias is detected at any level, design weights will be adjusted within cells indexed by variables that are deemed strong predictors of response status. To identify such variables, which typically include sampling stratification variables and indicators that can efficiently partition units into homogeneous segments, classification procedures such as CHAID (Chi-square automatic interaction detection method) will be relied on. CHAID is a hierarchical clustering algorithm that successively partitions units according to a categorical characteristic. The algorithm begins with all sample units as a whole and cycles over each predictor to find the optimal partition of the units. The most significant predictor is identified, resulting in partitioning of units into smaller subsets. Next, the algorithm is applied to each partitioned subset of units to find further partitions using the remaining predictors. The process stops after a specified number of partitioning steps or if none of the partitions at a given step is found to be significant.

For HSLS:09 all weight adjustments—including those for nonresponse and poststratification—will be calculated using RTI International’s generalized exponential model (GEM) software.² GEM is a raking procedure that is a generalization of the logic-type model, which has been proven to produce weights with less variability than what is achievable via traditional methods. GEM is superior to standard raking methods in two regards. First, it allows a much larger set of variables and their interactions to be used during the model development for nonresponse and raking adjustments, hence enabling the weighted data to mimic the distribution of the target universe with respect to a more comprehensive set of indices. Second, this desirable property is achieved while preventing the adjusted weights from becoming too extreme. That is, GEM produces study estimates that better represent the target universe without increasing variance of estimates significantly, which would otherwise reduce the power of statistical tests.

B.2.f Variance Estimation

For variance estimation, sets of 200 balanced repeated replication (BRR) weights will be created for school and student samples. The BRR weights are appropriate for use in NCES’s EDAT and do not affect the analysis weights used for point estimation. The BRR weighting process will replicate the full weighting process and will use procedures developed for a number of other studies, including ELS:2002 and the National Study of Postsecondary Faculty. In addition, analysis strata and primary sampling units (PSUs) created from the sampling PSUs will be included on the electronic code book for analysts wanting to use Taylor series variance estimation rather than BRR weights.

B.2.g Imputation of Missing Data

Missing values due to item nonresponse will be imputed after the data are edited. Imputation will be performed for items commonly used to define analysis domains, items that are frequently used in cross-tabulations, and items needed for weighting. Items from HSLS:09 that are subject to imputation will be imputed using a weighted sequential hot deck procedure.³ By incorporating the sampling weights, this method of imputation takes into account the unequal probabilities of selection in the original sample while controlling the expected number of times a particular respondent’s answer will be used as a donor.

B.3 Methods for Maximizing Response Rates

Procedures for maximizing response rates at the institution and respondent levels are based on successful experiences on predecessor and other similar studies. In this section, methods for maximizing response rates for students, parents, and school staff are discussed. Plans for maximizing response rates for school recruitment were presented and approved by OMB in an earlier submission (OMB# 1850-0852 v.6).

Student. The majority of students will participate in HSLS:09 during an in-school session. A subset of the students selected to participate in the base-year study will no longer be enrolled in the base-year school at the time of the first follow-up and will therefore be contacted to complete the questionnaire and assessment via Web, CATI, or CAPI. Methods for maximizing response rates among students participating in-school will be discussed, followed by methods for maximizing response rates for students participating out-of-school.

Ensuring a high student response at each school begins several weeks prior to the student session. Session administrators will work closely with the school coordinators to coordinate the logistics of the sessions and notify students about the sessions. Because the sampled students are dispersed across multiple classes, there is a heavy burden on the school coordinator to inform students about the session, distribute parental consent materials, and ensure that students arrive at the prescribed location at the scheduled date and time. Session administrators will assume as much of this burden as is possible and permissible by the school.

From past experience, ensuring that students are made aware of the session is the most critical aspect of making sure they arrive at the session at the scheduled time. Despite receiving the consent form to take home, students do not necessarily distinguish the form from other materials they take home, and they often forget about the session without frequent reminders. To help remind students about the sessions, the study will implement options such as distributing postcard reminders a day or two prior to the session, notifying the teachers of selected students from whose classes students will be pulled out for HSLS:09, asking the school coordinator to make an announcement on the PA system, and having the Session Administrator visit a few days prior to the session and convene a brief meeting of the student sample members to encourage participation. Parent contact information will be collected from each school from which the parent survey will be conducted. If phone numbers are provided, the Session Administrator will contact parents a day or two prior to the session to remind the students when they should arrive.

Each week, project staff will conduct group strategy calls with the Session Administrators to discuss the status of the schools with test dates scheduled for the coming 2 weeks. The purpose of these conference calls is to learn about the preparedness of each school for the student session, identify any concerns about anticipated response rate or computer capabilities at the school, provide a forum for brainstorming solutions to anticipated problems, and share success stories and lessons learned from other schools. Project staff will follow up frequently with Session Administrators who report problems or concerns with the preparations for student sessions at particular schools.

Plans for student incentives in the field test were described in an earlier submission. Each student participating in school will receive $10, as was offered in the base-year field test. This same incentive level is requested for the main study. In the field test, this incentive will help to achieve the required student yield given that makeup sessions will not be conducted as a cost containment measure. For the main study, offering an incentive to students participating in-school will help to maximize student participation in-school, which is a lower cost option than following students outside of school to participate in the follow-up study.

Students who have left the base-year school or who are absent for the in-school session will be contacted outside of school to participate in the study via Web or CATI administration. Parental consent is required before the student takes part in the study. We plan to collect parental consent in one of two ways. Parents will receive credentials to log into the study website and provide consent online for the student to complete the questionnaire. Parents may also provide consent verbally to a telephone interviewer for their teenager’s participation.

Parents or guardians will be the recipient of the initial letter and brochure inviting students to participate to encourage parents to provide parental consent online for the student to participate. Once parent permission is obtained, subsequent reminder notices will be sent directly to the student. Telephone interviewers will prompt students by phone to complete the questionnaire and assessment, with the opportunity to complete the questionnaire by phone. A small number of students will be contacted for CAPI administration if we are unable to reach them for a Web or phone interview.

Our experience on ELS:02/04 showed that it is more challenging to obtain high response rates among students participating outside of school than their in-school counterparts. We expect that students with the lowest propensity to respond would be students who did not participate in the base year study and those who have dropped out of school. Gaining the cooperation of these two groups are critical, as students must participate in either the base year or first follow-up study to persist in later rounds of data collection, and dropouts are a group of particular interest in the research community. For that reason, we propose to target students with the lowest propensity to respond. We anticipate that this will increase the response rate between these two groups as well as increase the weighted response rate overall. In addition, we predict that this would result in less biased survey estimates because the focus will be on cases that are dissimilar from those that have already responded (Merkle and Edelman 2009).

In an effort to increase response rates and minimize nonresponse bias among the students with the lowest propensity to respond, NCES proposes to identify low propensity (i.e., difficult to complete) cases prospectively and offer to them an increased incentive after the three-week early web data collection window has expired. At the start of the data collection, all students contacted to participate outside of school would be offered $15 to complete the questionnaire and $10 for completing the mathematics assessment, for a total of $25. After the three-week early web data collection period expires, low propensity to respond students would be offered $40 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $50. All other students would continue to be offered the original incentive of $15 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $25.

Parent. Parents of base-year sampled students received a panel maintenance letter in the fall of 2010 indicating that they would be contacted to participate in the first follow-up in the spring of 2011. There will be several additional opportunities to interact with parents to encourage their participation in the study. The parental consent form will be sent home with the students several weeks before the student session, and the letter will mention that the parent interview is forthcoming. A letter will be sent to the parent via e-mail and U.S. Mail to initiate the parent interview, providing a URL and credentials for the web instrument and a telephone number that can be used for a telephone interview. If a telephone number is available, the SA will contact the parent to remind him or her of the student session, and will take the opportunity to build a relationship with the parent and encourage participation from both the student and parent. Parents who do not complete the web instrument will be followed up via CATI, with CAPI data collection conducted as a last resort. Unless the field test (all electronic) conclusively suggests that this is not needed, paper-and-pencil versions of the questionnaire will be available for parents in the main study who do not have a telephone or Internet access (and/or as a last resort). The parent interview will be translated into Spanish to accommodate limited English proficient and nonproficient parents. Less than one percent of parents were excluded in the base year due to other language needs, thus it is not proposed to translate the interview into other languages.

To mitigate the challenges experienced in the parent data collection for the HSLS first follow-up field test in 2011, NCES is proposing to implement propensity modeling for case prioritization on the parent survey. The approach we propose is a different survey mode (CAPI) for low-propensity cases. CAPI would be provided for those cases with (a) a lower propensity to respond (as modeled based on past survey behavior in the base year and sociodemographic information) and (b) a greater biasing impact on the estimates. Using propensity modeling for case prioritization is a more sophisticated method for assigning interventions for nonrespondents than was used in the base-year study. In addition, this approach assumes that it is preferable to directly confront the problem of minimization of bias, rather than, as in the earlier round, to primarily manipulate and measure the response rate, which is only a crude proxy for a lesser probability of bias.

A parent incentive of $20 is proposed for a percentage of main study parents to help achieve high parent response among the most challenging cases. The proposed incentives are based on the incentive experiment conducted toward the end of the base-year parent data collection. The decision to offer an incentive for parents will be determined by rules similar to those implemented in the base-year incentive experiment, consisting of sample members who have not responded after receiving a high number of calls from RTI, refusals, and sample members for whom we have a good address but no good phone number. This method of determining timing of offering an incentive was effective in the base year experiment and is proposed for the follow-up study as well. No parent incentive is requested for the field test.

School Staff (School Administrators and School Counselors). School staff will receive a letter to initiate their questionnaire at the start of the data collection period. The Session Administrator will work with the school coordinator to prompt school staff to complete their interview. While at the school, the Session Administrator will prompt for any outstanding staff questionnaires. If the questionnaires still have not been completed by 1 week after the session(s) are complete in the school, CATI follow-up will commence. Schools that decline to participate in the student component of the first follow-up will still be contacted to complete the school staff questionnaires.

Laura LoGerfo and Jeff Owings are the primary contacts for the HSLS:09 study at NCES. Exhibit B-1 provides the names of contractor-affiliated consultants on statistical aspects of HSLS:09.

Exhibit B-1. Consultants on Statistical Aspects of HSLS:09

Name	Affiliation	Telephone
James Chromy	RTI	(919) 541-7019
Steven J. Ingels	RTI	(202) 974-7834
Jill Dever	RTI	(202) 974-7846
Peter H. Siegel	RTI	(919) 541-5902
Daniel J. Pratt	RTI	(919) 541-6615
John Riccobono	RTI	(919) 541-7006
David Wilson	RTI	(919) 541-6990
Gary Phillips	AIR	(202) 403-6916
Steve Leinwand	AIR	(202) 403-6926

References

Mason, R. E., Wheeless, S. C., George, B. J., Dever, J. A., Riemer, R. A., & Elig, T. W. (1995). Sample allocation for the status of the Armed Forces surveys. Proceedings of the American Statistical Association, Section on Survey Research Methods, 769–774. Paper presented at the Joint Statistical Meetings, Orlando, FL.

Merkle, D. M., & Edelman, M. (2009). An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error. Survey Practice, (March)

1 An optimization algorithm, in general terms, is defined as a mathematical model that identifies the best solution to a set of specified requirements (criteria) for the sample. For example, one criterion might specify the maximum survey cost (e.g., 11,500 sample cases), while the second and third criteria specify the minimum sample size for two overlapping analysis domains (e.g., race/ethnicity and parental education). The algorithm iteratively solves the system of equations in an attempt to find the best solution that meets all or most of the specified criteria (see, e.g., Mason et al. 1995).

2 Folsom, R.E., and A.C. Singh (2000). “The Generalized Exponential Model for Sampling Weight Calibration for Extreme Values, Nonresponse, and Poststratification.” Proceedings of the Section on Survey Research Methods of the American Statistical Association, pp. 598–603.

3 Iannacchione, V.G. (1982). “Weighted Sequential Hot Deck Imputation Macros.” In Proceedings of the Seventh Annual SAS User’s Group International Conference (pp.759–763). Cary, NC: SAS Institute, Inc.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Chapter 2
Author	spowell
File Modified	0000-00-00
File Created	2021-02-01