Part A HSLS-09 First Follow-up 2012

The submittal requests clearance under the Paperwork Reduction Act of 1995 (5 CFR 1320) for the High School Longitudinal Study of 2009 (HSLS:09) first follow-up main study to be conducted in 2012 by the National Center for Education Statistics (NCES), within the U.S. Department of Education. This submission contains first follow-up questionnaires for students, parents, school administrators, and counselors, which have been revised based on results of the spring 2011 field test and the 2011 cognitive labs This study is supported by an NCES contract (Contract# ED-04-CO-0036) with he Research Triangle Institute (RTI) International and subcontracts with the American Institutes for Research (AIR), Windwalker Corporation, Horizon Research Inc., Research Support Services (RSS), and MPR Associates (MPR). Per the field test approval (OMB# 1850-0852 v.7) this submission is subject to a 60-day Federal Register notice waiver.

A. Justification

A.1 Circumstances Necessitating Collection of Information

A.1.a Purpose of this Submission

The materials in this document support a request for clearance for the first follow-up main data collection for HSLS:09, which has the following basic components and key design features:

Base Year

survey of high school 9th-graders in fall 2009 with an emphasis on STEM course-taking;
mathematics assessment;
surveys of parents, mathematics and science teachers, school administrators, and school counselors;
sample sizes of 944 schools from which more than 21,000 students participated in data collection (schools are the first-stage unit of selection, with 9th-graders randomly selected within schools);
oversampling of private Catholic schools,
public school sample augmentation in 10 states, sponsored by the National Science Foundation; and
oversampling of Asian students.

Features of the first follow-up are summarized below. Further follow-ups are also planned.

First Follow-up and College Update and High School Transcript Components

follow-up in spring 2012, when most sample members are high school juniors, also including those who have dropped out or been retained;
student questionnaires, mathematics assessment, parent survey, and school counselor and administrator questionnaires to be administered;
returning to the same schools, but separately following transfer students and those that are no longer in high school;
a “college update” with parents or students in the summer/fall after modal senior year (2013); and
high school transcript component in 2013-2014 academic year (a record of all courses taken and letter grades received in grades 9–12).

Subsequent Follow-ups

post–high school follow-ups by web survey and computer-assisted telephone interview (CATI) (The next follow-up, inquiring about postsecondary education and workforce outcomes and experience, is scheduled for spring 2015.); and
an additional web/CATI follow-up (also to include a postsecondary transcripts study) is tentatively scheduled for spring 2021.

HSLS:09 links to its predecessor longitudinal studies by addressing many of the same issues of transition from high school to postsecondary education and the labor force. At the same time, HSLS:09 sets a new focus by supporting the American Competitiveness Initiative (ACI), which aims to strengthen math and science education and to improve the high school experience in the United States, and by addressing recent calls to expand college access. HSLS:09 is designed to measure math achievement gains in the first three years of high school, but also to relate this achievement to students’ choice, access, and persistence of courses, college, and careers, especially in science, technology, engineering, and mathematics (STEM) pipelines. That is, the HSLS:09 assessment will serve not just as an outcome measure, but also as a predictor of college readiness and, in particular, of readiness to undertake STEM courses and careers.

There are several reasons the transition into adulthood is of special interest to federal policy and programs. Adolescents’ attitudes, aspirations, expectations, plans, and decisions are shaped by their experiences. Parents, educators, and those involved in education policy all need to understand the effects that the presence or absence of good guidance from the school, in combination with that from the home, can have on the educational, occupational, and social success of youth.HSLS:09 follows students as they transition from early adolescence to adulthood and captures at the individual level, education attainment, personal development, demographic information, and cognitive growth. Questionnaires focus on factors that shape students’ decision-making about courses and postsecondary options, including what factors, from parental input to considerations of financial aid for postsecondary education, enter into these decisions. The assessment administered in the first follow-up will have the same two-stage adaptive design as the base-year assessment. The addition of some new items to the assessment is required to avoid ceiling effects from the more advanced students who will take the test and to ensure that academic growth among all participants is measured accurately.

At the institutional level, HSLS:09 focuses on school effectiveness issues, including resources, strategies, and programs that may affect students’ mathematics and science course choices and achievement, as well as college entry in general. Specifically, HSLS:09 collects information from school counselors on these topics, and no secondary longitudinal survey in the last 25 years has surveyed counselors. By collecting extensive information from students, parents, school staff, and school records, it will be possible to investigate the relationship between student outcomes and the provision of resources at home and school that assists students through the college decision process, from information-seeking behaviors to filing financial aid forms.

Because the initial survey focused on 9th-graders, it will also permit the identification and study of high school dropouts and underwrite trend comparisons with dropouts identified and surveyed in the High School and Beyond Longitudinal Study (HS&B), the National Education Longitudinal Study of 1988 (NELS:88), and the Education Longitudinal Study of 2002 (ELS:2002). The HSLS:09 data will not only update the dynamics of school disengagement, but also, given the early starting point, capture early dropouts as well as the late (post sophomore year) dropouts. HSLS:09’s starting point enables analysts to study the sensitivity of the marginal student’s decision to drop out, which has never been possible with previous studies.

Further, the HSLS:09 dataset is significantly enhanced by the National Science Foundation-sponsored augmentation, which ensures that representative state-level public school data will be available for 10 of the states in HSLS:09. Given that much educational policy is made at the state level, this information will prove valuable, especially given that the study provides national norms that will further aid the interpretation of state-level findings. An ongoing plan involves linking these state representative data to state records data available through state data warehouses to create a more complete academic picture for HSLS:09 participants.

In sum, HSLS:09 data allow researchers, educators, and policymakers to examine motivation, achievement, and persistence in STEM course-taking and careers. More generally, HSLS:09 data allow researchers and educators from a variety of disciplines to examine issues of college entry, persistence, and success, and how changes in young people’s lives and their connections with communities, schools, teachers, families, parents, and friends affect these decisions, including:

academic (especially in math and science), social, and personal growth;
transitions from high school to postsecondary education, and from school to work;
students’ choices about, access to, and persistence in math and science courses, majors, and careers;
the characteristics of high schools and postsecondary institutions and their impact on student outcomes;
family formation, including marriage and family development, and how prior experiences in and out of school correlate with these decisions; and
the contexts of education, including how minority and at-risk status is associated with education and labor market outcomes.

A.1.b Legislative Authorization

HSLS:09 is sponsored by NCES, within the Institute of Education Sciences (IES), in close consultation with other offices and organizations within and outside the U.S. Department of Education (ED). HSLS:09 is authorized under Section 9543 of the Education Sciences Reform Act of 2002 (20 U.S.C).

A.1.c Prior and Related Studies

In 1970, NCES initiated a program of longitudinal high school studies. The program’s purpose was to gather time-series data on nationally representative samples of high school students that would be pertinent to the formulation and evaluation of education polices. Starting in 1972, with the National Longitudinal Study of the High School Class of 1972 (NLS:72), NCES began providing education policymakers and researchers with longitudinal data that linked education experiences with later outcomes, such as early labor market experiences and postsecondary education enrollment and attainment. Almost 10 years later, in 1980, the second in the series of NCES longitudinal high school surveys was launched – High School and Beyond (HS&B), which included one cohort of high school seniors comparable to the seniors in NLS:72. NCES’ third longitudinal study of students was the National Education Longitudinal Study of 1988 (NELS:88), which began with a cohort of 8th-graders, and the Education Longitudinal Study of 2002 (ELS:2002) followed more than a decade later with a sophomore cohort.

HSLS:09 can link to these earlier studies in several ways. The high school transcripts files will be comparable to the transcript components of the earlier studies, and the postsecondary follow-ups will also offer points of comparison. While different grades are studied in the high school years, HSLS:09 and the earlier cohorts model the same transition from high school to postsecondary education and the labor force, and in that respect can be compared. HSLS:09 will continue on the path of its predecessors while also focusing on the factors associated with choosing, persisting in, and succeeding in STEM course-taking and careers.

However, compared to its earlier counterparts, there are considerable changes to the design of HSLS:09 that will affect the ability to produce highly specific trend comparisons. NELS:88 began with an 8th-grade cohort in the spring term; although this cohort is not markedly different from the fall-term 9th-grade cohort of HSLS:09 in terms of student knowledge base, it differs at the school level in that the HSLS:09 time point represents the beginning of high school rather than the point of departure from middle school. ELS:2002 started with a cohort of 10th graders, however, this approach excluded students at risk of dropping out in 9th grade. HSLS:09 thus starts in 9th grade to capture such students and includes a spring-term 11th-grade follow-up (even though none of the predecessor studies do) because only modest gains have been seen on assessments in the final year of high school, and the 11th-grade follow-up minimizes response problems associated with testing in the spring term of the senior year.

A.2 Purpose and Use of Information Collection

HSLS:09 is intended to be a general-purpose dataset; that is, it is designed to serve multiple policy objectives. Policy issues studied through HSLS:09 include the identification of school attributes associated with mathematics achievement, college entry, and career choice; the influence that parents, teachers, and peers have on students’ achievement and development; the factors associated with dropping out of the education system; and the transition of different groups (for example, racial and ethnic, gender, and socioeconomic status groups) from high school to postsecondary institutions and the labor market, and especially into STEM curricula and careers. HSLS:09 inquires into students’ values and goals, factors affecting risk and resiliency, the social capital available to sample members, the nature of student interests and decision-making, and students’ curricular and extracurricular experiences. HSLS:09 also includes measures of school climate; each student’s native language and language use; student and parental education expectations; attendance at school; course and program selection; college plans, preparation, and information-seeking behavior; interactions with teachers and peers; as well as parental resources and support. The HSLS:09 data elements are designed to support research that speaks to the underlying dynamics and education processes that influence student achievement and development over time. In short, HSLS:09 must support both longitudinal and cross-cohort analyses and important descriptive cross-sectional analyses. HSLS:09 is first and foremost a longitudinal study; hence survey items are chosen for their usefulness in predicting or explaining future outcomes as measured in later survey waves.

A.2.a Content Justifications

While the content of the field test questionnaires was justified in the approved field test OMB submission, there will be some changes in content based on the findings of the field test and the deliberations and recommendations of the Technical Review Panel. These changes are of four kinds: some field test items have been deleted, some items have been added, some field test items have moved to a different questionnaire, and some field test items have been revised. On an item-by-item basis, these various changes are summarized and justified in a grid or matrix for each questionnaire. The justifications matrix are provided in part D (student questionnaire), part E (parent questionnaire), part F (administrator questionnaire), and part G (counselor questionnaire) of this submission.

A.3 Use of Improved Information Technology and Burden Reduction

The HSLS:09 first follow-up will follow the path forged by the base year, with virtually all questionnaire data collected in electronic media only. In addition, the student assessment will again be a computer-assisted two-stage adaptive test. For the student component, the school’s computer lab will be used when available, and, as a backup, multiple laptops will be supplied for use by the sampled students. A trained session administrator will assist students with computer issues as needed. This is the same approach that proved effective in the HSLS:09 base year administration. However, because of the presence of out-of-school students such as dropouts and transfer students in this round of data collection, we will conduct out-of-school computerized assessment and questionnaire self-administration as well as CATI and CAPI (computer-assisted personal interviews).

School administrators, counselors, and parents will be given a username and password and will be asked to complete their relevant questionnaires via the Internet. There will be a CATI follow-up for school staff and parents who do not complete the web questionnaire by self-administration. Computer-controlled interviewing offers accurate and efficient management of survey activities, including case management, scheduling of calls, generation of reports on sample disposition, data quality monitoring, interviewer performance, and flow of information between telephone and field operations.

Additional features of the CATI system include (1) online help for each screen to assist interviewers in question administration; (2) full documentation of all instrument components, including variable ranges, formats, record layouts, labels, question wording, and flow logic; (3) capability for creating and processing hierarchical data structures to eliminate data redundancy and conserve computer resources; (4) a scheduler system to manage the flow and assignment of cases to interviewers by time zone, case status, appointment information, and prior cases disposition; (5) an integrated case-level control system to track the status of each sample member across the various data collection activities; (6) automatic audit file creation and timed backup to ensure that, if an interview is terminated prematurely and later restarted, all data entered during the earlier portion of the interview can be retrieved; and (7) a screen library containing the survey instrument as displayed to the interviewer.

A.4 Efforts to Identify Duplication and Use of Similar Information

Since the inception of its secondary education longitudinal studies program in 1970, NCES has consulted with other federal offices to ensure that the data collected in this important series of longitudinal studies do not duplicate the information from any other national data sources within the U.S. Department of Education or other government agencies. In addition, NCES staff have regularly consulted with nonfederal associations such as the College Board, American Educational Research Association, the American Association of Community Colleges, National Association for College Admission Counseling, and other groups to confirm that the data to be collected through this study series are not available from any other sources. These consultations also provided, and continue to provide through the HSLS:09 Technical Review Panel, methodological insights from the results of other studies of secondary and postsecondary students and labor force members, and they ensure that the data collected through HSLS:09 will meet the needs of the federal government and other interested agencies and organizations. Other longitudinal studies of secondary and postsecondary students (i.e., NLS:72, HS&B, NELS:88, ELS:2002) have been sponsored by NCES in the past. HSLS:09 builds on, improves upon, and extends these studies rather than duplicating them.

First, the instrumentation and design of HSLS:09 explicitly complement but not replicate the redesign of NPSAS and BPS to ensure alignment across NCES studies. HSLS:09 essentially provides data that BPS and NPSAS cannot – data from postsecondary students’ high school years. Second, design articulation with prior NCES secondary longitudinal studies also show coordination, not duplication. These earlier studies were conducted during the 1970s, 1980s, 1990s, and the early 2000s and represent education, employment, and social experiences and environments different from those experienced by the HSLS:09 student sample. In addition to extending prior studies temporally as a time series, HSLS:09 extends them conceptually. To a greater degree than the previous secondary longitudinal studies, HSLS:09 provides data to understand the development of student commitment to attend higher education and to take the steps necessary to succeed in college (taking the right courses, taking courses in specific sequences, etc.). Further, HSLS:09 focuses on the factors associated with choosing and persisting in mathematics and science course-taking and STEM careers. These focal points present a marked difference between HSLS:09 and its predecessor studies.

The only other dataset that offers so large an opportunity to understand the key transitions into postsecondary institutions or the world of work is the Department of Labor (Bureau of Labor Statistics) longitudinal cohorts, the National Longitudinal Survey of Youth 1979 and 1997 cohorts (NLSY79, NLSY97). However, the NLSY youth cohorts represent temporally earlier cohorts than HSLS:09. There are also important design differences between NLSY79/ NLSY97 and HSLS:09 that render them more complementary than duplicative. NLSY is a household-based longitudinal survey; HSLS:09 is school-based. For both NLSY cohorts, base year Armed Service Vocational Aptitude Battery (ASVAB) test data are available, but there is no longitudinal high school achievement measure. Although NLSY97 also gathers information from schools (including principal and teacher reports and high school transcripts), it cannot study school processes in the same way as HSLS:09, given its household sampling basis. Any given school contains only one to a handful of NLSY97 sample members, a number that constitutes neither a representative sample of students in the school nor a sufficient number to provide within-school estimates. Thus, although both studies provide important information for understanding the transition from high school to the labor market, HSLS:09 is uniquely able to provide information about education processes and within-school dynamics and how these affect both school achievement and ultimate labor market outcomes, including outcomes in science, technology, engineering, and mathematics education and occupations.

A.5 Impact on Small Businesses or Other Small Entities

This section has limited applicability to the proposed data collection effort. Target respondents for HSLS:09 are individuals (typically nested within an institutional context) of public and private schools; first follow-up data collection activities will involve no particular burden to small businesses or entities.

A.6 Consequences of Collecting the Information Less Frequently

This submission describes the main study data collection for the first follow-up of HSLS:09. The first follow-up main study will take place in the spring of 2012, and was preceded by a field test in the spring of 2011. A college update interview will take place in the summer/fall of 2013, and a high school transcript collection in the 2013-2014 academic year. A subsequent follow-up is scheduled for the spring of 2015. The tentative design for the study calls for another round at about age 26 (2021). Recent education and social welfare reform initiatives, changes in federal policy concerning postsecondary student support, and other interventions necessitate frequent studies. Repeated surveys are also necessary because of rapid changes in the secondary and postsecondary education environments and the world of work. Important areas of change for which better information is needed include the increasing role of community colleges, the needs of demographic minorities, and the challenges faced by first-generation college-goers. Indeed, longitudinal information arguably provides better measures of the effects of program, policy, and environmental changes than would multiple cross-sectional studies.

The HSLS:09 cohort is first surveyed at the very beginning of high school to provide a baseline which also includes the full pool of potential high school dropouts. The first follow-up occurs in what will be, for most, the spring of their junior year. Since seniors tend to be disengaging from school, and since some seniors are no longer enrolled in mathematics, spring of junior year is the ideal time point for measuring achievement gain in algebraic reasoning and for examining postsecondary plans. The College Update, which will occur in the months after the cohort’s intended graduation date, records sample members’ status in terms of the transition to higher education and the work force, with an anchor in expected status as of October 2013. The timing is important in that it provides a fresh and immediate look at the outcomes of the cohort’s postsecondary planning. High school transcripts will be collected in the 2013-14 academic year, when most cohort members have completed high school. Postsecondary follow-ups are tentatively planned for the modal two-years-out of high school time point, the ideal juncture at which to study postsecondary access and choice, and for eight-years-out of high school, to capture final outcomes. None of these dates could be changed without damage to the design. While an argument could be made for additional data points, less frequent collection would adversely affect the study’s ability to meet its goals.

A.7 Special Circumstances Relating to Guidelines of 5 CFR 1320.5

All data collection guidelines in 5 CFR 1320.5 are being followed. No special circumstances of data collection are anticipated.

A.8 Consultations Outside NCES

Consultations with persons and organizations both internal and external to NCES and the federal government have been pursued. In the planning stage for HSLS:09, there were many efforts to obtain critical review and to acquire comments regarding project plans and interim and final products. The first follow-up Technical Review Panel (TRP) has also been convened and serves as the major vehicle through which future consultation will be achieved in the course of the project. The TRP met in September of 2010 and in June of 2011, and its recommendations, based on field test results presented at the June 2011 session, have been taken into consideration in revising instruments for the main study.

For base year and first follow-up assessment development, a mathematics advisory panel comprising the following experts was formed:

Hyman Bass, Professor of Mathematics, University of Michigan;
Katherine Halvorsen, Professor of Mathematics and Statistics, Smith College;
Joan Leitzel, President Emeritus, University of New Hampshire and Professor of Mathematics (retired), Ohio State University;
Mark Saul, Mathematics Teacher (retired), Bronxville High School, NY; and
Ann Shannon, Mathematics Education Consultant, Oakland, CA.

Additional consultants outside ED and members of the base-year and first follow-up Technical Review Panels include the following individuals:

Base-Year Technical Review Panel and NCES Research Consultants

Dr. Clifford Adelman
The Institute for Higher Education Policy
1320 19th Street, NW, Suite 400
Washington, DC 20036
Phone: (202) 861-8223 ext. 228
Fax: (202) 861-9307
E-mail: [email protected]

Dr. Kathy Borman
Department of Anthropology, SOC 107
University of South Florida
4202 Fowler Avenue
Tampa, FL 33620
Phone: (813) 974-9058
E-mail: [email protected]

Dr. Daryl E. Chubin
Director
Center for Advancing Science & Engineering Capacity
American Association for the Advancement of Science (AAAS)
1200 New York Avenue, NW
Washington, DC 20005

Dr. Jeremy Finn
State University of New York at Buffalo
Graduate School of Education
409 Baldy Hall
Buffalo, NY 14260
Phone: (716) 645-2484
E-mail: [email protected]

Dr. Thomas Hoffer
NORC
1155 E. 60th Street
Chicago, IL 60637
Phone: (773) 256-6097
E-mail: [email protected]

Dr. Vinetta Jones
Howard University
525 Bryant Street NW
Academic Support Building
Washington, DC 20059
Phone: (202) 806-7340 or (301) 395-5335
E-mail: [email protected]

Dr. Donald Rock
Before 10/15: K11 Shirley Lane
Trenton NJ 08648
Phone: 609-896-2659
After 10/15: 9357 Blind Pass Rd, #503
St. Pete Beach, FL 33706
Phone: (727) 363-3717
E-mail: [email protected]

Dr. James Rosenbaum
Institute for Policy Research
Education and Social Policy
Annenberg Hall 110 EV2610
Evanston, IL 60204
Phone: (847) 491-3795
E-mail: [email protected]

Dr. Russ Rumberger
Gevirtz Graduate School of Education
University of California
Santa Barbara, CA 93106
Phone: (805) 893-3385
E-mail: [email protected]

Dr. Philip Sadler
Harvard-Smithsonian Center for Astrophysics
60 Garden St., MS 71
Office D-315
Cambridge, MA 02138.
Phone: (617) 496-4709
Fax: (617) 496-5405
E-mail: [email protected]

Dr. Sharon Senk
Department of Mathematics
Division of Science and Mathematics Education
D320 Wells Hall
East Lansing, MI 48824
Phone: (517) 353-4691 (office)
E‑mail: [email protected]

Dr. Timothy Urdan
Santa Clara University
Department of Psychology
500 El Camino Real
Santa Clara, CA 95053
Phone: (408) 554-4495
Fax: (408) 554-5241
E-mail: [email protected]

Other Consultants Outside ED

Dr. Eric Bettinger
Associate Professor, Economics
Case Western Reserve University
Weatherhead School of Management
10900 Euclid Avenue
Cleveland, OH 44106
Phone: (216) 386-2184
E-mail: [email protected]

Dr. Audrey Champagne
Professor Emerita
University of Albany
Educational Theory and Practice
Education 119
1400 Washington Avenue
Albany, NY 12222
Phone: (518) 442-5982

Dr. Stefanie DeLuca
Assistant Professor
Johns Hopkins University
School of Arts and Sciences
Department of Sociology
532 Mergenthaler Hall
3400 North Charles Street
Baltimore, MD 21218
Phone: (410) 516-7629
E-mail: [email protected]

Dr. Laura Hamilton
RAND Corporation
4570 Fifth Avenue, Suite 600
Pittsburgh, PA 15213
Phone: (412) 683-2300 ext. 4403
E‑mail: [email protected]

Dr. Jacqueline King
Director for Policy Analysis
Division of Programs and Analysis
American Council for Education
Center for Policy Analysis
One Dupont Circle, NW
Washington, DC, 20036
Phone: (202) 939-9551
Fax: 202-785-2990
E-mail: j[email protected]

Dr. Joanna Kulikowich
Professor of Education
The Pennsylvania State University
232 CEDAR Building
University Park, PA 16802-3108
Phone: (814) 863-2261
E‑mail: [email protected]

Dr. Daniel McCaffrey
RAND Corporation
4570 Fifth Avenue, Suite 600
Pittsburgh, PA 15213
Phone: (412) 683-2300 ext. 4919
E-mail: [email protected]

Dr. Jeylan Mortimer
University of Minnesota—Dept. of Sociology
909 Social Sciences Building
267 19th Avenue South
Room 1014a Social Sciences
Minneapolis, MN 55455
Phone: (612) 624-4064
E-mail: [email protected]

Dr. Aaron Pallas
Teachers College
Columbia University
New York, NY 10027
Phone: (646) 228-7414
E-mail: [email protected]

Ms. Senta Raizen
Director
WestEd
National Center For Improving Science Education
1840 Wilson Blvd., Suite 201A
Arlington, VA 22201-3000
Phone: (703) 875-0496
Fax: (703) 875-0479
E-mail: [email protected]

First Follow-Up Technical Review Panel

Brian Cook
American Council on Education
One Dupont Circle NW, Suite 800
Washington, DC 20036
Voice: (202) 939-9381
Email: [email protected]

Regina Deil-Amen
Center for the Study of Higher Education
University of Arizona
1430 E. Second Street
Tucson, AZ 85721
Voice: (520) 621-8468, or (520) 444-7441
Email: [email protected]

Jeremy Finn
State University of New York at Buffalo
Graduate School of Education
409 Baldy Hall
Buffalo, NY 14260
Voice: (716) 645-6116, or (716) 645-2484 x1071
Email: [email protected]

Thomas Hoffer
National Opinion Research Center (NORC)
1155 E. 60th Street
Chicago, IL 60637
Voice: (773) 256-6097
Email: [email protected]

Vinetta Jones
Howard University
525 Bryant Street NW
Academic Support Building
Washingon, DC 20059
Voice: (202) 806-4947, or (301) 395-5335
Email: [email protected]

Amaury Nora
University of Texas at San Antonio
College of Education and Human Development
One UTSA Circle
San Antonio, Texas 78294
Voice: (210) 458-5436, or (210) 458-7394
Email: [email protected]

Jesse Rothstein
Richard and Rhoda Goldman Schol of Public Policy
University of California, Berkeley
2607 Hearst Avenue
Berkeley, CA 94720-7320
Voice: (510) 643-8561
Email: [email protected]

Russ Rumberger
University of California, Santa Barbara
Gevirtz Graduate School of Education
2329 Phelps Hall
Santa Barbara, CA 93106
Voice: (805) 451-6091
Email: [email protected]

Sarah E. Turner
Department of Leadership, Foundations and Policy
294 Ruffner Hall
University of Virginia
Charlottesville, VA 22903-2495
Voice: (434) 982-2383
Email: [email protected]

Timothy Urdan
Santa Clara University
Department of Psychology
500 El Camino Real
Santa Clara, CA 95053
Voice: (408) 554-4495
Email: [email protected]

A.9 Explanation of Payment or Gift to Respondents

Incentives are proposed to maximize school participation and student participation within schools and to encourage students and parents to participate outside of school. Incentives are also intended to help improve the chances of study participation from previously non-responding sample members. The use of incentives provides significant advantages to the government in terms of increased overall response rates, timely data collection, decreased data collection costs, and higher quality data through the minimization of nonresponse bias. The incentive structure presented in this section is, in part, based on scientific experimentation.

The incentive structure requested for the HSLS:09 first follow-up main study was approved by OMB and is presented by respondent type in Exhibit A.1 (OMB# 1850-0852 v.6). The incentive structure is further broken out by the type of case for each respondent and data collection phase in Exhibit A.2. A description and rationale for each incentive is provided below.

Exhibit A-1. Incentives by respondent type approved for main study

Respondent	Incentive/Honorarium
School	Magazine subscription equivalent; list of incentive choices ~$50
School Coordinator	$100 plus $25 for ≥ 85% or $50 for ≥ 92% student participation
IT Coordinator	$50
School Reimbursement for Costs Incurred/Lists	Up to $100 as required by schools
In-School Student	$10
Out Of School Student (OOS)	$15 for completing questionnaire plus $10 for completing assessment for students still enrolled at the school but unable to participate during the in-school session. For students no longer enrolled, $40 for questionnaire completion plus $10 for completing assessment.
Parents	$20 offered in the last two months of data collection for “most challenging” cases, plus a $5 pre-paid incentive in last month of data collection along with a hardcopy abbreviated questionnaire for subset of “most challenging” cases
School Administrators	None
School Counselors	None

NOTE: In the case of parents, incentives are only offered to the subset of the population who become the “most challenging cases” (defined as cases who have not responded after receiving a high number of calls from RTI [i.e., 15+ phone calls], refusals, and sample members for whom there is a good address but no good phone number and estimated to be 20% of parents at $20).

Exhibit A-2. Incentives by type of case and data collection phase for main study

Type of case and phase	% of sample	Response rate	% of respondent by phase	Number of respondents	Survey incentive amount	Additional assessment incentive	Total incentive amount
In-School Student	83	90%		18,829	$10	NA	$10
Out-of-school student*	17	80%
Early Web			30	1,531	$15	$10	$25
Production – students enrolled at base year school.			50	2,551	$15	$10	$25
Production – students who left the base year school			20	1,020	$40	$10	$50
Parent	45	75%
Early Web			30	2,577	NA	NA	NA
Production			50	4,294	NA	NA	NA
”Most Challenging” cases			20	1,717	$20	NA	$20

Note: In-school nonrespondents will be contacted out-of-school. Student enrollment status is provided by the school in the fall of 2011. Percent of sample refers to the percent of the overall sample for each category (i.e., in-school student, dropout, etc.) and the percentages associated with the data collection periods are the percent of responding sample members to participate within each data collection period.

Incentives for schools. As in the base year, an honorarium of $100 with the opportunity to earn an additional $25 for achieving at least an 85% student participation rate or an additional $50 for achieving a student response rate of 92% or better at the school was approved for one School Coordinator at each school. A modest token of appreciation to the schools, with an estimated value of $50 per school, in the form of a choice of 1-year science- or math-related magazine subscriptions for the school media center, was approved for the field test, with the understanding that during the field test recruitment effort, schools will be solicited for additional, alternative low-cost options that are meaningful to the schools. Staff at the field test schools reported that they were happy with the magazine subscription options and did not have suggestions for alternatives, with the one exception being cash toward the purchase of a graphing calculator. In addition, NCES proposed to send an educational "thank you" token, with a monetary value of $0.50, to each of the 24 field test schools for submitting enrollment lists and testing the survey CD (as part of the survey administration). This token consists of a set of 3 physics-related comic books published by the American Physics Society about Lasers (coinciding with the 50th anniversary of when the laser was first demonstrated; released copies can be accessed at http://www.physicscentral.org/experiment/physicsquest/upload/spectra.pdf and http://www.physicscentral.org/experiment/physicsquest/past/upload/spectra2.pdf). This token was well received by the schools and is proposed again for the main study.

Incentives for students. The use of a $10 monetary student incentive was approved by OMB for students participating in in-school sessions for the base year and first follow-up field test (OMB# 1850-0852 v.2 and v.6) and for the first follow-up main study recruitment change memo (OMB# 1850-0852 v7). Most students participating in the HSLS:09 First Follow-up will be nearing the end of their junior year of high school, making them similar to high school seniors for whom research has demonstrated the importance of incentives to participate in voluntary research studies (National Commission on NAEP 12^th Grade Assessment and Reporting, 2004; National Research Council 2003). An experiment conducted during the ELS:2002 First Follow-Up Field Test found that high school seniors were more likely to participate when receiving a $20 cash incentive (95.2% student response rate) than a token incentive (86.8% response rate). In addition, the cash incentive responds to the increased student reluctance to participate in voluntary research and the perceived stress of missing class to take another assessment encountered in the base year study. Finally, the $10 incentive would help to increase response rates for the in-school session, thus reducing the number of students requiring the costlier web, CATI, or field follow-up.

Experiences on the HSLS:09 first follow-up field test and the ELS:2002/04 follow-up demonstrated that additional incentives were necessary to gain cooperation from students who were no longer enrolled at their base year school. It is anticipated that 75% of students will be available to participate in the HSLS:09 First Follow-up through in-school data collection. An estimated 8% of students will be enrolled in the base-year school but will be absent or unable to participate in the in-school session and will need to be contacted for an out-of-school administration. The remaining 17% of students will no longer be enrolled in the base-year school and will need to be contacted out of school for the study. Indeed, the HSLS:09 first follow-up field test demonstrates that the students with the least likelihood of responding are those who had left the base-year school prior to the follow-up study. Rather than run a propensity model for these students, the incentive will be increased for students who left the base-year school prior to the first follow-up data collection.

Students identified by the school as having left the base-year school would be offered $40 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $50. All other students (including those who were absent or otherwise missed the in-school session) would continue to be offered the original incentive of $15 for completing the questionnaire and $10 for completing the mathematics assessment, for a total of $25. Students still at the base-year school would not be informed of the increased incentive for an out-of-school administration until after the in-school session has been completed.

IT coordinators. During the base-year field test, an IT coordinator was necessary at each school to facilitate the use of the school computer labs and to ensure that the school’s computers and network connectivity are compatible with the Sojourn CD which provides a secure connection between the school’s computer and the NCES website for data collection. A $50 honorarium would be offered to each school’s IT coordinator for the first follow-up main study.

Incentive for school counselors. No incentive is proposed for the counselors to complete their questionnaires. This precedent was set in the base-year study, which realized high counselor response rates without the use of monetary incentives. Counselors would typically provide the information requested in the questionnaire as well as the administrative records as part of their normal duties. Because of the nature of the study, NCES suspects that many school administrators will designate a counselor to perform the school coordinator duties, in which case the counselor will receive the coordinator honorarium as was previously approved by OMB.

Incentive for school administrators. NCES has achieved high response rates for the school administrator questionnaire on the HSLS:09 base year data collection and on the ELS:2002 base year and first follow-up rounds. Based on past experience, no incentive will be offered for this round of the school administrator questionnaire on HSLS:09.

Incentives for parents. For the parent data collection, OMB has approved a plan to offer an incentive for a subset of parents for the main study (OMB# 1850-0852 v.7). Parent response rate requirements for the main study, combined with the positive HSLS:09 base year experience with the parent incentive experiment, justifies the $20 incentive for “most challenging parent cases.” In the base year, we experienced challenges eliciting parent response and used an incentive experiment to determine the most effective incentive structure. The successful results of the experiment (submitted to OMB last year) support a $20 incentive for nonresponse follow-up among the most challenging cases. The decision to offer parents an incentive will be determined by the same rules implemented in the base-year incentive experiment, consisting of parents who have not responded after receiving a high number of calls from RTI (i.e., 15+ phone calls), who have refused, and who have a good address but no good phone number.^¹ Given the two-year lapse of time between data collections and the effectiveness of the experiment, using these conditions to dictate timing for offering incentives to parents should be equally effective for the first follow-up study.

As an extension of the already-approved $20 parent incentive, a pre-paid incentive should be useful for the last month of data collection for the subset of “most challenging” parent cases who still have not responded. Two months before the end of data collection, sample members eligible for the $20 parent incentive would receive notification of the incentive offer. A month later, sample members eligible for the incentive who have not yet responded would receive an overnight express delivery mailing with a $5 pre-paid incentive and a hard-copy abbreviated parent questionnaire. Parents receiving the overnight-delivery mailing would be informed that they could complete and return the hard-copy form, or they could complete the full questionnaire and receive the $20 incentive. A pre-paid incentive, and specifically the $5 pre-paid incentive level, has been demonstrated to be effective in increasing response rates in mail surveys (Dillman, 2011).

Reimbursement of reasonable school expenses. In some cases, there may be requests from schools for reimbursement of expenses associated with the testing session. For example, a number of base-year schools requested reimbursement for the production of enrollment lists and three others asked for reimbursement to keep the school open for testing sessions that occurred outside of normal school hours. Such cases will be reviewed by project staff on an individual basis and will be approved if the request is deemed reasonable.

A.10 Assurance of Confidentiality Provided to Respondents

A data security plan (DSP) for HSLS:09 was developed and approved by the computer security review board for the base-year and first follow-up studies. The HSLS:09 plan represents best-practice survey systems and procedures for protecting respondent confidentiality and securing survey data. An outline of this plan is provided in Exhibit A-3. The HSLS:09 DSP

establishes clear responsibility and accountability for data security and the protection of respondent confidentiality with corporate oversight to ensure adequate investment of resources;
details a structured approach for considering and addressing risk at each step in the survey process and establish mechanisms for monitoring performance and adapting to new security concerns;
includes technological and procedural solutions that mitigate risk and emphasize the necessary training to capitalize on these approaches; and
is supported by the implementation of data security controls recommended by the National Institute of Standards and Technology for protecting federal information systems.

Exhibit A-3. HSLS:09 Data Security Plan Outline

HSLS:09 Data Security Plan Summary

Maintaining the Data Security Plan

Information Collection Request

Our Promise to Secure Data and Protect Confidentiality

Personally Identifying Information That We Collect and/or Manage

Institutional Review Board Human Subject Protection Requirements

Process for Addressing Survey Participant Concerns

Computing System Summary

General Description of the RTI Networks

General Description of the Data Management, Data Collection, and Data Processing Systems

Integrated Monitoring System

Receipt Control System

Instrument Development and Documentation System

Data Collection System

Document Archive and Data Library

Employee-Level Controls

Security Clearance Procedures

Nondisclosure Affidavit Collection and Storage

Security Awareness Training

Staff Termination/Transfer Procedures

Subcontractor Procedures

Physical Environment Protections

System Access Controls

Survey Data Collection/Management Procedures

Protecting Electronic Media

Encryption

Data Transmission

Storage/Archival/Destruction

Protecting Hard-Copy Media

Internal Hard-Copy Communications

External Communications to Respondents

Handling of Mail Returns, Hard-Copy Student Lists, and Parental Consent Forms

Handling and Transfer of Data Collection Materials

Tracing Operations

Software Security Controls

Data File Development: Disclosure Avoidance Plan

Data Security Monitoring

Survey Protocol Monitoring

System/Data Access Monitoring

Protocol for Reporting Potential Breaches of Confidentiality

Specific Procedures for Field Staff

All invitation letters sent to sample members will include a statement about the voluntary nature of the survey and of the confidentiality provision in the initial cover letter and on the questionnaires, stating that their responses may be used for statistical purposes only and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183]. The material sent will also include a brochure describing the study and the extent to which respondents and their responses will be kept confidential (Appendix A.)

Additionally, HSLS:09 will conform to NCES Restricted Use Data Procedures Manual and NCES Standards and Policies. The plan for maintaining confidentiality includes obtaining signed confidentiality agreements and notarized nondisclosure affidavits from all personnel who will have access to individual identifiers. Each individual working on HSLS:09 will complete the e-QIP clearance process. The plan includes annual personnel training regarding the meaning of confidentiality and the procedures associated with maintaining confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses. The training will cover controlled and protected access to computer files under the control of a single database manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

A.11 Justification for Sensitive Questions

No sensitive questions are included on the in-school student survey. Some moderately sensitive questions (e.g., incarcerations, expulsions) are posed to out-of-school 9th-grade cohort members, such as dropouts. Though sensitive, the importance of this information, warrants collecting it, not only in terms of its educational impacts, but in terms of possible links to other outcomes, such as crime. Income is not asked of students but is asked of parents. Income is needed to support poverty and socioeconomic status variables. All responses are voluntary. For parents reluctant to give a precise amount, answers within a broad categorical range may be recorded.

A.12 Estimates of Annualized Burden Hours and Their Cost to Respondents

Estimates of response burden for the HSLS:09 first follow-up main study data collection activities are shown in Exhibit A-4. Estimates of response burden are based on estimates developed from experience with the first follow-up field test and base-year HSLS:09 questionnaires and experience on other education longitudinal studies (e.g., ELS:2002, NELS:88, HS&B). Please note that the time students will spend completing the cognitive assessment has not been included in the estimated burden. Just as in the base year, the in-school session remains 90 minutes, including survey, assessment, and instructions.

Exhibit A-4. Estimated Burden for HSLS:09 First Follow-up Main Study

Respondents	Sample	Expected response rate	Number of respondents	Number of responses	Average burden per response¹	Range of response times	Total burden (hours)
School Coordinators	944	92%	868	868	240 minutes	180 – 300 minutes	3,472
IT Coordinators	944	92%	868	868	120 minutes	60-180 minutes	1,736
School Administrators	944	92%	868	868	30 minutes	25-35 minutes	434
Transfer School Administrators	1,875	70%	1,312	1,312	10 minutes	5-15 minutes	219
School Counselors	944	92%	868	868	30 minutes	25-35 minutes	434
Students—Questionnaire	25,206	92%	23,190	23,190	35 minutes	30-40 minutes	13,528
Students—Assessment	25,206	85%	21,425	21,425	40 minutes	40 minutes	14,283
Parents – Full Questionnaire	11,450	75%	8,015	8,015	30 minutes	25-35 minutes	4,008
– Abbreviated Questionnaire		5%	573	573	10 minutes	5-15 minutes	96
Panel Maintenance (Parents)	25,206	30%	7,562	7,562	3 minutes	2-4 minutes	378

Total			36,562	44,124			24,305

¹Burden represents the time associated with preparing for and assisting with the conduct of the data collection. Burden associated with recruiting the school and providing enrollment status updates was provided in the recruitment change memo (1850-0852 v.6) in August 2010.

The cost to the school coordinator and IT coordinator is estimated at $20 per hour. The cost for the school coordinator for data collection activities is estimated at $69,440 for the main study. The cost for the IT coordinator is estimated at $34,720 for main study.

Assuming an hourly rate of $7.25 per hour, the estimated cost to student participants is estimated at $98,078 for the main study. For parents, assuming a $20 hourly wage, the cost to parent respondents is estimated to be $85,880 for the main study questionnaire and $12,600 for the panel maintenance update.

For school administrators, the questionnaire can be completed by the principal, or the principal may delegate approximately three-fourths of the questionnaire to another knowledgeable administrator at the school with the last section completed by the school principal. Again assuming a $20 hourly cost, the cost to respondents is $8,680 for the main study.

To extend and enrich the coverage of the school-level data linked to HSLS:09 sample members, we propose to ask school officials at an estimated 1,875 high schools to which students have transferred to complete an abbreviated version of the administrator survey. This survey will not require the response of the principal, but instead may be completed by any knowledgeable administrator at the given transfer school. Assuming a 70% response rate for this effort and a $20 hourly cost, the cost to respondents is $4,380.

For the counselor questionnaire, the respondent dollar cost, assuming an average hourly rate of $20 for school employees, is estimated to be $8,680 for the main study.

Included in the parent, school administrator, and counselor notification letters will be the following burden statement:

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number of this voluntary information collection is 1850-0852. The time required to complete this information collection is estimated to average 30 minutes for the parent, teacher, and school administrator questionnaires, including the time to review instructions and complete and review the information collection. The student questionnaire will be no more than 35 minutes in length, and the math test will take about 40 minutes. If you have any comments concerning the accuracy of the time estimate or suggestions for improving the interview, please write to: High School Longitudinal Study of 2009 (HSLS:09), National Center for Education Statistics, 1990 K Street NW, Washington, DC 20006.

A.13 Estimates of Total Annual Cost Burden to Respondents

There are no capital, startup, or operating costs to respondents for participation in the project. No equipment, printing, or postage charges will be incurred.

A.14 Annualized Cost to the Federal Government

Estimated costs to the federal government for HSLS:09 are shown in Exhibit A-5. The estimated costs to the government for data collection for the field test and main study are presented separately. Included in the contract estimates are all staff time, reproduction, postage, and telephone costs associated with the management, data collection, analysis, and reporting for which clearance is requested.

Exhibit A-5. Total Costs to NCES

Costs to NCES	Amount
Total HSLS:09 first follow-up costs	$ 15,719,840
Salaries and expenses	719,900
Contract costs	14,999,940
Field test (2011)	3,109,942
Salaries and expenses	215,648
Contract costs	2,894,294
Main study (2012)	12,609,898
Salaries and expenses	504,252
Contract costs	12,105,646

NOTE: All costs quoted are exclusive of incentives. Field test costs represent Task 2 of the HSLS:09 contract; base-year main study costs include task 3.

A.15 Reasons for Program Changes

There are no program changes associated with this submission. The apparent increase in the respondent burden time for this collection is due to the fact that the last OMB approval was for the first follow-up field test data collection, while this request is for data collection for the HSLS:09 first follow-up full scale study.

A.16 Publication Plans and Project Schedule

The formal contract for HSLS:09 requires the following reports, publications, or other public information releases:

1. a detailed methodological report describing all aspects of the main study design and data collection procedures (a working paper detailing the methodological findings from the field test will also be produced);

2. complete data files and documentation for research data users in the form of both a restricted-use and public-use electronic codebook (ECB) and a public-use data tool (i.e., EDAT); and

3. a descriptive First Look Report, reporting initial findings on issues of interest to the secondary school and higher education community, as determined by NCES.

The operational schedule for the HSLS:09 field test and main study is shown in Exhibit A‑6.

Exhibit A-6. Operational Schedule for HSLS:09

HSLS:09 activity	Start date	End date
Field test
School recruitment*	Sept. 2010	May 2011
Enrollment status verification*	Oct. 2010	Dec. 2010
Parent address update*	Oct. 2010	Dec. 2010
Cognitive interviewing*	Dec. 2010	Jan. 2011
Batch tracing*	Jan. 2011	Jan. 2011
Student in-school data collection*	March 2011	June 2011
Self-administered web-based data collection*	March 2011	June 2011
Conduct telephone interviews *	March 2011	June 2011
Conduct field interviews*	March 2011	June 2011
Process data, construct data files*	June 2011	Aug. 2011
Prepare/update field test reports*	June 2011	Dec. 2012
College update	June 2012	Oct. 2012
Transcript collection/keying/coding	Sept. 2012	May 2013
Main study
School recruitment*	Jan. 2011	May 2012
Enrollment status verification*	Sept. 2011	Dec. 2011
Parent address update*	Sept. 2011	Dec. 2011
Batch tracing*	Oct. 2011	Oct. 2011
Student in-school data collection	Jan. 2012	June 2012
Self-administered web-based data collection	Feb. 2012	Oct. 2012
Conduct telephone interviews	Feb. 2012	Oct. 2012
Conduct field interviews	Feb. 2012	Aug. 2012
Process data, construct data files	Nov. 2012	Sept. 2013
Prepare/update reports	June 2012	Sept. 2013
College update	June 2013	Oct. 2013
Transcript collection/keying/coding	Sept. 2013	Aug. 2014

* Denotes activities already approved by OMB.
Note:The current request for OMB review includes only data collection activities for the main study.

A.17 Reason(s) Display of OMB Expiration Date Is Inappropriate

The expiration date for OMB approval of the information collection will be displayed on data collection instruments and materials. No special exception to this requirement is requested.

A.18 Exceptions to Certification for Paperwork Reduction Act Statement

There are no exceptions to the certification statement identified in the Certification for Paperwork Reduction Act Submissions of OMB Form 83-I.

References

Dillman, D.A. (2011, April). Reconsidering Mail Survey Methods in an Internet World. American Statistical Association/American Association for Public Opinion Research Webinar.

National Commission on NAEP 12th Grade Assessment and Reporting. (2004, March). 12^th Grade Student Achievement in America: A New Vision for NAEP. Retrieved September 22, 2010, from http://www.nagb.org/publications/12_gr_commission_rpt.pdf

National Research Council, Committee on Increasing High School Students’ Engagement and Motivation to Learn. (2003). Engaging Schools: Fostering High School Students’ Motivation to Learn. Washington, DC: National Academies Press.

1 It should be noted that there is also a propensity model connected to parent response rates that is not tied to incentives. Incentives are determined by directly measured data collection indicators such as number of calls, refusals, and sample members for whom we have a good address but no good phone number. The propensity model for the parent cases is simply to inform the decision about when to move the cases from CATI (telephone) to CAPI (in-person) collection.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Chapter 2
Author	spowell
File Modified	0000-00-00
File Created	2021-01-31