Justification Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: November 22, 2010

FROM: Laura LoGerfo, NCES

TO: Shelly Martinez, OMB

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Cognitive Interviews for High School Longitudinal Study of 2009 (HSLS:09) Student and Parent Survey Draft Items (OMB# 1850-0803 v.37)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct cognitive interviews with 11th graders and their parents, the results of which will guide development of newly-written or proposed student and parent questionnaire items for the HSLS:09 field test. The field test will provide further data on the social, family, and education items that will be the subject of cognitive labs.

Background

The High School Longitudinal Study of 2009 is a longitudinal study of a nationally representative 9th grade cohort. HSLS:09 links to its predecessor longitudinal studies by addressing many of the same issues of transition from high school to postsecondary education and the labor force. At the same time, HSLS:09 brings a new emphasis to the study of youth transition by exploring the path that leads students to pursue and persist in courses and careers in the fields of science, technology, engineering, and mathematics (STEM). HSLS:09 is designed to measure math achievement gains in the first 3 years of high school, but also to relate tested achievement to students’ choice, access, and persistence of courses, college, and careers. The HSLS:09 assessment of algebraic reasoning will serve not just as an outcome measure, but also as a predictor of readiness to proceed into college and, in particular, STEM courses and careers. The assessment administered in the first follow up will be the same as the base year assessment, with fewer than twenty new items. The addition of these new items is required to avoid ceiling effects from the more advanced students who will take the test. Questionnaires focus on factors that shape students’ decision-making about courses and postsecondary options, including what factors, from parental input to considerations of financial aid for postsecondary education, enter into these decisions.

HSLS:09 supports two of the three goals of the American Competitiveness Initiative (ACI), which aims to strengthen math and science education, foreign language studies, and the high school experience in the United States. Information collected from students, parents, and school staff will help to inform and shape efforts to improve the quality of math and science education in the United States, increase the nation’s competitiveness in STEM-related fields abroad, and improve the high school experience.

There are several reasons the transition into adulthood is of special interest to federal policy and programs. Adolescence is a time of physical and psychological changes. Attitudes, aspirations, and expectations are sensitive to the stimuli that adolescents experience, and environments influence the process of choosing among opportunities. Parents, educators, and policymakers all share the need to understand the effects that the presence or absence of guidance from the school, in combination with that from the home, can have on the educational, occupational, and social success of youth.

These patterns of transition cover individual and institutional characteristics. At the individual level, the study will look into educational attainment and personal development. In response to policy and scientific issues, data will also be provided on the demographic and background correlates of education outcomes. At the institutional level, HSLS:09 will focus on school effectiveness issues, including resources, strategies, and programs that may affect students’ mathematics and science courses and achievement, as well as college entry in general.

By collecting extensive information from students, parents, school staff, and school records, it will be possible to investigate the relationship between home and school factors and academic achievement, interests, and social development at this critical juncture. The extent to which schools are expected to provide special services to selected groups of students to compensate for limitations and poor performance (including special services to assist those lagging in their understanding of mathematics and science) will be examined. Resources to assist in guiding parents and students through the college decision process, from information-seeking behaviors to filing financial aid forms, will be explored in how they relate to college entry. Moreover, the study will focus, for example, on basic policy issues related to parents’ role in the success of their children, including parents’ education expectations for their children, beliefs about and attitudes toward curricular and postsecondary education choices, and any preparation made for life past high school.

Additionally, because the initial survey focused on ninth-graders, it will also permit the identification and study of high school dropouts and underwrite trend comparisons with dropouts identified and surveyed in the High School and Beyond Longitudinal Study (HS&B), the National Education Longitudinal Study of 1988 (NELS:88), and the Education Longitudinal Study of 2002 (ELS:2002).

In sum, through its core and supplemental components, HSLS:09 data will allow researchers, educators, and policymakers to examine motivation, achievement, and persistence in STEM course-taking and careers. More generally, HSLS:09 data will allow researchers from a variety of disciplines to examine issues of college entry, persistence, and success, and how changes in young people’s lives and their connections with communities, schools, teachers, families, parents, and friends affect these decision, including:

• academic (especially in math and science), social, and interpersonal growth;

• transitions from high school to postsecondary education, and from school to work;

• students’ choices about, access to, and persistence in math and science courses, majors, and careers;

• the characteristics of high schools and postsecondary institutions and their impact on student outcomes;

• family formation, including marriage and family development, and how prior experiences in and out of school correlate with these decisions; and

• the contexts of education, including how minority and at-risk status is associated with education and labor market outcomes.

Design and Context

Cognitive Labs. The current request is for approval to conduct a series of cognitive interviews in November of 2010. The cognitive research report based on the results will be written in December. The contractor team – RTI and Research Support Services (RSS) of Evanston, Illinois – has drafted cognitive research materials, which include items that will explore the usability and refinement of the specially selected student and parent questionnaire items. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process.

RSS will draw cognitive research participants from the greater Chicago area (their offices are located in Evanston, Illinois). Three cognitive interview forms will be used (Attachment IV), about 12 students assigned to each of two forms, and 12 parents assigned to one form, for a total of 36 participants and 30+ items (about 10 per form). Participants will be selected to provide representation of the eleventh grade population based on sociodemographic diversity. Attachment I provides additional detail about recruitment procedures. Attachment II presents the screening questions that will be used to determine eligibility for cognitive lab participation. A copy of the participant information sheet is provided in Attachment III. Attachment IV contains the interview protocol, including all test items, and Attachment V the Assurance of Confidentiality. Finally, Attachment VI, contains the Affidavit of Non-disclosure form that RTI and RSS staff assigned to the cognitive labs task have signed.

The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Sessions will be held at times convenient for worker and student schedules. Each interview, of approximately 60 minutes’ duration, will be conducted by RSS researchers with extensive experience in cognitive testing of youth and adults. The audio recordings or their transcriptions will be made available to RTI and NCES for review.

The cognitive labs will involve intensive one-on-one interviews. The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of error in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant response, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes—examples would be “How certain are you of your answer” or “How easy or difficult was it to answer this question?”—seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or to elicit notions that the respondent thought critically relevant to but absent from the question.

The cognitive labs will provide an opportunity to hone and improve the items to be tested. The cognitive interview protocol is contained in Attachment IV of this submittal.

Assurance of Confidentiality

Cognitive lab participants will be informed that their participation is voluntary and that the information they give us will be combined with the responses of others in a summary report that does not identify anyone as an individual, and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183] (see attachment V). Participants will be assigned a unique student or parent identifier (ID), which will be created solely for data file management and used to keep all student and parent materials together. The respondent ID will not be linked to the respondent name in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is released. The student consent process will include procedures for obtaining parental permission. Contractor staff have in all instances signed the memorandum of non-disclosure (Attachment VI).

Project Schedule

Operational schedule for HSLS:09 First Follow-up, Transcripts, and College Update

HSLS:09 activity	Start date	End date
Field test
School recruitment*	Sept. 2010	May 2011
Enrollment status verification*	Oct 2010	Dec. 2010
Parent address update*	Oct. 2010	Dec. 2010
Cognitive interviewing	Nov. 2010	Dec. 2010
Batch tracing*	Jan 2011	Jan. 2011
Student in-school data collection	March 2011	Jun. 2011
Self-administered web-based data collection	March 2011	Jun. 2011
Conduct telephone interviews	March 2011	Jun. 2011
Conduct field interviews	March 2011	Jun. 2011
Process data, construct data files	Jun. 2011	Aug. 2011
Prepare/update field test reports	Jun. 2011	Dec. 2012
College update	Jun. 2012	Oct. 2012
Transcript collection	Sept. 2012	Jan. 2013
Panel maintenance (address update)	Fall 2013
Full-scale study
School recruitment	Jan. 2011	May 2012
Enrollment status verification	Sept. 2011	Dec. 2011
Parent address update	Sept. 2011	Dec. 2011
Cognitive interviewing	Sept, 2011	Oct. 2011
Batch tracing	Oct 2011	Oct. 2011
Student in-school data collection	Jan. 2012	Jun. 2012
Self-administered web-based data collection	Feb. 2012	Aug. 2012
Conduct telephone interviews	Feb. 2012	Aug. 2012
Conduct field interviews	Feb. 2012	Aug. 2012
Process data, construct data files	Jun. 2012	Dec. 2012
Prepare/update reports	Jun. 2012	Dec. 2012
College update	Jun. 2013	Oct. 2013
Transcript collection	Sept. 2013	Jan. 2014
Panel maintenance (address update)	Fall 2014

* Denotes activities already approved by OMB.

Estimate of Hour Burden

Thirty-six cognitive interviews are planned. Each interview session is expected to last approximately 60 minutes. The interview burden is therefore 36 hours, exclusive of travel time. However, there is also a screener, which is estimated to be taken by 60 individuals and will take on average 4 minutes to complete; this constitutes 4 hours, for a total burden of 40 hours.

Estimate of Costs for Recruiting and Paying Respondents

The respondents will be paid not only to motivate them to participate but also to compensate them for their participation. effort. Because both parents and students will participate in separate interviews, the following compensation plan will be used:

• If both parent and his/her student participate in their respective cognitive interviews, the parent will receive $40 and the child will receive $30.

• If only the student is interviewed, the student will receive $30, and the parent who is facilitating the interview by providing permission and transportation will receive $10.

A payment to participants is deemed necessary both to compensate individuals for their time, effort, and inconvenience, but also to ensure a sample with minimal bias based on attitudes or lifestyle (e.g., only those people who are more curious or those who are less busy may be more likely to attend). The interviews occur outside of academic school hours in a centrally located facility in the Chicago area that will likely require transportation (car or public transit), thus an incentive helps ensure student motivation and participation.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to Federal Government

The cost of conducting the cognitive interviews will be $16,053, under the RSS subcontract to RTI International, including recruitment, interviewing, transcription, analysis, report writing, and a participant incentive of $40 each per cognitive interviewee (for students: $30 plus $10 to the parent; for parents: $40).