Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: August 2, 2011
FROM: Elise Christopher, NCES
TO: Shelly Martinez, OMB
THROUGH: Kashka Kubzdela, NCES
SUBJECT: Request to conduct Cognitive Interviews for Education Longitudinal Study of 2002 Third Follow-up 2012 (ELS:2002/12) Full Scale Survey Draft Items under NCES Generic Cognitive Clearance (OMB# 1850-0803 v.54)
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct cognitive interviews with young adults (age 24-28), the results of which will guide development of questionnaire items for the ELS:2002/12 main study interview.
Housed in NCES’s Elementary-Secondary and Libraries Studies Division, the 2012 Education Longitudinal Study of 2002 third follow-up (ELS:2002/12) is the final round in a longitudinal study of sophomore (2002) and senior (2004) cohorts of high school students followed from 10th grade through the transition period to young adulthood. In the high school years, ELS:2002 was able to obtain student questionnaire and assessment data, supplemented by school and home contextual data from teachers, administrators, and parents, as well as school records data from high school transcripts. As a result, the study was able to identify the correlates of achievement gain in the last half of high school, and to plot the education trajectories of both students and dropouts. Two years after high school graduation (the second follow-up in 2006), the study was able to gather information for describing and understanding issues of postsecondary education access and choice. The third follow-up in 2012 (with a field test currently on-going in 2011) will take place at a time point – a modal age for the cohort of age 26 – when many will have completed higher education degrees, and processes and outcomes such as postsecondary education persistence and baccalaureate attainment can be studied. Final outcomes will be gathered at minimum in terms of current job or career, postsecondary education history, and other life-course transitions to adulthood, such as family formation, and civic engagement. The ELS:2002/12 questionnaire data will be supplemented by postsecondary education transcript data.
We propose to conduct cognitive interviews for a set of approximately 30 items that we expect to be prime candidates for use in the main study. Although all are items new to ELS:2002, the sources of the items to be tested are various. The financial aid items, on both loans and grants or scholarships, have been developed collaboratively, with the joint effort of the postsecondary (PACE) division and the elementary-secondary division of NCES. Financial literacy items have been taken from the PISA 2012 field test instruments, with some modification to address the difference in populations (PISA is a study of 15-year-olds, while in ELS:2002 the modal age is 26).The items seek to expand coverage in several areas: financial aid (both grant and loan experiences), financial literacy, sense of financial burden, volunteerism, voting, and social capital.
The ELS:2002 field test questionnaire asks about amount of money owed but contains nothing on the burden this debt is imposing. Items on financial burden and stress have not been asked before in ELS:2002 but indebtedness is an important consequence of loan-based aid, as well as a major factor in the more general economic situation of young adults. The three financial burden items proposed for cognitive interviews are taken from the Youth Development Survey.
Two items tap charitable giving and volunteering. The source for these items is the NLSY. A further topical area is voting and civic engagement. The items that are to be tested complement and extend existing items in the approved field test questionnaire. A 2010 voter registration item is taken from the 2010 CPS Voting Supplement. The field test questionnaire asks about the 2008 election, but for a 2012 interview, 2010 may be an additional marker that includes both a presidential and non-presidential election cycle. A broad civic involvement measure covering activities in the preceding twelve months is from Ad Health. Finally, two items, one overlapping with civic participation, represent social capital, and are taken from the General Social Survey. Despite the influence of Coleman’s work on the secondary longitudinal studies, and despite some importantly used measures of social capital and intergenerational closure in the ELS:2002 base year, the construct of social capital has not been represented on the later ELS:2002 questionnaires.
Cognitive Labs. The current request is for approval to conduct cognitive interviews in August-September 2011. The cognitive research report will be written in October, so that findings and recommendations can be shared with the Technical Review Panel, which is tentatively scheduled to meet in the first week of November, 2011. The contractor team for this project is RTI and Research Support Services (RSS) of Evanston, Illinois. The presented here cognitive research materials will explore the usability and refinement of the specially selected items.
RSS will draw cognitive research participants from the greater Chicago area. Participants will include representatives of both the college-going and non-college-going young adult population, but will emphasize current and former students who received financial aid in the form of grant or loan. Three cognitive interview forms will be used (Attachment IV), and about 10 young adults will be assigned to each, for a total of 30 participants. Participants will be selected to provide representation of the young adult population based on demographic diversity, as well as age and college-going experience (or the lack thereof). The cognitive labs will be advertised at a variety of locations and through a variety of media in the Chicago metropolitan area to maximize exposure to diverse young adult populations, such as:
Central locations at colleges, universities, and other postsecondary institutions;
Community locations such as coffee shops;
On-line resources such as Craigslist, and
College/university and city newspapers.
The advertisements (see Attachment I) will describe the purpose of the cognitive research and details of participation, including the time commitment, incentive for participation, and contact information. The flyer will also describe the age requirements. Individuals who are interested in participating in a cognitive lab will be asked to call the RSS office to complete a brief screening interview to determine eligibility (see Attachment II). Eligible participants will be those determined to be within age range (24-28), with a range of demographic characteristics and education experiences. Postsecondary grant and loan recipients will be heavily represented. It will be expected that 10 respondents shall respond to educational loan items, 10 to grant/scholarship items, and 10 to a medley of topics--volunteerism, voting, financial burden and social capital.
Attachment I contains a sample recruitment flyer, Attachment II a recruitment screener, Attachment II the participant information sheet, Attachment IV interview protocol including all test items, and Attachment V the consent form.
The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Sessions will be held at times convenient for worker and student schedules. Each interview, of approximately 60 minutes’ duration, will be conducted by RSS researchers with extensive experience of cognitive testing of youth and adults. The audio recordings will be made available to RTI and NCES for review.
The cognitive labs will involve intensive one-on-one interviews. The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of error in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant response, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes (e.g., “How certain are you of your answer” or “How easy or difficult was it to answer this question?”) seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, and identify critical information that the respondent feels was left out of the question. The cognitive labs will provide an opportunity to hone and improve the questions and their responsive options.
Cognitive lab participants will be informed that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, Section 9573] (see Attachment V). Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all student materials together. The respondent ID will not be linked to the respondent name in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is released.
Thirty cognitive interviews are planned. Each interview session is expected to last approximately 60 minutes. The interview burden is therefore 30 hours, exclusive of travel time. To yield 30 participants, we expect to screen 60 individuals during the recruitment process. We expect the recruitment to take an average of 4 minutes; for a total of 4 hours. Therefore, the overall total respondent burden time for this study is estimated to be 34 hours.
To compensate the respondents for their time and effort, they will receive $40 for their participation in and completion of the cognitive interview.
The cost of conducting the cognitive interviews will be $16,832, under the RSS subcontract to RTI International, including recruitment, interviewing, transcription, analysis, report writing, and a participant incentive of $40 each per cognitive interviewee.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Memorandum |
| Author | mcominole |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |