Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: June 21, 2010
FROM: John Wirt, NCES
TO: Shelly Martinez, OMB
THROUGH: Kashka Kubzdela, NCES
SUBJECT: Request to conduct Cognitive Interviews for Education Longitudinal Study of 2002 Third Follow-up 2012 (ELS:2002/12) Survey Draft Items under NCES Generic Cognitive Clearance (OMB# 1850-0803 v.31)
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct cognitive interviews with young adults (age 24-28), the results of which will guide development of questionnaire items for the ELS:2002/12 field test interview. The field test will provide further data on the social, cognitive, career, and education items that will be the subject of cognitive labs, and will provide empirical data on the coefficient alpha reliability of the five career and four education scales.
Housed in NCES’s Elementary-Secondary and Libraries Studies Division, the 2012 Education Longitudinal Study of 2002 third follow-up (ELS:2002/12) is the final round in a longitudinal study of sophomore (2002) and senior (2004) cohorts of high school students followed from 10th grade through the transition period to young adulthood. In the high school years, ELS:2002 was able to obtain student questionnaire and assessment data, supplemented by school and home contextual data from teachers, administrators, and parents, as well as school records data from high school transcripts. As a result, the study was able to identify the correlates of achievement gain in the last half of high school, and to plot the education trajectories of both students and dropouts. Two years after high school graduation (the second follow-up in 2006), the study was able to gather information for describing and understanding issues of postsecondary education access and choice. The third follow-up in 2012 (with a field test in 2011) will take place at a time point – a modal age for the cohort of age 26 – when many will have completed higher education degrees, and processes and outcomes such as postsecondary education persistence and baccalaureate attainment can be studied. Final outcomes will be gathered at minimum in terms of current job or career, postsecondary education history, family formation, and civic engagement. Additional topics will also be considered for the questionnaires. A potential new topic area, the outcome adjustment to a career or education program, is the subject of the cognitive research described in this submission. The ELS:2002/12 questionnaire data will be supplemented by postsecondary education transcript data, and, if it proves feasible, comprehensive financial aid records for the college-going stream of the cohort.
We propose to conduct cognitive interviews for a set of 30 items that we expect to be used in the field test. This qualitative evaluation will help refine and test the adequacy of items whose measurement qualities are not precisely known. These items, though based on similar items in published research1 have been newly written to adapt them to the needs of ELS:2002/12 by Dr. Robert Lent of the University of Maryland, a foremost authority in social cognitive career theory. As the only items currently under consideration that are truly newly written items, this set of questions constitutes the best use of the project’s cognitive research resources. These items pertain to specific outcomes—young adult adjustment and persistence intentions— in two distinct domains: the occupation and the education domains. Based on the results from the prior study of respondents at age 26, NELS:88 in 2000, it is our expectation that roughly 90 percent of cohort members will be either working or enrolled in postsecondary education, or both. In NELS:88/2000, 9 percent of cohort members were neither working nor enrolled at age 26, with 70 percent working but not enrolled, 16 percent working while enrolled, and 4 percent enrolled but not working (Ingels et al., 2002). The social cognitive career and education items will therefore provide broad outcome coverage of a very diverse sample that has followed divergent pathways from high school to adulthood.
The social cognitive items build directly on the high school data, that is to say, they are designed to complement the predictors (and intermediate outcomes) already included in the ELS:2002 survey, particularly those grounded in the base year and first follow-up, when subject-specific self-efficacy and various self-regulated learning questions were asked. In some cases, the new material also extends items from the predecessor high school cohort studies. For example, items about current satisfaction with particular job facets are anchored in some of the general satisfaction items pertaining to the job as a whole (the literature distinguishes facets from general satisfaction, but most research relating social cognitive theory to work satisfaction outcomes has been focused on general job satisfaction).
Theoretical rationales for a social cognitive approach to career adjustment, including job satisfaction, career and education persistence intentions, and other outcomes, are elaborated in extensive literature (in particular Anderson and Betz, 2001; Betz, 2007; Duffy and Lent, 2009; Lent and Brown et al., 2003; and Bandura, 1998).
The test items refer to work tasks generically, that is without describing a specific job. For interpreting the work adjustment construct, there are scales that include measures of self-efficacy, outcome expectations, interests, environmental support/barriers, persistence intentions (i.e. goal to remain in one’s current job), and an expanded measure of overall job satisfaction (to complement and extend the satisfaction items used in predecessor studies such as NELS:88/2000). For education adjustment, a parallel set of scales has been stipulated: education self-efficacy, interests, supports, and persistence intentions.
Cognitive Labs. The current request is for approval to conduct a series of cognitive interviews between July and August 2010. The cognitive research report will be written in September, so that findings and recommendations can be shared with the Technical Review Panel, which is scheduled to meet in the last week of September, 2010. The contractor team – RTI and Research Support Services (RSS) of Evanston, Illinois – has drafted cognitive research materials, which include items that will explore the usability and refinement of the specially selected work and education subset of the ELS:2002/12 item pool that was described above. An abbreviated questionnaire will collect standard demographic information about each of the participants during the screening process.
RSS will draw cognitive research participants from the greater Chicago area (their offices are located in Evanston, Illinois). Participants will include representatives of both the college-going and non-college-going young adult population, including some who are currently in college. Three cognitive interview forms will be used (Attachment IV), and about 10 young adults assigned to each, for a total of 30 participants. Participants will be selected to provide representation of the young adult population based on demographic diversity, as well as age and college-going experience (or the lack thereof). It will be expected that 20 respondents shall respond to work items (there are 21 work items), and 10 to education items (there are 14 education items). Attachment I provides additional detail about recruitment procedures. Attachment II presents the screening questions that will be used to determine eligibility for cognitive lab participation. A copy of the participant information sheet is provided in Attachment III. Attachment IV contains the interview protocol, including all test items, and Attachment V the Assurance of Confidentiality.
The cognitive interviews will be held in a facility that is centrally located, easily accessible by car and public transportation, and allows for professional audio recording. Sessions will be held at times convenient for worker and student schedules. Each interview, of approximately 60 minutes’ duration, will be conducted by RSS researchers with extensive experience of cognitive testing of youth and adults. The audio recordings will be made available to RTI and NCES for review.
The cognitive labs will involve intensive one-on-one interviews. The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of error in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant response, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes—examples would be “How certain are you of your answer” or “How easy or difficult was it to answer this question?”—seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or to elicit notions that the respondent thought critically relevant to but absent from the question.
The cognitive labs will provide an opportunity to hone and improve the nine scales. Then the field test will yield empirical data about the reliability of the scales.
The cognitive interview protocol is contained in Attachment IV of this submittal.
Cognitive lab participants will be informed that their participation is voluntary and confidential (see attachment V). Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all student materials together. The respondent ID will not be linked to the respondent name in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is released.
Overall schedule for ELS:2002/12 field test and full-scale activities
Activity |
Start |
End |
Field test |
|
|
Cognitive testing of items, results report |
7/2010 |
9/2010 |
Panel maintenance: contact updates for sample |
9/2010 |
6/2011 |
Technical Review Panel meeting |
9/2010 |
9/2010 |
Data collection |
7/2011 |
12/2011 |
Full-Scale Study |
|
|
Panel maintenance: contact updates for sample |
9/2010 |
6/2012 |
Technical Review Panel Meeting |
1/2012 |
1/2012 |
Data collection |
7/2012 |
1/2013 |
Transcript and Student Aid Data |
|
|
Pilot testing of operations |
2/2013 |
8/2013 |
Transcript and student aid data collection |
8/2013 |
3/2014 |
Transcript keying and coding |
11/2013 |
8/2014 |
Thirty cognitive interviews are planned. Each interview session is expected to last approximately 60 minutes. The interview burden is therefore 30 hours, exclusive of travel time. However, there is also a screener and it is estimated that it will be completed by 60 individuals with an average completion time of 4 minutes; this constitutes 4 hours, for a total burden of 34 hours.
To compensate the respondents for their time and effort, they will receive $40 for their participation.
There are no direct costs to participants.
The cost of conducting the cognitive interviews will be $16,832, under the RSS subcontract to RTI International, including recruitment, interviewing, transcription, analysis, report writing, and a participant incentive of $40 each per cognitive interviewee.
Anderson, S.L., and Betz, N.E. (2001). “Sources of Social Self-efficacy Expectations: Their Measurement and Relation to Career Development.” Journal of Vocational Behavior, 58: 98-117.
Bandura, A. (1997). Self-Efficacy: The Exercise of Control. New York: W.H. Freeman & Co.
Betz, N.E. (2007). “Career Self-efficacy: Exemplary Recent Research and Emerging Directions.” Journal of Career Assessment, 15: 403-422.
Duffy, R.D., and Lent, R.W. (2009). “Test of a Social Cognitive Model of Work Satisfaction in Teachers.” Journal of Vocational Behavior 75: 212-233.
Ingels, S.J., Curtin, T.R., Kaufman, P, Alt, M.N. and Chen, X. (2002). Coming of Age in the 1990s: The Eighth-Grade Class of 1988 12 Years Later. (NCES 2002-321). Washington, DC: National Center for Education Statistics.
Lent, R.W., Brown, S.D., Schmidt, J. Brenner, B., Lyons, H. and Treistman, D. (2003). “Relation of Contextual Supports and Barriers to Choice Behavior in Engineering Majors: Test of Alternative Social Cognitive Models. Journal of Counseling Psychology, 30(4): 458-465.
1 The following public domain sources have served as models for various scales, suggesting some of the dimensions that should be covered: Job Satisfaction (Brayfield & Rothe Index of Job Satisfaction 1951); Job Persistence (Camman et al. 1979); and Occupation Self-efficacy (Dunn & O’Brien’s 2009 Self Efficacy in Work Domains scale).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Memorandum |
| Author | mcominole |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |