High School Longitudinal Study of 2009 First Follow-up (HSLS:09)
2010 Cognitive Interviews
Attachment IV
Cognitive Interview Protocols
HSLS:09 First Follow-up
Protocol for Cognitive Interviews with Students and Parents
INTERVIEW PROTOCOL AS EXPLAINED TO PARTICIPANTS
Introduction and Welcome
Hello, my name is ___________ and I will interview you today. This is ____________ and we are from Research Support Services, or RSS, an organization that is working with RTI International. RSS and RTI International are conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. We are developing a questionnaire that will gather information on the work and education experiences of a sample of young adults who have been followed since high school. We need the input of both students and parents.
As we mentioned when you were recruited, we would like to record this session so that we can review your ideas and comments after this meeting is over. We will prepare a report that will not mention any names and will not single out any one person. It will only summarize what we heard from interviewees as a whole. We will keep the recording only until our report is finished.
All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) Public Law 107-279, Section 183].
Before we begin, I would like to go over the format for our session. Basically, I will ask you to answer a series of questions on paper. I will then ask you about your understanding of each question and the process by which you reached or determined your answer.
CONTENT: Interview Questions (with justifications for OMB inserted).
Cognitive Lab Questions for HSLS:09 Student Survey
Form A
1. Do you plan to go to college after high school?
Response options:
1 Yes, at a 4-year college or university
2 Yes, at a 2-year community college or career college [alternate form for “career college:” “technical institute”]
3 No
4 Don’t know
Justification: This question will be asked of all students and is necessary to route to or around certain college-specific questions. For this question specifically, the cognitive test will explore whether “career college” is an appropriate shorthand for a postsecondary career or technical school or institute.
2. [For those with a college intention (answer 1 or 2 on question #1) ONLY] Which of the following reasons apply to your decision to go to college? Mark all that apply, then pick the strongest reason:
It’s never been an option not to go;
To get a good job;
To participate on sports teams;
Your parents expect you to;
You don't know what else you would do;
Your friends are going;
You are interested in a certain subject;
You like school
Other—please specify
Response options:
1 Applies
2 Doesn’t apply
Justification: This is a new question. The goals are to make sure the question is understandable, that the items are not confusing (e.g., there is a double-negative in the first item), that a combined “mark all/choose best” instruction is not difficult for students, and that we elicit potential other reasons for intending college.
3. What factor would you consider MOST important in choosing a college?
Response options:
1 It is close to home
2 There is a lot to do nearby
3 Overall quality of its academic programs
4 A specific academic program
5 Good sports teams
6 It is affordable
7 A good social life
8 A good record of placing graduates in jobs
9 Friends or relatives are there
10 Other—please specify
Justification: This is a new question. The intent is to understand the most important college choice factor among a list of general but discrete options; a companion parent version is also planned (and will be cognitively tested), making direct comparison between parent and student decision-making factors possible. The goal of cognitively testing this question with students is to ascertain any confusion about response options (e.g., that “specific academic program” means major or area of study), perceptions of overlap between options, and alternative options. One major question is whether students who say they have no college intention find this question appropriate or confusing. The confidence or certainty of answers is a focus for all students; there is a concern that such a question may be too early for 11th graders to answer with definitiveness. Finally, we can compare student interpretations of this question to question #6, which is keyed to a specific college.
4. [For those with a college intention (answer 1 or 2 on question #1) ONLY] What field of study are you considering? Check all that apply:
Computer and information sciences
Engineering and engineering technologies
Biological sciences
Mathematics and statistics
Physical sciences
Mechanic or repair work, such as car repair or HVAC
Health professions, such as doctor, nurse, clinical work
Liberal arts or humanities
Business
Security and protective services, such as police or firefighting
Social sciences and history
Education or teaching
Psychology
Other—please specify
Don’t know
Response options:
1 Applies
2 Doesn’t apply
Justification: This is a new item. At issue is the question wording (do students appropriately interpret it as a major or program of study at college?) and the set of items. The items derive from a list of popular majors as well as from inclusion of specifically science, engineering, mathematics, and technology (STEM) majors. Questions about the items include: Is the list overwhelming to respondents; are individual items clear; do students perceive overlap between items; and with what certainty can 11th graders identify items.
5. [For those with a college intention (answer 1 or 2 on question #1)] Earlier, you told us you planned to enroll in college after you finished high school. Which career college, community college, or other college or university would you most likely attend? [For those with no college intention (answer 3 or 4 on question #1)] Earlier, you told us that you did not plan to continue your education or were unsure about continuing your education after high school. If you did continue your education at a career college, community college, or other college or university, which school would you most likely attend?
Open-ended response (college name text)
Don’t know [skip next question]
Justification: This is a new item. There is concern that 11th graders may not be able to identify a college, and uncertainty about how they might pick one (based on awareness of a prominent college or state university? Based on a nearby school?). This question is an anchor point for subsequent question about college costs. Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
6. [For those with a college intention (answer 1 or 2 on question #1) and a specifically named college in question #5] Why do you have this specific college in mind? Please choose the most important reason:
Response options:
1 It is close to home
2 There is a lot to do nearby
3 Overall quality of its academic programs
4 A specific academic program
5 Good sports teams
6 It is affordable
7 A good social life
8 A good record of placing graduates in jobs
9 Friends or relatives are there
10 Other—please specify
Justification: This is a new question that is the college-specific version of the question about the major factor in choosing a college. The goals are the same as with the prior question: to ascertain any confusion about response options, perceptions of overlap between options, and alternative options. Also, the confidence or certainty of answers is a focus; there is a concern that such a question may be too early for 11th graders to answer with definitiveness. Finally, we can compare student interpretations of this question with the generic one.
7. How likely is it that difficulty in paying for college will keep you from attending school?
Response options:
1 Very likely
2 Somewhat likely
3 Somewhat unlikely
4 Very unlikely
Justification: This is a new question. Though the question is straightforward, we would like to understand if the question is beyond an 11th grader’s consideration—are they likely to have thought about it? What do “unlikely” responses represent? Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
8. How likely is it that difficulty in paying for college will keep you from attending your preferred school?
Response options:
1 Very likely
2 Somewhat likely
3 Somewhat unlikely
4 Very unlikely
Justification: This is a new question. Though the question is straightforward, we would like to understand if the question is beyond an 11th grader’s consideration—are they likely to have thought about it? For students without a preferred school, how do they interpret it? What do “unlikely” responses represent—a likely expensive school or lack of family resources? Also, how do students without any college intention, or with no specific college indicated, interpret this question—is it confusing or strange to them to be asked this?
9. On average, how much do you think a year of school costs at [COLLEGE/”college” if “don’t know” response from question #5]? Please do not include living costs such as on-campus housing (room or board) or rent and utilities.
Open-ended response (dollar figure)
Don’t know
Justification: This is a new question. The goal of cognitively testing this question is to understand whether students can adequately distinguish tuition and mandatory fee costs (the aim of the question) from other costs often bundled with college costs. Other questions include: With what certainty are students providing college cost estimates? Are students easily drawn to answering “don’t know” in order to avoid guessing? What might be ways of mitigating that possibility, yet ensuring reasonably thoughtful answers? Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
10. To help pay for their education, students sometimes receive scholarships or grants that do not have to be paid back. How much of this help do you think you would receive each year if you went to [COLLEGE/”college” if “don’t know” response from earlier]?
Open-ended response (dollar figure)
Don’t know
Justification: This is a new question. The goal of cognitively testing this question is to understand whether students understand that this is a non-loan question, and their interpretation of the sources or kinds of scholarships and grants being included in the question (all types are intended). Other questions include: With what certainty are students providing non-loan estimates? Are students easily drawn to answering “don’t know” in order to avoid guessing? What might be ways of mitigating that possibility, yet ensuring reasonably thoughtful answers? Would reversed ordering with the subsequent loan question be more sensible? Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
11. [For those with a college intention (answer 1 or 2 on question #1) How much would you or your family be willing to borrow in the form of a loan to pay for a year of school at [COLLEGE/”college” if “don’t know” response from earlier]? [For those with no college intention (answer 3 or 4 on question #1)] If you went to [COLLEGE/”college” if “don’t know” response from earlier], how much would you or your family be willing to borrow in the form of a loan to pay for a year of school?
Open-ended response (dollar figure)
Don’t know
Justification: This is a new question. The goal of cognitively testing this question is to understand whether students understand that loans must be paid back (as distinct from the prior question) and whether all types of loans are being asked about (federal, private). Also, do students make a specific interpretation of “you or your family?” I.e., might they answer differently if this were worded “you” only or “you and your family?” Other questions include: With what certainty are students providing loan estimates? Are students easily drawn to answering “don’t know” in order to avoid guessing? What might be ways of mitigating that possibility, yet ensuring reasonably thoughtful answers? Would reversed ordering with the prior non-loan question be more sensible? Is the lack of parallelism with the prior question (here, asking willingness, with the prior question asking how much they expect to receive) pose a problem in comprehension? Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
1. Have you participated in any of the following activities to prepare for life after high school? Please check all that apply.
Attended career day or job fair;
Attended college night;
Visited college campus;
Taken an internship or apprenticeship;
Done any job shadowing;
Searched internet for college options;
Talked with school counselor about options;
Talked with teachers about options;
Talked with friends about options;
Talked with parents about options;
Sat in on or taken a college class;
Met with college athletic team or coach;
Other—please specify
Response options:
1 Yes
2 No
Justification: This is a question adapted from a Ferris State University survey. The cognitive testing should help determine if there is any overlap or redundancy in the item list; how certain specific items are interpreted (does talking with friends include online interaction?); and whether other options might be included.
2. What do you look forward to most at school?
Response options:
1 Don’t look forward to anything
2 Sports
3 Physical education/gym
4 Seeing friends
5 Art, music, or theater
6 Mathematics
7 Science
8 English
9 Another subject—please specify
10 Other—please specify
Justification: This is a new item. The goal is to ensure that students are interpreting the question to include both extracurricular and curricular activities; that response options are sensible to them, particularly the combination of both “other subject” and “other” options; and that there aren’t other key items that could be included.
3. Have you taken any courses that have a specific occupational orientation? These include courses in areas such as agriculture, business, computer technologies, construction, or health care.
Response options:
1 Yes [Go to next question, then skip subsequent question]
2 No [Skip next question]
Justification: This is a new question. It is necessary for subsequent questions, but it also can be tested for understanding of “specific occupational orientation.” Are the provided examples helpful and sufficient (or superfluous)?
4. If yes, what was the MAIN reason for taking these courses?
Response options:
1 To meet graduation requirements;
2 Personal interest in the subject;
3 To prepare for college;
4 To prepare for a job you are interested in;
5 To explore a career option;
6 To learn things that will be useful for work in general.
7 Other—please specify
Justification: This is a new question. Though intended to identify a single reason, students may see multiple response options as applicable; cognitively testing will help examine how students interpret and answer the question in that situation. Overlap and missing options are also a focus.
5. What is the MAIN reason you haven't taken such a course?
Response options:
1 Not really related to what you want to do;
2 You don't know what occupation you want;
3 No such courses are offered through your school;
4 Those courses are off-site and require travel;
5 Other courses are more important;
6 Other—please specify
Justification: This is a new question. Though intended to identify a single reason, students may see multiple response options as applicable; cognitive testing will help examine how students interpret and answer the question in that situation. Determining any overlap and missing options is also a focus for testing this question.
6. Have you received college credit for any of your high school courses?
Response options:
1 Yes, in math
2 Yes, in science
3 Yes, in another subject
4 No
5 I don't know
Justification: This is a question from the base year survey intended to identify students with a dual enrollment experience. There is a concern that some students may not know if they’ve received dual enrollment, or may count some courses (e.g. advanced placement) as dual enrollment when they are not. The cognitive lab will ask respondents about this issue.
7. Which one of the following BEST describes how you have decided which high school courses to take this year and next year?
Response options:
1 I am taking the courses needed to meet high school graduation requirements
2 I am taking courses to prepare for a job after high school
3 I am taking the courses needed to meet college entry requirements
4 I am taking the courses needed to do well in college
5 I am taking the courses that best match my interests and abilities
Justification: This is a new question. Though intended to identify a single reason, students may see multiple response options as applicable; cognitively testing will help examine how students interpret and answer the question in that situation. Overlap and missing options are also a focus.
8. How likely is it that difficulty in paying for college will keep you from attending school?
Response options:
1 Very likely
2 Somewhat likely
3 Somewhat unlikely
4 Very unlikely
Justification: This is a new question. Though the question is straightforward, we would like to understand if the question is beyond an 11th grader’s consideration—are they likely to have thought about it? What do “unlikely” responses represent? Also, how do students without any college intention interpret this question—is it confusing or strange to them to be asked this?
9. How much money do you think you would earn in a year of working if you had the following levels of education? Please provide your best guess.
If I left high school today, I'd make $___ in a year.
If I finish high school with a GED, , I'd make $___ in a year
If I finish high school with a diploma, I'd make $___ in a year
If I finish college, I'd make $___ in a year
Justification: This is a new question gauging expectations of returns to education. Though a narrowly economic interpretation of returns to schooling, it may provide helpful information about why students make decisions to complete given levels of education. Questions to be addressed in the cognitive lab include: do students distinguish between different categories, and could a further category (graduate or professional degree) be useful? How are students creating estimates—are they calculating based on smaller periods (hourly, weekly wages) or using an income level as a base for a given educational level and then adding/subtracting some amount to obtain the others? Will students not respond because they lack information or the is the question too demanding, and if so, how can this possibility be identified and mitigated?
10. How certain are you about your answers to the previous question? Please answer from 0 to 100, with 0 being “completely uncertain” and 100 being “completely certain.”
0-100 response
Justification: This new item tests a specific way of inquiring about certainty, which both addresses an issue with the prior question and investigates whether students are likely to provide a reasonable diversity of responses in this format.
During the last 12 months, which of the following activities have you or another family member done with your teenager?
Visited a zoo, planetarium, natural history museum, transportation museum, or a similar museum
Worked or played on a computer together
Built or fixed something such as a vehicle or appliance
Attended a school science fair
Helped [subject] with a school science fair project
Discussed a program or article about math, science, or technology
Visited a library
Gone to a play, concert, or other live show
Other specify
None of these
Justification: This question was new in the base year of HSLS. The age appropriateness of the items will be evaluated. Parents will be asked to identify other STEM activities in which they may engage with their teenager.
A. By the time your teenager graduates from high school, do you think he/she will have the minimum grade point average needed for admission to…
a 2-year community college or technical institute?
a 4-year public college in your state?
a 4-year public college out of state?
a 4-year private college?
Yes
No
Don't know
By the time your teenager graduates from high school, do you think he/she will have completed the minimum courses needed for admission to…
a 2-year community college or technical institute?
a 4-year public college in your state?
a 4-year public college out of state?
a 4-year private college?
Yes
No
Don't know
Justification: This newly written pair of questions will be asked of all parents. The goal of testing will be to determine if the questions make sense for each type of institution. Should these questions be asked about 2-year colleges since they only require a high school diploma or GED? Also, can parents meaningfully answer this without a specific institution in mind? We will also use cognitive testing to determine if GPA or course credits are more relevant. Another objective will be to see if parents understand “technical institute.”
How have you gotten information about financial aid?
Talked with financial aid office staff at a college
Talked with a school counselor at [subject]'s high school
Researched financial aid options on the Internet
Other (specify)
Justification: This is a new question. The goal of testing this item will be to determine whether parents have gotten information about financial aid yet and if so, if there are ways that are not represented in the response options.
A. As far as you know, would your teenager qualify for financial aid for college?
Definitely
Probably
Probably not
Definitely not
Don't know
B. Why do you think your teenager would not qualify for financial aid?
Family income is too high to qualify
Student’s grades or test scores are too low to qualify
Part time student ineligible for aid
Family has bad credit
Another family member did not qualify for aid
Other specify
Yes
No
Justification: The first question will not be cognitively tested. It is just used to determine which respondents should answer the follow-up question. Parents who answer “probably not” or “definitely not” will be asked B.
The follow-up question is new, but partly based on a question from NPSAS. The primary goals of cognitive testing the follow-up question are to evaluate if parents of 11th graders can answer this question or if it is too early, to identify other reasons that are not listed, and to evaluate clarity.
Families have different ideas about what they would like their teenagers to do in the first year after high school. How strongly would you encourage or discourage your teenager from choosing each of the following activities as his/her main activity?
Enroll in an Associate's degree program in a two-year community college or technical institute
Enroll in a Bachelor's degree program in a college or university
Obtain a license or certificate in a career field
Participate in a registered apprenticeship program
Join the armed services
Get a job
Start a family
Travel
Do volunteer or missionary work
Strongly Encourage
Encourage
Neither encourage nor discourage
Discourage
Strongly Discourage
Justification: This is a new question that has never been fielded. We have several objectives for cognitive testing. We want to learn whether a 5-point scale provides sufficient degrees of intensity. In addition, we seek to know whether the responses will be sensitive to the order and the mode in which the items are presented (i.e., visually versus orally). We also will evaluate clarity of terms such as technical institute, license, certificate, registered apprenticeship, etc. Finally, we will determine if community college should be separated from technical institute.
A. What is your teenager most likely to do as his/her main activity in the first year after high school?
Enroll in an Associate's degree program in a two-year community college or technical institute
Enroll in a Bachelor's degree program in a college or university
Obtain a license or certificate in a career field
Attend a registered apprenticeship program
Join the armed services
Get a job
Start a family
Travel
Do volunteer or missionary work
Something else (specify)
B. There are many reasons why people do not enroll in an Associate’s or Bachelor’s degree program in the first year after high school. Are any of the following reasons true for your teenager?
[Subject's] grades are not high enough
[Subject's] college admission test scores are not high enough
[Subject] cannot afford it
[Subject] does not have transportation to and from school
[Subject] needs to earn money to support family
[Subject] does not know what [he/she] wants to study yet
[Subject] does not like school
[Subject] will not need an Associate's or Bachelor's degree for the job [he/she] wants
[Subject] wants a break from school
Personal reasons
[His/Her] friends are not going to enroll in a degree granting program
[Subject] has health problems
Does not want to go into debt to pay for school
Other reason (specify)
Yes
No
Justification: The first question will not be cognitively tested. It is just used to determine which respondents should answer the follow-up question. Parents who think their student will not enroll in an Associate’s or Bachelor’s degree program will be asked B.
The primary goals of cognitive testing the new follow-up question are to identify other reasons that are not listed and to evaluate clarity.
Which of the following aspects of a college are important to you in choosing a college or university for your teenager to attend?
Close to home
A lot to do nearby
Overall quality of academic programs
Has a specific academic program
Has good sports teams
It is affordable
Has a good social life
Good record of placing graduates in jobs
Friends or relatives are there
Other (specify)
Very important
Somewhat important
Not at all important
Justification: The goal for cognitive testing is to evaluate the set of options listed for comprehensiveness and comprehensibility. In both the cognitive test and the field test we will elicit other options by way of an “other” category in which respondents can specify. A similar question is also asked of students.
If your child chooses to attend college, what are the sources of support you anticipate your child will use to pay for college costs, including tuition, room, and board?
The student will pay from their earnings and savings
I/We (parents/guardians) will pay
The student is the beneficiary of a pre-paid tuition program or 529 plan
Other relatives (e.g. grandparents) will pay
The student will receive financial aid based on financial need
The student will receive financial aid based on academic merit
The student will receive financial aid through an athletic scholarship
The student will take out a loan or loans
I/We (parents/guardians) will take out a loan or loans
Other, Please specify: __________________________
Don't know
Yes
No
Justification: This question was used previously in a small survey. We would like to more rigorously test its viability. The question will be evaluated for clarity (e.g., what is included in college costs, what is a pre-paid tuition program, academic merit). We will also evaluate whether the set of options is exhaustive of the most common sources of support.
Do you believe that your teenager should help to pay for his/her own education after high school?
Yes
No
Justification: The objective of testing this question will be to determine whether parents will be able to answer this question without further information. Does the answer depend on circumstances or what costs we cite specifically (e.g., tuition versus room and board)?
Approximately what percentage of the total costs of his/her education do you think he/she should finance?
Justification: The objective of testing this question will be to determine whether parents will be able to answer this question without further information. Does the answer depend on circumstances or what costs we are referring to (e.g., tuition versus room and board)? Also, should the response be open-ended or categorical?
Which of the following steps do you expect him/her to take to finance his/her education?
Work during high school evenings and/or weekends
Work during high school summers
Work up to 20 hours per week while attending college
Work more than 20 hours per week while attending college
Yes
No
Justification: The objective of testing this question will be to determine if there are other items that should be added.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Memorandum |
| Author | mcominole |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |