2014 National Household Education Survey (NHES) After School Programs and Activities (ASPA) Cognitive Interviews

NHES ASPA Cognitive Interview Protocols

July 19, 2013

Part A. After School Programs and Activities (ASPA)

General Introduction

I would like to begin by thanking you for coming here today to help us out. <This is my colleague from the U.S Department of Education who will be listening in on the interview today.> We’re scheduled to be here for 60 minutes, and I won’t keep you any longer. The reason we’ve asked you to come is that the US Department of Education conducts a survey roughly every two years to describe children’s’ educational experiences, starting in early childhood. Today I’m going to ask you to fill out a questionnaire that is being developed for 2014 and to provide feedback so that we can help the Department of Education make sure the questionnaire will be understood by all different types of parents. I don’t work for the Department of Education, so feel free to be honest with your comments. My role is to find out from you how YOU interpret the questions.

• Because it would be hard to keep track of everything you say today, we're going to audio-record this session.

• And I’ll also be taking notes, so you might notice me writing as we are talking.

• I want you to know that your responses are voluntary and confidential. You do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

In just a minute I'm going to hand you the questionnaire and I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions.

I would also like you to think “out loud” as you answer. That is, tell me how you go about answering a question, tell me if something doesn’t sound right, is hard to answer, etcetera.

Now because some people aren't used to reading, thinking and expressing their feelings aloud, I'd like to begin today with a short practice question. I’d like you to practice with this by reading the question I’m about to give you out loud and telling me your thoughts and feelings from the moment I hand you the question until you are finished explaining your answer.

QUESTION: How many times did you eat a meal out at a restaurant in the past week?

Thank you…

SCREENER
	First I’d like you to fill out this form. HAND RESPONDENT THE SCREENER. Think out loud as you answer. Tell me what is going through your mind and if anything is unclear or you are unsure of something as you fill it out.
Q6	WHEN FINISHED Let’s look at question 6. Please tell me in your own words what this question is asking. What does the phrase “homeschool instead of school for some or all classes” mean to you?
Q4	Now let’s look at question 4. How did you arrive to the answer for this question? How did you react to being asked this question? How do you think people will react to being asked about their child’s month and year of birth?
ALL	Was there anything confusing, awkward, or unclear about this form?

Okay. Now here is the survey, please read it and fill it out as if you were actually responding to the survey. As you go through, if any question sticks out to you in any way (maybe you think a question is hard to answer, hard to understand, in a strange place, or you just don’t like it)—anything at all that you would like to say about an item, please tell me so that we can discuss it. Remember that I’ll be stopping you here and there as you complete the survey so that we can discuss it.

(GIVE QUESTIONNAIRE TO RESPONDENT)

Specific probes for main questionnaire

INTERVIEWER - If you feel there was uncertainty about a respondent’s answer, probe retrospectively on those items. Pay attention to respondent behavior here. Do they hesitate? Ask questions? Make remarks that indicate the question is awkward? If yes, PROBE.

Pay attention to SKIP errors. Do they have trouble figuring out which question to go to next? Do they go to the wrong question? If you notice they go to the wrong question, give them a chance to figure it out on their own, but if they don’t, point out to them that they skipped to the wrong question, PROBE about what was confusing about it. NOTE these errors.

Pay attention to how they answer the multiple response items where they are supposed to mark either yes or no for all response options. Do they skip some?

After the respondent completes each section:

General Probes
Problem items	What is this question asking, in your own words?
	What were you thinking about when you answered this question?
Skip and marking issues	I noticed you skipped this/checked this. Did you notice the instructions here?
	What were you thinking about when you answered this question?
Child’s Schooling
Please stop here for a minute so I can ask you a few questions.
Q2	Is this child homeschooled instead of going to a public or private school for some classes or subjects? What does this question mean to you? Can you tell me more about your answer? If answer is “Yes” How did you arrive to your answer of number of hours the child attends a public or private school?
Q3	Now let’s look at the next question. What type of school does this child attend? What is this question asking, in your own words? Can you tell me more about your answer? If answer is “Yes” How did you arrive to your answer?
IF APPLICABLE:
Q4	Is it his/her regularly assigned school? What is this question asking, in your own words? What does the term “regularly assigned” mean to you? I’d like to know the name of your child’s school so we can make sure our questions are the right questions to ask about local families’ schools.
Q7	If answer “Mostly A’s” Let’s look at this question. Please tell us about this child’s grades during this school year. Overall, across all subjects, what grades does this child get? I see your child receives mostly A’s. Does your child have an education plan with the school for high achieving children? If yes, can you tell me more about the plan or program? What is the name of the plan or program?
ASPA Specific Questions
Qs 21, 22 “How important” Grid Questions	How important was each of the following reasons when your family was making decisions about where this child spends after-school hours and weekends? Mark (X) one box for each line. Note to interviewer: Pay attention to whether the respondent marks somewhat important or very important to all the items. If yes, then probe on that: I noticed you answered somewhat important and very important for these questions. Can you tell me more about that? Was there a reason that you thought was missing from the list? Is there another reason that was not included in the table? If yes, can you tell me about that? How did you decide how to rate these reasons? What were you thinking about as you were rating these reasons? Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Were there any words or phrases that you did not understand? If yes, what were they? When does your child get out of school? During what hours were you thinking about when you answered this question? Other than the reasons above, how important was each of the following reasons when your family was making decisions about where this child spends after-school hours and weekends? Mark (X) one box for each line. Note to interviewer: Pay attention to whether the respondent marks somewhat important or very important to all the items. If yes, then probe on that: I noticed you answered somewhat important and very important for these questions. Can you tell me more about that? Was there a reason that you thought was missing from the list? Is there another reason that was not included in the table? If yes, can you tell me about that? How did you decide how to rate these reasons? What were you thinking about as you were rating these reasons? Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Were there any words or phrases that you did not understand? If yes, what were they? During what hours were you thinking about when you answered this question?
Qs 24 to 27 CALENDAR	During last week (or the most recent typical week) this school year, where is this child after school hours and weekends? For each day of the week below, mark [X] the box that corresponds to the location where this child spends at least 30 minutes time after school hours and weekends. Mark all that apply. Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Was there a location that you thought that was missing from the list? Is there another location where your child spends time that was not included in the table? If yes, can you tell me about that? Can you tell me more about where your child spends time after school hours and on the weekends? Were there any words or phrases that you did not understand? If yes, what were they? During what hours were you thinking about when you answered this question about where your child spends time after school hours and on the weekend? Did you report on all locations or did you just include where the child spent 30 minutes or more? During last week (or the most recent typical week) this school year, who looks after this child after school and weekends? For each day of the week, mark [X] the box with the individual who typically looks after for this child after school and weekends for at least 30 minutes. Mark all that apply. Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Was there anyone that you thought that was missing from the list? Is there another person who looks after your child that was not included in the table? If yes, can you tell me about who that person is? Can you tell me more about who your child spends time with after school hours and on the weekends? Were there any words or phrases that you did not understand? If yes, what were they? During what hours were you thinking about when you answered this question about who your child spends time with after school hours and on the weekend? Did you report on all people or did you just include who the child spent 30 minutes or more with? During last week (or the most recent typical week) this school year, which of the following activities does this child do after school and weekends, if any? For each day of the week, mark [X] the box if this child typically participated participates in the listed activity. Mark all that apply. Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Was there an activity that you thought that was missing from the list? Is there another activity that was not included in the table? If yes, can you tell me about that activity? Can you tell me more about activities your child does after school hours and on the weekends? Were there any words or phrases that you did not understand? If yes, what were they? During what hours were you thinking about when you answered this question about activities your child participates in after school hours and on the weekend? Did you report on all activities or did you just include activities where the child spent 30 minutes or more? During last week (or the most recent typical week) this school year, which of the following activities does this child do after school and on weekends, if any(CONTINUED/DIFFERENT SET OF ACTIVITIES)? For each day of the week, mark [X] the box if this child typically participates in the listed activity. Mark all that apply. Can you tell me in your own words what this question is asking? Was this table format easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete? Was there an activity that you thought that was missing from the list? Is there another activity that was not included in the table? If yes, can you tell me about that activity? Can you tell me more about activities your child does after school hours and on the weekends? Were there any words or phrases that you did not understand? If yes, what were they? During what hours were you thinking about when you answered this question about activities your child participates in after school hours and on the weekend? Did you report on all activities or did you just include activities where the child spent 30 minutes or more?
Formal After School Programs	32. How many total hours each week does this child spend at this after-school program? Can you tell me what this question is asking in your own words? If respondent marks that child is in more than one after-school program in the previous question, ask: How did you determine which after-school program to report on? Was this question easy or difficult to answer? If difficult, what made it difficult? These next questions ask about the after-school program where this child spends the most time. Note to interviewer: If the respondent reports that the child has more than one formal after-school care arrangement, we want to probe on how they answer the questions in this section that asks about where the child spends the most time. How did they choose which program to report about? Were they consistent about reporting on that one arrangement? 35. Do any of the following people, programs, or organizations help pay for this child to go to the program? Can you tell me what this question is asking in your own words? Can you tell me what “Temporary Assistance for Needy Families, or TANF” is in your own words? 40. Is this program run by his or her school? How many programs does child participate in that are run by his or her school? Can you tell me in your own words what this question is asking? What does “run by” mean to you in your own words? Can you tell me more about this program? Is it run by the school or is it run by an outside organization in the school?
Organized After-School Activities	How many activities does child participate in that are run by his or her school? Can you tell me more about these activities? Is it run by the school or is it run by an outside organization at the school?
Child Health Section
Q56	Has a health or education professional told you that this child has any of the following conditions? Let’s look at question 42. Were there any parts of this question that you did not understand or did not know how to answer? Were you able to find a suitable answer option, or do you think there were any options missing? Did you read all the response options? How far did you get?
Q57	Did you mark yes to any condition in question 56? For question 57 how did you arrive at your answer?
IF APPLICABLE:
Q59	Are any of these services provided through an Individualized Education Program (IEP)? What does the term “Individual Education Program (IEP)” mean to you? For private school students who answer yes – Is this the right term? For private school students who answer no – Does your child have some type of education or service plan with the school? What is it called?
(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
ALL	In Section 6 did you find the arrows and the instructions easy or difficult to follow?
ALL	In section 6 were there any other questions that were confusing or a bit difficult for you to answer? Which ones? PROBE.
Child’s Background Stop after section
Q71 If marks yes	Split time between households Can you describe this child’s living arrangements? How much time does he or she spend there? Which place do you consider to be the child’s primary home?
Q72	What language does this child speak most at home? Let’s look at question 72. Please, tell me in your own words what this question is saying. Did you find the arrows and the instructions easy or difficult to follow?
IF APPLICABLE:
Q73	Is this child currently enrolled in English as a second language, bilingual education, or an English immersion program? Now let’s look at question 73. What does the phrase “English as a second language” mean to you? What does the phrase “bilingual education” mean to you? What does the phrase “English immersion” mean to you?
(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
ALL	In section 7 were there any other questions that were confusing or a bit difficult for you to answer? Which ones? PROBE.
You can continue from question 74 and remember to tell me what you are thinking.
Child’s Family STOP RESPONDENT AFTER 78
I’d like you to stop here.
Q76	Marital status What do you think this question is getting at? Were you able to find a suitable answer option, or do you think there were any options missing?
IF APPLICABLE:
Q77	Living with partner Please, tell me in your own words what this question is asking?
Q78	registered domestic partnership or civil union? What do you think this question is getting at? How did you react to being asked this question?
(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
ALL	In section 7 were there any other questions that were confusing or a bit difficult for you to answer? Which ones? PROBE.
You can continue from question 79 and remember to tell me what you are thinking.
Q89 - I’d like you to stop here. IF APPLICABLE:
Q89	What do you think this question is getting at? Were you able to find a suitable answer option, or do you think there were any options missing?
IF APPLICABLE:
Q93	Please, tell me in your own words what this question is asking?
(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
ALL	In remaining part of section 8 were there any other questions that were confusing or a bit difficult for you to answer? Which ones? PROBE.
You can continue from question 99 (IF APPLICABLE) and remember to tell me what you are thinking.
Your Household STOP RESPONDENT AFTER 120
I’d like you to stop here.
Q119	How many of the following people live in this household with this child? Let’s look at question 1119. In your own words, who lives in your household? Did you feel any confusion about who to think about when responding to this question? How easy or difficult did you find this question to answer?
Q120	Enter the total number of people living in this household with this child. (This number should be equal to the sum of a through k above). Now let’s look at question 120. Please tell me what this question is asking, in your own words. How did you arrive to your answer?
Go ahead and finish the rest of the survey now and let me know if any questions are confusing or difficult to answer in any way or if you are unsure of anything.
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? REMEMBER TO ASK NCES STAFF FOR INPUT, GIVE GIFT CARD AT END OF INTERVIEW
	Thank you…

Additional Notes:

7