Cognitive Interviews for the 2010 National Household Education Survey (NHES) Study Draft Questionnaires

ECPP Protocol

General Introduction

I would like to begin by thanking you for coming here today to help us out. We’re scheduled to be here for 90 minutes, but I’ll let you go early if we finish sooner. The reason we’ve asked you to come is that the US Department of Education conducts a survey roughly every two years to describe children’s’ educational experiences, starting in early childhood. Today I’m going to ask you to fill out a questionnaire that is being developed for 2011 and to provide feedback so that we can help the Department of Education make sure the questionnaire will be understood by all different types of parents.

Because it would be hard to keep track of everything you say today, we're going to tape-record this session. And I’ll also be taking notes, so you might notice me writing as we are talking. There’s a one-way mirror behind me, and there are a few of my coworkers behind it watching. I want you to know that your responses are voluntary and confidential.

In just a minute I'm going to hand you the questionnaire and I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions.

There will also be a few questions where I will ask you to think “out loud” as you answer them. For these few questions, it is important that you tell me what you are thinking and feeling while you are actually in the process of coming up with your answers.

Now because some people aren't used to reading, thinking and expressing their feelings aloud, I'd like to begin today with a short practice question. I’d like you to practice with this by reading the question I’m about to give you out loud and telling me your thoughts and feelings from the moment I hand you the question until you are finished explaining your answer.

QUESTION: How many times did you eat a meal out at a restaurant in the past week?

Thank you. I’m going to hand you the questionnaire now. Please read it and fill it out as if you were actually responding to the survey. As you go through the survey, if any question sticks out to you in any way (maybe you think a question is hard to answer, hard to understand, in a strange place, or you just don’t like it)—anything at all that you would like to say about an item, please tell me so that we can discuss it.

For the purposes of this survey, the sampled child is ____________________________

(GIVE QUESTIONNAIRE TO RESPONDENT)

Question-by-Question Follow-Ups

PAGES 1-3	Did respondent carefully read instructions?
	Did respondent have any concerns or questions about the instructions?
SECTION 1
Q1-16	If respondent answers yes to Q1 and Q2, stop them after Q16 to provide feedback on this section. Interrupt them at Q13 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q1	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What does the term “care” mean to you in this question?
	What types of child care did you think were covered in this question?
	Which child were you thinking about? How easy or difficult is it to keep your focus on the “sampled child” as you are answering these questions?
Q6-7	How did you come up with your answers to these two questions?
Q11	What is this question asking, in your own words? (What is meant by “charge” or “fee for care”?)
Q12	Was there anything confusing about this question?
	(If respondent marked “yes” to any in Q12) What were you thinking when you answered “yes” to this?
Q13	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q1-16	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
Q17-33	If respondent answers yes to Q17 and Q18, stop them after Q33 to provide feedback on this section. Interrupt them at Q30 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q17	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What types of child care did you think were covered in this question?
Q21-22	How did you come up with your answers to these two questions?
Q28	What is this question asking, in your own words? (What is meant by “charge” or “fee for care”?)
Q29	Was there anything confusing about this question?
	(If respondent marked “yes” to any in Q29) What were you thinking when you answered “yes” to this?
Q30	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q17-33	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
Q34-49	If respondent answers yes to Q34 and Q35, stop them after Q49 to provide feedback on this section. Interrupt them at Q46 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q34	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What types of child care did you think were covered in this question?
Q36A	How did you arrive at your answer for this question?
Q36B	What did you think of the definition that was provided? How certain are you of your answer?
Q36C	How did you come up with your answer to this question? Was it difficult to narrow it down to one response? How would you determine when to use the option for “its own building or storefront”?
Q37-38	How easy or difficult were these questions to answer? How were they different from Q36C?
Q39-40	How did you come up with your answers to these two questions?
Q41	What do you think “program” means in this question?
Q43	What was this question asking, in your own words? What is meant by “provide”?
Q44	What was this question asking? What is meant by charge or fee for care?
Q45	Was there anything confusing about this question?
	(If respondent marked “yes” to any in Q45) What were you thinking when you answered “yes” to this?
Q46	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q34-49	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Did you find the arrows and the instructions easy or difficult to follow?
	Are there any other items in this section you would like to comment on?
SECTION 2
Q50-55	Did you find any questions in this section to be difficult to answer?
Q50	Did you pay attention to the callout box? Was there anything confusing about the definition?
Q51	If more than one arrangement, how did you come up with your answer to this?
Q52	What did you think about the response choices offered for this question? Was anything missing?
Q53	What were you thinking when you answered this question?
Q55	What did you think of these categories? Were there any other major reasons why you chose the child care arrangement you did?
Q50-55g	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 3
Q57-61	For this whole section, please think out loud as you are coming up with your answers to these questions. Tell me what is going through your mind as you determine how to answer them.
	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 4
Q62-69	Did you find any questions in this section to be difficult to answer?
Q68	What is this question asking, in your own words?
Q62-69	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 5
Q70-78	Did you find any questions in this section to be difficult to answer?
Q71	Were there any items in this question that you did not understand or did not know how to answer?
	What did the term “condition” mean to you and how did you determine whether or not to classify something as a condition?
Q72	Please tell me how you interpreted the skip pattern before Q72. Is there anything that could be changed to make this clearer?
	What was Q72 asking, in your own words? What does the term “services” mean to you in this question?
Q74	What was this question asking, in your own words?
Q75	What was this question asking, in your own words?
Q70-78	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 6
Q79-86	Did you find any questions in this section to be difficult to answer?
SECTION 7
Q87	Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it. Did you feel any confusion about who to think about when responding?
Q88	Who are you answering this question about? How did you determine who the “first” parent or guardian is?
	Did you notice the instruction at the bottom of the 1st column? How could this instruction be more clear?
Q103	Who were you thinking about when you answered Q103?
Q87-117	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 8
Q121	Please tell me what this question is asking, in your own words.
Q126	What were you thinking as you answered this question? How easy or difficult was it for you to determine how to answer these questions?
Q121-127	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 9
Q128-138	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?
	Okay, that’s all the questions. I want to thank you very much for all of the help you have given us. We appreciate it very much.

7