The Adult Training and Education Survey (ATES) 2010 Pilot Test and the Spanish Cognitive Interviews for the 2010 National Household Education Survey (NHES) Study Draft Questionnaires

ECPP Protocol

General Introduction

I would like to begin by thanking you for coming here today to help us out. We’re scheduled to be here for 90 minutes, and I won’t keep you any longer. The reason we’ve asked you to come is that the US Department of Education conducts a survey roughly every two years to describe children’s’ educational experiences, starting in early childhood. Today I’m going to ask you to fill out a questionnaire that is being developed for 2011 and to provide feedback so that we can help the Department of Education make sure the questionnaire will be understood by all different types of parents.

• Because it would be hard to keep track of everything you say today, we're going to tape-record this session.

• And I’ll also be taking notes, so you might notice me writing as we are talking.

• There’s a one-way mirror behind me, and there are a few of my coworkers behind it watching.

• I want you to know that your responses are voluntary and confidential.

In just a minute I'm going to hand you the questionnaire and I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions.

There will also be a few questions where I will ask you to think “out loud” as you answer them. For these few questions, it is important that you tell me what you are thinking and feeling while you are actually in the process of coming up with your answers.

Now because some people aren't used to reading, thinking and expressing their feelings aloud, I'd like to begin today with a short practice question. I’d like you to practice with this by reading the question I’m about to give you out loud and telling me your thoughts and feelings from the moment I hand you the question until you are finished explaining your answer.

QUESTION: How many times did you eat a meal out at a restaurant in the past week?

Thank you…

SCREENER LETTER and CONNECTION BETWEEN SURVEYS

When you came to here today, you were given this package in the waiting area and asked to complete the survey inside.

Could you tell me what you did with the letter when you opened the package?

What message did you take away from it?

Any other thoughts you had about the letter? Anything specific jump out at you about it?

And after you completed the first survey in the waiting area, did you have any impressions of it?

If you received that first survey at home, would you be expecting something else to come?

Thank you. I’m about to hand you the questionnaire. When I hand you the questionnaire, please read it and fill it out as if you were actually responding to the survey. As you go through the survey, if any question sticks out to you in any way (maybe you think a question is hard to answer, hard to understand, in a strange place, or you just don’t like it)—anything at all that you would like to say about an item, please tell me so that we can discuss it.

Remember that I’ll be stopping you here and there as you complete the survey so that we can discuss it.

(GIVE QUESTIONNAIRE TO RESPONDENT)

Here is the package you would receive in the mail at home. Go ahead and open this as you would if you were at home. Talk to me about it.

What parts of the package does the respondent look at?:

PAGES 1-3	Did respondent carefully read instructions?
	Did respondent have any concerns or questions about the instructions?
	What impressions did you get from the cover?

ENVELOPES

Would one of these mailings stand out more to you than the other? Why or why not?

Would one of these envelopes draw your attention more than the other?

Any other thoughts on receiving this first survey and then receiving this second one?

FAQ	Take a look at the FAQs. Do you think these are useful? Anything you would add?

Please continue with the questionnaire.

Question-by-Question Follow-Ups

PAGES 1-3	Did respondent carefully read instructions?
	Did respondent have any concerns or questions about the instructions?
SECTION 1
Q1-16	If respondent answers yes to Q1 and Q2, stop them after Q16 to provide feedback on this section. Interrupt them at Q13 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q1	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What does the term “care” mean to you in this question?
	What types of child care did you think were covered in this question?
	Which child were you thinking about? How easy or difficult is it to keep your focus on the “sampled child” as you are answering these questions?
Q6-7	How did you come up with your answers to these two questions?
Q11	What is this question asking, in your own words? (What is meant by “charge” or “fee for care”?)
Q12	In your own words, what is this question asking?
	(If respondent marked “yes” to any in Q12) What were you thinking about when you answered “yes” to this?
Q13	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q1-16	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
Q17-33	If respondent answers yes to Q17 and Q18, stop them after Q33 to provide feedback on this section. Interrupt them at Q30 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q17	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What types of child care did you think were covered in this question?
Q21-22	How did you come up with your answers to these two questions?
Q28	What is this question asking, in your own words? (What is meant by “charge” or “fee for care”?)
Q29	In your own words, what is this question asking?
	(If respondent marked “yes” to any in Q29) What were you thinking about when you answered “yes” to this?
Q30	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q17-33	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
Q34-49	If respondent answers yes to Q34 and Q35, stop them after Q49 to provide feedback on this section. Interrupt them at Q46 to do a “concurrent” think aloud.
Intro	Did you notice the callout box? What did you think it was saying in your own words? How helpful was this?
Q34	How easy or difficult was it for you to answer this question?
	Who were you thinking about when you answered this question?
	What types of child care did you think were covered in this question?
Q36a	How did you arrive at your answer for this question?
Q36b	What did you think of the definition that was provided? How certain are you of your answer?
Q36c	How did you come up with your answer to this question? Was it difficult to narrow it down to one response? How would you determine when to use the option for “its own building or storefront”?
Q39-40	How did you come up with your answers to these two questions?
Q41	What do you think “program” means in this question?
Q43	What was this question asking, in your own words? What is meant by “provide”?
Q44	What was this question asking? What is meant by charge or fee for care?
Q45	In your own words, what is this question asking?
	(If respondent marked “yes” to any in Q45) What were you thinking about when you answered “yes” to this?
Q46	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q34-49	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Did you find the arrows and the instructions easy or difficult to follow?
	Are there any other items in this section you would like to comment on?
SECTION 2
Q50-55g	Did you find any questions in this section to be difficult to answer?
Q50	Did you pay attention to the callout box? Was there anything confusing about the definition?
Q52	What did you think about the response choices offered for this question? Was anything missing?
Q53	What were you thinking about when you answered this question?
Q55	What did you think of these categories? Were there any other major reasons why you chose the child care arrangement you did?
Q50-55g	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
Diary of last week’s care	STOP: I’d like to ask you to set the questionnaire aside for a moment and do something a little different. I’m going to hand you this calendar for last week. Could you mark on this calendar where CHILD was this past week and where you were this past week? And please talk to me about your answers while you are marking the calendar. [If needed: For example, CHILD was with a grandparent from 6 to 8 in the morning and then at a daycare from 9 to 6 on Monday.] Possible Probes: Regular care arrangement? How much do these hours change from week to week? If we had talked about a different week this year, would there have been different care arrangements? Now let’s look at how you answered the survey questions related to CHILD’s care. Why or why not did you include x?
SECTION 3
Q57-61B	For this whole section, please think out loud as you are coming up with your answers to these questions. Tell me what is going through your mind as you determine how to answer them.
Q60a.	What does this question ask in your own words? Are you counting reading to the child? What kind of story are you thinking of?
	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 4
Q62-69	Did you find any questions in this section to be difficult to answer?
Q67	What is this question asking, in your own words?
Q62-69	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 5
Q70-78A	Did you find any questions in this section to be difficult to answer?
Q71A	Were there any items in this question that you did not understand or did not know how to answer?
	What did the term “condition” mean to you and how did you determine whether or not to classify something as a condition?
	In question 71A, I noticed you checked XXX. Are those the ones that apply to your child? Did you notice the instruction to Mark one box for each item?
Q72	Please tell me how you interpreted the skip pattern before Q72. Is there anything that could be changed to make this clearer?
	What was Q72 asking, in your own words? What does the term “services” mean to you in this question?
Q74	What was this question asking, in your own words?
Q75	What was this question asking, in your own words?
Q70-78A	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 6
Q79-86	Did you find any questions in this section to be difficult to answer?
SECTION 7
Q88	Who are you answering this question about? How did you determine who the “first” parent or guardian is?
	Please tell me what this question is asking in your own words.
Q104	Who were you thinking about when you answered Q104?
Q88-120B	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 8
Q121A	Please tell me what this question is asking, in your own words.
Q121A-138	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?
	There may be things that my colleagues behind the mirror would like me to ask you about. Would you mind waiting for a moment while I check in with them and come back?
	Thank you.

PFI Protocol

General Introduction

I would like to begin by thanking you for coming here today to help us out. We’re scheduled to be here for 90 minutes, and I won’t keep you any longer. The reason we’ve asked you to come is that the US Department of Education conducts a survey roughly every two years to describe children’s educational experiences. Today I’m going to ask you to fill out a questionnaire that is being developed for 2011 and to provide feedback so that we can help the Department of Education make sure the questionnaire will be understood by all different types of parents.

• Because it would be hard to keep track of everything you say today, we're going to tape-record this session.

• And I’ll also be taking notes, so you might notice me writing as we are talking.

• There’s a one-way mirror behind me, and there are a few of my coworkers behind it watching.

• I want you to know that your responses are voluntary and confidential.

In just a minute I'm going to hand you the questionnaire and I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions.

There will also be a few questions where I will ask you to think “out loud” as you answer them. For these few questions, it is important that you tell me what you are thinking and feeling while you are actually in the process of coming up with your answers.

Now because some people aren't used to reading, thinking and expressing their feelings aloud, I'd like to begin today with a short practice question. I’d like you to practice with this by reading the question I’m about to give you out loud and telling me your thoughts and feelings from the moment I hand you the question until you are finished explaining your answer.

QUESTION: How many times did you eat a meal out at a restaurant in the past week?

Thank you…

SCREENER LETTER and CONNECTION BETWEEN SURVEYS

When you came to here today, you were given this package in the waiting area and asked to complete the survey inside.

Could you tell me what you did with the letter when you opened the package?

What message did you take away from it?

Any other thoughts you had about the letter? Anything specific jump out at you about it?

And after you completed the first survey in the waiting area, did you have any impressions of it?

If you received that first survey at home, would you be expecting something else to come?

Thank you. I’m about to hand you the questionnaire. When I hand you the questionnaire, please read it and fill it out as if you were actually responding to the survey. As you go through the survey, if any question sticks out to you in any way (maybe you think a question is hard to answer, hard to understand, in a strange place, or you just don’t like it)—anything at all that you would like to say about an item, please tell me so that we can discuss it.

Remember that I’ll be stopping you here and there as you complete the survey so that we can discuss it.

(GIVE QUESTIONNAIRE TO RESPONDENT)

Here is the package you would receive in the mail at home. Go ahead and open this as you would if you were at home. Talk to me about it.

What parts of the package does the respondent look at?:

PAGES 1-3	Did respondent carefully read instructions?
	Did respondent have any concerns or questions about the instructions?
	What impressions did you get from the cover?

ENVELOPES
Would one of these mailings stand out more to you than the other? Why or why not?
	Would one of these envelopes draw your attention more than the other?
	Any other thoughts on receiving this first survey and then receiving this second one?

FAQ	Take a look at the Commonly Asked Questions on the back. Do you think these are useful? Anything you would add?

Please continue with the questionnaire.

Question-by-Question Follow-Ups

SECTION 1
Q1-6	Stop respondent after completing the first page to gather feedback.
	Which child are you answering these questions about? How easy or difficult is it to keep your focus on the “sampled child” as you are answering these questions?
Q4	What is this question asking, in your own words?
	What were you thinking about when you answered this question?
Q5	What do you think of when you hear the words “charter school?” What does it mean to you? Do you know any examples of charter schools? How confident are you that CHILD’s school is/is not a charter school?
Q1-6	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS PAGE)
	Are there any other items on this page you would like to comment on?
Q7	Why did you answer yes/no for this question?
	How easy or difficult is it for you to recall whether you considered other schools?
Q9 IF NOT SKIPPED.	What was this question asking, in your own words?
	What were you thinking about as you answered this question?
Q11	(Monitor to make sure respondent enters numbers instead of characters)
Q12	Were you satisfied with these response options?
Q14	How did you answer these questions? What were you thinking about for a? b? c? d? What kinds of contact? Email? Phone? What kinds of contact do you get from the school? Are there other kinds of contact that you have gotten from the school or teachers that didn’t fit into these categories? What about specifically for CHILD?
Q14A	How did you come up with your answer for this one?
Q7-19C	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Did you find the arrows and the instructions easy or difficult to follow?
	Are there any other items in this section you would like to comment on?
SECTION 2
Q20	STOP: Please think out loud as you are coming up with your answer to this question. Tell me what is going through your mind as you determine how to answer it.
Q20-25	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 3
Q26-31	For this whole section, please think out loud as you are coming up with your answers to these questions. Tell me what is going through your mind as you determine how to answer them.
	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 4
Q32-34	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 5
Q35-43A	Did you find any questions in this section to be difficult to answer?
Q36A	Were there any items in this question that you did not understand or did not know how to answer?
	What did the term “condition” mean to you and how did you determine whether or not to classify something as a condition?
	In question 36A, I noticed you checked XXX. Are those the ones that apply to your child? Did you notice the instruction to Mark yes or no for each item?
Q37	Please tell me how you interpreted the skip pattern before Q37. Is there anything that could be changed to make this clearer?
	What was Q37 asking, in your own words? What does the term “services” mean to you in this question?
Q35-43A	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 6
Q44-51	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Did you find any questions in this section to be difficult to answer?
SECTION 7
Q88	Who are you answering this question about? How did you determine who the “first” parent or guardian is?
	Please tell me what this question is asking in your own words.
Q104	Who were you thinking about when you answered Q104?
Q88-120B	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
	Are there any other items in this section you would like to comment on?
SECTION 8
Q121A	Please tell me what this question is asking, in your own words.
Q121A-138	(GO OVER ANY OBSERVATIONS YOU HAD ON ANY OTHER ITEMS IN THIS SECTION)
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?
	There may be things that my colleagues behind the mirror would like me to ask you about. Would you mind waiting for a moment while I check in with them and come back?
	Thank you…