National Household Education Survey (NHES) 2011 Field Test Respondent Debriefing Interviews

NHES:2011 Field Test

Respondent Debriefing: Respondents to Mail Screener and Mail Topical

March 7, 2011

Interview # ________

Screener Version: SCREENOUT ENGAGING DUAL BILINGUAL

Prenotice Letter: YES NO

Screener Incentive: $2 $5

Screener Response Stage: INITIAL 1^ST FOLLOW-UP 2^ND FOLLOW-UP

Topical Version: PFIALT ECPPALT PFIMAIN ECPPMAIN

Topical Response Stage: INITIAL 1^ST FOLLOW-UP 2^ND FOLLOW-UP

Topical Package: STANDARD ENV ALTERNATE ENV

Topical Delivery: 1^ST CLASS PRIORITY

INTRODUCTION

Hello, I’m (name) calling from Westat. I’m calling to speak with the person who recently completed a survey by mail for us. It was about education. Would that be you?

When would be a convenient time to contact {NAME/the person who completed the survey}?

[IF DO NOT HAVE NAME: Who should I ask for when I call back?__________________]

WHEN THE RESPONDENT COMES TO THE PHONE

Hello, I understand that you recently completed an education survey. You gave us a phone number on the survey in case we had any more questions. We actually do have a few more questions. Could we talk to you about these?

IF YES

We are calling to try to find out some important information about what people who completed the survey thought about it. This information will help us improve the survey. This is a research project and your participation is voluntary. You can stop at any time and you can skip any question you wish. We expect this to take less than 20 minutes.

All information you provide may only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). I would also appreciate your permission to audio record this conversation. The recording will be for note-taking purposes only. This allows me to listen to what you say and not have to worry about trying to write down what you are saying at the same time. When the recorder starts, I’ll ask again for your permission.

[IF RESPONDENT AGREES, START RECORDER] I have started the recorder. Do I have your permission to record our discussion?

► START OF DEBRIEFING

Think back to when you first received a survey packet in the mail. When it came in the mail, what did you think?

IN THIS SECTION PROBE RESPONDENT COMMENTS TO ASCERTAIN WHETHER HE/SHE IS RECALLING THE SCREENER OR TOPICAL INTERVIEW. ASK RESPONDENT TO DESCRIBE THE PACKAGE.

SCREENER EXPERIENCE

I’d like you to think about the first survey you filled out…

General Probes:

Can you tell me about what you first received in the mail?

IF RESPONDENT SPECIFICALLY MENTIONS A LETTER AND WAS ASSIGNED TO RECEIVE A PRENOTIFICATION LETTER:

What were your thoughts about the letter?

IF RESPONDENT DISCUSSES SCREENER MATERIALS:

What do you remember being in the envelope?

Survey Components Recalled:

• Prenotice letter

• Cover Letter

• Survey

• Incentive

• Magnet

• Return envelope

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Do you remember who the packet was from/sponsor?

Among the different things that came in the mail package, what was most important to you? (Why is that?)

Component Specific Probes:

Letter

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Could you tell me what you remember about the letter that came with the first survey?

Would you say you read the letter or did you skip it?

Can you tell me what you liked or disliked about the letter?

Incentive

POINTS TO COVER ABOUT THE INCENTIVE IF MENTIONED:

MAKE SURE TO VERIFY R IS TALKING ABOUT SCREENER INCENTIVE

[IF NEEDED] You said there was money, how much was that?

How did you feel about the ($X / money) that came with the survey?

Survey

Now I want to talk about the first questionnaire you completed.

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Thinking about the questionnaire you filled in, what do you remember most about it?

Was there anything you liked or did not like about it? (PROBE ON EACH)

POINTS TO COVER ON ITEM SENSITIVITY; ADDITIONAL PROBES IF NEEDED:

We asked some information about you and your child/children. The first question I would like to talk about is the one asking for your telephone number. Why did you think we were asking for this? [IF NEEDED] How did you feel about giving your telephone number?

We also asked for your child’s name. What did you think that information would be used for?

How did you feel about providing this information? SPECIFICALLY PROBE FOR WHAT IF ANYTHING SUPPORTED THE LEGITIMACY OF THE SURVEY. FOR EXAMPLE; SPONSOR, CONTENT, PERCEIVED RELEVANCE, TEXT IN LETTER]

Spanish Materials

IF ASSIGNED DUAL OR BILINGUAL SCREENER (AND NOT ALREADY ADDRESSED)

[IF NEEDED]: In the materials you received in your survey package, was everything in English?

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Was it easy to find what you needed in all the materials?

How did you feel about receiving materials in both English and Spanish?

Do you usually receive other materials in both English and Spanish?

Next Phase

[IF RESPONDED TO STAGE 2 (1^ST FU) OR STAGE 3 (2^ND FU) – IF NOT ALREADY DISCUSSED]

Before you mailed that (the first) survey to us, did you receive other mailings of the same survey?

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Could you tell me why you completed the (2^nd/3^rd) one we sent instead of the other(s)?

TOPICAL EXPERIENCE

We sent your household another questionnaire mailing a couple of weeks after you sent back the first survey form; that was a questionnaire about one of the children in your household. Can you tell me what you thought when that arrived?

General Probes

What do you remember most about that second survey packet?

Survey Components Recalled:

• Cover Letter

• Survey

• Incentive

• Return Envelope

Other Features of the Topical survey mentioned:

Did you receive the survey or did someone else in the household? [PROBE FOR WHETHER SCREENER AND TOPICAL RESPONDENT WERE THE SAME PERSON].

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

Did you remember the first survey when you got this one?

Did you think that this (second, longer) survey was connected to the first survey? How did you come to that conclusion?

Was there any discussion in your household about the second survey and its content?

Did you complete this on your own, or did you ask anyone else in the household some of the questions?

About how long do you think it took you to complete this survey (the second longer survey)?

Did you have any difficulty answering any of the items?

Component Specific Probes:

Sensitivity

This second survey was a little longer and asked more detailed questions about your child. Was there anything that made you hesitant about completing this survey?

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

How did you feel about seeing the child’s name on the survey?

[IF THE CHILD’S NAME WAS NOT REPORTED] How did you know which child to respond about?

Incentive

POINTS TO COVER ABOUT THE INCENTIVE IF MENTIONED:

MAKE SURE TO VERIFY R IS TALKING ABOUT TOPICAL INCENTIVE

[IF NEEDED] You said there was money, how much was that?

How did you feel about the ($X / money) that came with the survey?

Survey Package/Delivery

IF NOT ALREADY COVERED UNDER SCREENER – NEXT PHASE

Was there anything that made this survey package stand out or seem different? [PROBE TO SEE IF RESPONDENTS RECEIVING PRIORITY MAIL OR ALTERNATE ENVELOPE REACT TO THESE FEATURES.]

Survey

Could you tell me what you remember about the topics of the questions asked?

What were your general impressions of the survey? Did any questions stick out to you for any reason? Could you tell me more about that?

POINTS TO COVER; ADDITIONAL PROBES IF NEEDED:

IF ANYTHING IRRELEVANT: Why do you think we were asking about (ITEM)?

You felt (ITEM(s)/SECTION(s)) were not relevant to the survey, but you still completed them, could you tell me about that?

NEXT SECTION – SPECIFIC QUESTIONS

IF RESPONDENT COMPLETED ALTERNATE PFI QUESTIONNAIRE, CONTINUE WITH PFI SECTION.

IF RESPONDENT COMPLETE ALTERNATE ECPP QUESTIONNAIRE, CONTINUE WITH ECPP SECTION (1 QUESTION OF INTEREST)

REMEMBER TO CLOSE WITH PROBES ABOUT COMMON PARENT SECTION.

Question Specific Probes:

Next, I would like to ask you about some specific questions in the second (or longer) survey you completed.

PFI

QUESTION 11 = YES

Question 11 in the survey you completed asked: “Since the beginning of the school year, has this child been in the same school?” And you answered ‘yes’.

How many schools has this child been enrolled in since starting kindergarten?

Tell me how you came up with [X] schools?

Could you tell me about the reason your child changed schools?

[IF NEEDED] Did you count changes due to moving to the next grade? For example, from middle school to high school?

QUESTION 11 = NO

How many schools has this child been enrolled in since starting kindergarten?

Have you lived at the same address since this child started kindergarten? (If no, ask: did the child change schools at that time?)

[IF NEEDED] Did you count changes due to moving to the next grade? For example, from middle school to high school?

Were there any times the child was temporarily assigned or attended another school?

QUESTIONS 4, 5 AND 7 WHERE EITHER 4 = NO OR 5 = YES, AND 7 = NO

Questions 4 through 7 ask about the type of school your child goes to and the options you have in choosing schools for your child. Let me re-read a few of these questions to you. I want you to try to recall and walk me through how you answered these questions.

After we ask what type of school your child attends we asked: “It is his/her regularly assigned school?” And you said “No”. [THIS MAY BE YES, BUT 5 MUST BE YES]

We then asked “Is this school a charter school?” And you said “Yes”. [THIS MAY BE NO, BUT 4 MUST BE NO].

Finally at question 7 we asked: “Does your public school district let you choose which public school you want this child to attend?” And you said “No”.

PROBES

Walk me through how you arrived at your answers to these questions.

Did you have any difficulty when answering this question?

How did you decide which school the child would attend?

QUESTION 7 SPECIFIC PROBE: in your own words, can you tell me what this question is asking? How does a school district let you ‘choose’ the school?

Was there anything about these questions that was confusing or that you found difficult to answer? [IF NEEDED] For example, do you remember re-reading these questions or skipping these and coming back to them?

SPECIFIC PROBES

Tell me a bit more about the school your child goes to.

[IF NEEDED] What grades does it have?

Does it have a special teaching focus or magnet program? (If so, please ask to describe).

FOR RESPONDENTS WHO ANSWERED NO OR DON’T KNOW TO ITEM 7:

You answered (no/don’t know), can you tell me how you came up with that answer?

QUESTION 15 (A – D) ANY A – D > 0

Question 15 asked: “Since the beginning of this school year, how many times have any of this child's teachers or school staff contacted your household about…

a. Behavior Problems this child is having in school.

b. Problems this child is having with school work.

c. Very good behavior.

d. Very good school work.

PROBES

Thinking of item [X], can you tell me about why you reported [X]? (For example, did the respondent recall specific events or estimate number of contacts?)

Can you give me some examples of what types of contacts you included?

Were there any contacts you did not include?

QUESTION 25 (A – I) ANY A – I > 0

Question 25 asked: Since the beginning of this school year, how many times has any adult in this child's household done any of the following things at this child's school?

a. Attended a school or class event, such as a play, dance, sports event, or science fair.

b. Served as a volunteer in this child's classroom or elsewhere in the school.

c. Attended a general school meeting, for example, an open house, or a back-to-school night.

d. Attended a meeting of the parent-teacher organization or association.

e. Gone to a regularly scheduled parent-teacher conference with this child's teacher.

f. Participated in fundraising for the school.

g. Served on a school committee.

h. Met with a guidance counselor in person.

i. Other (Something else).

PROBES

Thinking of item [X], can you tell me about why you reported [X]? (For example, did the respondent recall specific events or estimate number of events?)

Can you give me some examples of what types of events/activities you included?

Were there any events or activities you did not include? [PROBE FOR WHAT ‘OTHER’ ACTIVITIES MAY BE REPORTED.]

QUESTION 22 & 23 (ALL RESPONDENTS)

Questions 22 and 23 asked about internet access. I would like you think about the first question: “Some students take school-related courses over the Internet. Is this child receiving any instruction this way?”

PROBES

Thinking about this question, can you give me some examples of what comes to mind?

FOR ALL RESPONDENTS WHERE Q22 = YES

Can you tell me about the instruction your child receives over the Internet?

[IF NEEDED] Was this instruction (over the Internet) through a school or school district?

Who provided this instruction?

Why did the child receive this instruction? (PROBE FOR SCHOOL RELATED, HOBBY, OR OTHER REASONS.)

QUESTION 16 (ALL RESPONDENTS)

Questions 16 asked about school absences: “Since the beginning of the school year, how many days has this child been absent from school?”

PROBES

Thinking of this question, can you tell me about why you reported [X] absences? (For example, did the respondent recall specific events or estimate number of events?)

Did you include any days when the child left school early?

Did you include any school related closings (e.g., snow days)?

Now, I like to ask you about some specific questions in the second (or longer) survey you completed?

ECPP

QUESTION 71, 72, AND 73 = (QUESTION 71 = YES)

PROBES

I would like you to think about the following questions and tell me what comes to mind when you think about each:

Question 71: Does this child ever read or pretend to read storybooks on his/her own?

Question 72: Does this child actually read the words written in the book, or does he/she look at the book and pretend to read?

[IF NEEDED] You reported (reads/pretends to read), tell me about what this child does when (reading/pretending to read)?

Question 73: When this child pretends to read a book, does it sound like a connected story, or does he/she tell what is in each picture without much connection between them?

COMMON SECTIONS (PARENT)

PFI & ECPP

PROBES

In the questionnaire you completed, you were asked about your child’s parents or guardians living within the household. Can you tell me what you remember about these questions?

[INTERVIEWER: REFERENCE QUESTIONS UNDER PARENT 1 AND PARENT 2]

Was there anything that you found particularly easy or particularly difficult to answer?

Do you recall anything that you found confusing?

In this section, we asked first about parent 1 and then about parent 2. You used the child’s (relationship) for parent 1, can you tell me why you selected that person for parent 1.

CLOSING

Are there any additional comments you would like to make about your experience completing the survey?

Thank you very much for your contribution to this important study. Those are all the questions that I have for you.

Goodbye.

30