2016 National Household Education Surveys Program (NHES) Usability Testing

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] and highlighting represents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us out with a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. This survey asks about a range of education or work training experiences for children and adults of all ages.

Today I’m interested in getting your opinions about the website that we will use to conduct the survey. I will ask you to go through the survey and use the features as if you were taking the real survey, but I will also stop you once in a while to ask you questions or ask you to complete a specific task that you may or may not usually do. When I give you a task, I might also ask you to explain how you are going about the task. This is not an evaluation of how familiar you are with the Internet or computers. There are no right or wrong answers to the survey questions either. This testing will help us make the website better and clearer for other respondents just like you.

To make sure we learn everything we can from this session, I am going to be taking notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today {by video}].

Also, this interview will be audio recorded so researchers can review the recording later. Any information that refers to you personally, like your name or your child’s name or school, will not be included in our report, and your responses will be used for research purposes only.

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know. Would you like to be part of this study?

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on respondent’s preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent.]

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

While you are filling out the survey, there may be times when I want you to talk about what you are thinking when completing a task. Hearing you talk about how you figure out your answers to my questions will help me understand how to make the survey better. I am more interested in how you complete the tasks than in the answers you give on the survey. To help you get started on how to do this we are going to do a practice activity. I’m going to give you a task related to your [device/computer/phone/tablet]—not the survey yet-and ask you to speak out loud about what is on your mind while you complete it.

Use your internet browser to find the address for the Department of Education.

Please tell me what you are thinking while you are figuring out how to find the address.

[Probe as needed for detail: Where do you plan to click first? Can you tell me more about why you clicked [xxx]?]

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

Please keep talking.
Tell me what you are thinking.
What are you planning to do next?
Is there anything else you can do?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Can you tell me if there is anything confusing on the page?
[If you pick up on a visual cue that indicates confusion like going to previous pages or rereading instructions]: “Tell me what just happened. Tell me about what you were just doing.” [In response to R actions or comments]: What does the feature mean to you?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

Observations to keep in mind:

Does user try to use the enter button to move to the next page?
Does user try to use the browser back button to move backward in the survey, instead of the “Previous” link?
How does the user respond to error messages?
How does the user feel about the overall look of the site?
Does the user have a hard time finding the “Next” button or “Previous” link?
Does the user go back and forth between pages (follow up on why if not obvious)?
Does the user try to go back multiple pages, only to get stuck? How do they react?
Does the user skip any questions? Accidentally or on purpose?
Note if any difficulty is content or usability related (when expressed verbally)
Note if any difficulty with skips - are there places where user receives questions that are not applicable to them? When this happens, can they find their way out?
[If user is using mobile device or tablet: can they see what they need to see? Can they navigate? How challenging is it?
Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

Remember to think aloud and answer the questions as accurately as possible based on your situation. I am interested in hearing you talk about what you are thinking as you figure out your answers to the questions.

[Hand respondent invitation letter (which includes the User ID)]

Now it is time to complete the survey online. Please use this letter to sign in. Remember that I am very interested in hearing you talk about what you are thinking about as you figure out how to do the task.

Task 1: Log into the survey

OBSERVE if respondent has any difficulty using the invitation letter or finding their username

TASK 2:

[After logging on] BEFORE respondent begins the survey, ask these questions:

If you had a question about the survey, what would you do? (OBSERVE: Does respondent have any difficulties finding the FAQ section?)

If you can’t find the answer to your question, what would you do?

If respondent is bilingual in Spanish and English:

If you wanted to see an item or page in Spanish [or English], what would you do? [After they toggle to Spanish ask all bilingual respondents in round 2 to continue in Spanish]

INTERVIEWER: Thank you! Please continue.

Screener:

1. How many people live in this household? (Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home.)

_______

ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error

OBSERVE if the respondent has any difficulty with “writing in” the answer

Please answer the next few questions for each person living in this household. Include all adults and children, starting with yourself.

2. So that we can ask questions about specific people, we will ask you for the first name, initials, or nickname of each person in the household. What is your first name, initials, or nickname?

_________________________

OBSERVE if the respondent is uncomfortable with any of the questions in the screener and if it is related to not wanting to enter personal information online.

3. What is your month and year of birth?

Month _________ Year of birth _________ [drop down]

OBSERVE if the respondent has any difficulty with the drop down menu

4. What is your sex?

Male
Female

OBSERVE if the respondent has any difficulty with the radio buttons

Task 3. Hypothetical Scenario – LOG OUT, LOG IN

[Task 3 will occur for all respondents in Round 1, all bilingual respondents in Round 2, and removed in Round 3 as long as issues do not seem to remain]

INTERVIEWER: Stop respondent prior to question 5.

Let’s pretend that you didn’t have time to finish the survey right now and wanted to exit the survey and return to finish it later. What would you do?

Observe how the respondent exits the survey.

Does user have any trouble finding the “Save & Continue Later” button?

If…

Respondent finds the “Save & Continue Later” button: How easy or difficult was it to find the log out link?

Respondent closes the browser without using the “Save & Continue Later” button: Did you see that there was a logout button on the screen?

Why did you choose to close the browser instead of using the logout button?

Interviewer: Now, please go back to finish the screener

OBSERVE how the respondent goes back to finish the survey.

If respondent did not write down or memorize PIN:

I see that you don’t have your PIN. Do you think you will still be able to complete the survey?

If respondent says “Yes”:

What do you think you will have to do to complete the survey?

Would you attempt to contact someone to try to get your PIN?

Was it clear that you would not be able to return to the survey if you don’t have your PIN?

[Give respondent their PIN to reenter if they don’t have it]

Interviewer: Please continue answering the questions.

5. Are you currently in…

Homeschool instead of attending a public or private school for some or all classes,
Public or private school, or preschool,
College, university or vocational school, or (IF Question 1 = 1, END SURVEY; = 2+, SKIP TO QUESTION 7)
Not in school? (IF Question 1 = 1, END SURVEY; = 2+, SKIP TO QUESTION 7)

6. What is your current grade or equivalent?

Preschool (END SURVEY)
Kindergarten
First
Second
Third
Fourth
Fifth
Sixth
Seventh
Eighth
Ninth
Tenth
Eleventh
Twelfth
College, university or vocational school
None of these

7. What is the first name, initials, or nickname of the next person in this household?

_________________________

OBSERVE if the respondent has trouble remembering who or how many people they entered about previously.

8. What is [P2NAME]'s month and year of birth?

9. What is [P2NAME]’s sex?

10. Is [P2NAME] currently in…

Homeschool instead of attending a public or private school for some or all classes,
Public or private school, or preschool,
College, university or vocational school, or [IF Question 1 = 2, END SURVEY; = 3+, SKIP TO QUESTION 7]
Not in school? [IF Question 1 = 2, END SURVEY; = 3+, SKIP TO QUESTION 7]

11. What is [P2NAME]'s current grade or equivalent?

Preschool (END SURVEY)
Kindergarten
First
Second
Third
Fourth
Fifth
Sixth
Seventh
Eighth
Ninth
Tenth
Eleventh
Twelfth
College, university or vocational school
None of these

[Questions 7-11 REPEAT for as many people (up to 10) are in the household ]

[AT THIS POINT THE RESPONDENT WILL BE AUTOMATICALLY ROUTED TO ADDITIONAL SAMPLING QUESTIONS SPECIFIC TO THE DIFFERENT TOPICAL SURVEY DEPENDING UPON THE SAMPLING EQUATION. PLEASE PAY SPECIAL ATTENTION TO THE NEXT QUESTION.]

PFI (Enrolled and Homeschool) Sampling Questions

We would like to ask some additional questions about (SAMPLED CHILD)'s education.

12. Are you a parent or guardian in this household who knows about (SAMPLED CHILD)'s education?

No
Yes (SKIP TO QUESTION 14)

If respondent did not provide child’s name, OBSERVE how respondent handles the automatically filled in information “(SAMPLE CHILD)’s”

13. How are you related to [this child]?

Mother (birth, adoptive, step, or foster) (SKIP TO QUESTION 15)
Father (birth, adoptive, step, or foster) (SKIP TO QUESTION 15)
Aunt (SKIP TO QUESTION 15)
Uncle (SKIP TO QUESTION 15)
Grandmother (SKIP TO QUESTION 15)
Grandfather (SKIP TO QUESTION 15)
Parent’s girlfriend/boyfriend/partner (SKIP TO QUESTION 15)
Other relationship – Specify: _______________ (SKIP TO QUESTION 15)

If respondent selects “Other relationship”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other relationship” option, please note any reaction to the greyed out box.

14. Before we take you to the questions about (SAMPLED CHILD)'s education, would you please give us your email address in case we need to contact you further?

___________________ ________________________ (SKIPS TO PFI-EN or PFI-HS SURVEY)

This question has 2 places for responses. The first box is the email and the 2^nd box is the email confirmation. ERROR will appear if respondent doesn’t enter a valid email address or if the addresses don’t match: OBSERVE how respondent reacts to error messages or entering their email address twice.

15. Who is the person in this household who knows the most about (SAMPLED CHILD)'s education?

[LIST OF NAMES FROM QUESTIONS 1 through 11 will appear]

OBSERVE if respondent has any difficulty with automatically tabulated list

16. Would you please give us [NAME]'s email address/the email address of a person in the household knowledgeable about [sampled child]’s education] so that we may contact (him/her) to do a brief interview about (SAMPLED CHILD)'s education?

____________________ _______________________ (END SURVEY)

OBSERVE if the respondent is uncomfortable with giving out someone else’s email

This question has 2 places for responses. The first box is the email and the 2^nd box is the email confirmation. ERROR will appear if respondent doesn’t enter a valid email address or if the addresses don’t match: OBSERVE how respondent reacts to error messages or entering their email address twice.

ECPP Sampling Questions

17. Are you a parent or guardian in this household who knows about (SAMPLED CHILD)'s care and education?

No
Yes (SKIP TO QUESTION 19)

18. How are you related to [this child]?

Mother (birth, adoptive, step, or foster) (SKIP TO QUESTION 20)
Father (birth, adoptive, step, or foster) (SKIP TO QUESTION 20)
Aunt (SKIP TO QUESTION 20)
Uncle (SKIP TO QUESTION 20)
Grandmother (SKIP TO QUESTION 20)
Grandfather (SKIP TO QUESTION 20)
Parent’s girlfriend/boyfriend/partner (SKIP TO QUESTION 20)
Other relationship – Specify: _______________ (SKIP TO QUESTION 20)

19. Before we take you to the questions about (SAMPLED CHILD)'s education, would you please give us your email address in case we need to contact you further?

___________________ ________________________ (GO TO ECPP SURVEY)

20. Who is the person in this household who knows the most about (SAMPLED CHILD)'s education?

[LIST OF NAMES FROM QUESTIONS 1 through 11 will appear]

OBSERVE if respondent has any difficulty with automatically tabulated list

21. Would you please give us [NAME]'s email address/the email address of a person in the household knowledgeable about [sampled child]’s education] so that we may contact (him/her) to do a brief interview about (SAMPLED CHILD)'s education?

__________________ _________________________ (END SURVEY)

OBSERVE if the respondent is uncomfortable with giving out someone else’s email

ATES Sampling Questions

22. We would like to ask some additional questions about (SAMPLED PERSON)'s education and job training. Are you (SAMPLED PERSON)?

No (SKIP TO QUESTION 24)
Yes

23. Before we take you to the questions about education and job training, would you please give us your email address in case we need to contact you further?

_____________________ ______________________ (GO TO ATES SURVEY)

24. Would you please give us (SAMPLED PERSON)'s email address so that we may contact (him/her) to do a brief interview about (his/her) education and job training?

___________________________________________ (END SURVEY)

OBSERVE if the respondent is uncomfortable with giving out someone else’s email

This question will occur twice, with a confirmation email entry. ERROR will appear if respondent doesn’t enter a valid email address or if the addresses don’t match: OBSERVE how respondent reacts to error messages or entering their email address twice.

Thank you/not eligible/not sampled page

OBSERVE the respondents reaction to not being eligible or not sampled.

STEP 4: DEBRIEFING

After respondent has completed the SCREENER, STOP and ASK these debriefing questions:

What were your first impressions of the survey when you began it and got into the first several questions?

Do you think there are questions some people would not want to answer online? Which question(s)? Why?

We would like you to fill out a short survey about your experiences with this web survey you just completed. There are no right or wrong answers. Your responses are important to us and will only be used for our information. We may ask you some follow-up questions according to your responses so that we can get your feedback on how to improve our web survey.

[Give respondent the QUIS survey]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ________________ as a _______?

Ask the following questions:

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Do you have anything else you would like to mention about the survey that we haven’t had a chance to talk about?

If survey was conducted on a mobile device (tablet or cell):

Overall, what was your impression of conducting the survey on a mobile device?

STEP 5: THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this website better.

[Interviewer Instructions: Collect materials and escort respondent out of the interviewing room and to the lobby or street. Please give the respondent incentive envelope [$40 incentive], and thank them for their time and effort.]

Testing Protocol: Usability Evaluation of NHES ECPP

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us out with a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. This survey asks about children’s early learning, the programs in which children participate, and how families prepare children for school.

Also, this interview will be audio recorded so researchers can review the recording later. Any information that refers to you personally, like your name, will not be included in our report, and your responses will be used for research purposes only.

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know. Would you like to be part of this study?

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on respondent preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent.]

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

Use your internet browser to find the address for the Department of Education

Please tell me what you are thinking while you are figuring out how to find the address.

[Probe as needed for detail: Where do you plan to click first? Can you tell me more about why you clicked [xxx]?]

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

Please keep talking.
Tell me what you are thinking.
What are you planning to do next?
Is there anything else you can do?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Can you tell me if there is anything confusing on the page?
[If you pick up on a visual cue that indicates confusion like going to previous pages or rereading instructions]: “Tell me what just happened. Tell me about what you were just doing.” [In response to R actions or comments]: What does the feature mean to you?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

Observations to keep in mind:

Does user try to use the enter button to move to the next page?
Does user try to use the browser back button to move backward in the survey, instead of the “Previous” link?
How does the user respond to error messages?
How does the user feel about the overall look of the site?
Does the user have a hard time finding the “Next” button or “Previous” link?
Does the user go back and forth between pages (follow up on why, if not obvious)?
Does the user try to go back multiple pages, only to get stuck? How do they react?
Does the user skip any questions? Accidentally or on purpose?
Note if any difficulty is content or usability related (when expressed verbally)
Note if any difficulty with skips - are there places where user receives questions that are not applicable to them? When this happens, can they find their way out?
[If user is using mobile device or tablet: can they see what they need to see? Can they navigate? How challenging is it?
Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

[Hand respondent invitation letter (which includes the User ID)]

Task 1: Log into the survey

OBSERVE if respondent has any difficulty using the invitation letter or finding their username

[INSTRUCTIONS: Not all Respondents will proceed through the NHES SCREENER:

If the screener has already been completed (by AIR staff), they will automatically SKIP to the ECPP survey (TASK 2).
If the screener has not been completed, respondent will automatically be directed to the SCREENER (below).]

Screener:

1. How many people live in this household?

OBSERVE if the respondent has any difficulty with “writing in” the answer

2. What is your first name, initials, or nickname?

3. What is your month and year of birth?

OBSERVE if the respondent has any difficulty with the drop down menu

4. What is your sex?

OBSERVE if the respondent has any difficulty with the radio buttons

5. Are you currently in…?

6. What is your current grade or equivalent?

[Screener will repeat QUESTIONS 2-6 for as many household members as entered in QUESTION 1 then will automatically go to the additional ECPP screener questions]

7. Are you a parent or guardian in this household who knows about (SAMPLED CHILD)'s education?

If respondent did not provide child’s name, OBSERVE how respondent handles the automatically filled in information “(SAMPLED CHILD)’s”

8. How are you related to [this child]?

9. Before we take you to the questions about (SAMPLED CHILD)'s education, would you please give us your email address in case we need to contact you further?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

10. Who is the person in this household who knows the most about (SAMPLED CHILD)'s care and education?

OBSERVE if respondent has any difficulty with automatically tabulated list

11) Would you please give us [NAME]'s email address/the email address of a person in the household knowledgeable about [sampled child]’s care and education] so that we may contact (him/her) to do a brief interview about (SAMPLED CHILD)'s education?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

[OBSERVE if the respondent has trouble remembering who or how many people they entered previously.

After respondent has completed the screener, STOP and ASK these debriefing questions:

What were your first impressions of the survey when you began it and got into the first several questions?

Do you think there are questions some people would not want to answer online? Which question(s)? Why?

ECPP Survey

TASK 2:

INTERVIEWER: [After logging on] before respondent begins the survey, ask these questions:

If you had a question about the survey, what would you do? (OBSERVE: Does respondent have any difficulties finding the FAQ section?)

If you can’t find the answer to your question, what would you do?

If respondent is bilingual in Spanish and English:

If you wanted to see an item or page in Spanish [or English], what would you do? [After they toggle to Spanish ask all bilingual respondents in round 2 to continue in Spanish]

INTERVIEWER: Thank you! Please continue.

Childhood Care and Programs

Care Your Child Receives from Relatives

1. Is [this child] now receiving care from a relative other than a parent or guardian on a regular basis, for example, from grandparents, brothers or sisters, or any other relatives?

No (Go to Question 17)
Yes

If respondent did not provide child’s name, OBSERVE how respondent handles the automatically filled in information

OBSERVE if the respondent has any difficulty with the radio buttons

2. Are any of these care arrangements regularly scheduled at least once a week?

No(Go to Question 17)
Yes

3. These next questions are about the care that [this child] receives from the relative who provides the most care.

How is that relative related to this child?

4. How old is the [RCTYPE_ALT/relative] who provides the most care to [this child]?

___________

[ERROR will occur if respondent enters anything but a whole number or a number less than 12: OBSERVE how respondent handles error]

OBSERVE if the respondent has any difficulty with “writing in” the answer

5. Is this care provided in your home or another home?

6. How many days each week does [this child] receive care from this [RCTYPE_ALT/relative]?

____ days each week

[ERROR will occur if respondent enters anything but a whole number or a number out of range: OBSERVE how respondent handles error]

7. How many hours each week does [this child] receive care from this [RCTYPE_ALT/relative]?

_____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] NOTE/PROBE to see if the respondent is answering hours per week or hours per day

8. How old was [this child] in years and months when this particular regular care arrangement with [RCTYPE_ALT/relative] began?

_____ years ____months

[ERROR will occur if respondent enters age older than the one reported in the screener, anything but a whole number or a number out of range: OBSERVE how respondent handles error]

9. What language does this [RCTYPE_ALT/relative] speak most when caring for [this child]?

10. Will this [RCTYPE_ALT/relative] care for [this child] when [he/she] is… (Select one response for EACH ITEM below.)

a. Sick but does not have a fever?

b. Sick and has a fever

OBSERVE if the respondents clicks a response for each item or if they only select the ones that apply

11. Is there any charge or fee for the care [this child] receives from this [RCTYPE_ALT/relative], paid either by you or some other person or agency?

No (Go to question 15)
Yes

12. Do any of the following people, programs, or organizations help pay for this [RCTYPE_ALT/relative] to care for [this child]? (Select one response for EACH ITEM below)

a. A relative of [this child] outside your household who provides money specifically for that care, not including general child support

b. Temporary Assistance for Needy Families, or TANF

c. Another social service, welfare, or child care agency

d. An employer, not including a tax-free spending account for child care

e. Someone else

13. How much does your household pay for this [RCTYPE_ALT/relative] to care for [this child], not counting any money that may be received from others to help pay for care?

Enter ’0’ if your household does not pay this relative for care.

_____

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or does not enter a value.

Is that amount per…

Hour
Day
Week
Month
Year
Every 2 weeks
Other — Specify:-------------------------------------

If respondent selects “Other”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other” option, please note any reaction to the greyed out box.

14. How many children from your household is this amount for, including [this child]?

15. Does [this child] have any other care arrangements with a relative on a regular basis?

No (Go to question 17)
Yes

16. How many total hours each week does [this child] spend in those other care arrangements with relatives?

____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] [NOTE/PROBE hours per week versus hours per day]

Care Your Child Receives from Non-relatives

The next questions ask about any care [this child] receives from someone not related to [him/her], either in your home or someone else’s home. This includes home child care providers or neighbors, but not day care centers or preschools.

17. Is [this child] now receiving care in your home or another home on a regular basis from someone who is not related to [him/her]?

No (Go to Question 35)
Yes

18. Are any of these care arrangements regularly scheduled at least once a week?

No (Go to Question 35)
Yes

19. These next questions are about the care that [this child] receives from someone who is not related to him/her who provides the most care.

Is this care provided in your own home or in another home?

20. Does this person who cares for [this child] live in your household?

21. How many days each week does [this child] receive care from this person?

____ days each week

[ERROR will occur if respondent enters anything but a whole number or a number out of range: OBSERVE how respondent handles error]

22. How many hours each week does [this child] receive care from this person?

_____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] [NOTE/PROBE hours per week versus hours per day]

23. How old was [this child] in years and months when this particular regular care arrangement with this person began?

____ years ____ months

[ERROR will occur if respondent enters age older than the one reported in the screener, anything but a whole number or a number out of range: OBSERVE how respondent handles error]

24. Was this care provider someone you already knew?

25. Is [this child’s] care provider age 18 or older?

26. What language does this care provider speak most when caring for [this child]?

27. Will this care provider care for [this child] when [he/she] is… (Select one response for EACH ITEM below.)

a. Sick but does not have a fever?

b. Sick and has a fever

28. Would you recommend this care provider to another parent?

29. Is there any charge or fee for the care [this child] receives from this care provider, paid either by you or some other person or agency?

No (Go to question 33)
Yes

30. Do any of the following people, programs, or organizations help pay for this person to care for [this child]? (Select one response for EACH ITEM below)

a. A relative of [this child] outside your household who provides money specifically for that care, not including general child support

b. Temporary Assistance for Needy Families, or TANF

c. Another social service, welfare, or child care agency

d. An employer, not including a tax-free spending account for child care

e. Someone else

31. How much does your household pay for this person to care for [this child], not counting any money that may be received from others to help pay for care? (Enter ’0’ if your household does not pay this non-relative for care.)

____

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

Is that amount per…

Hour
Day
Week
Month
Year
Every 2 weeks
Other — Specify:

32. How many children from your household is this amount for, including [this child]?

33. Does [this child] have any other home-based care arrangements on a regular basis with someone who is not a relative? Do not include arrangements at day care centers or preschools.

No (Go to question 35)
Yes

34. How many total hours each week does [this child] spend in those other care arrangements with non-relatives?

____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] [NOTE/PROBE hours per week versus hours per day]

Day Care Centers and Preschool Programs Your Child Attends

The next questions ask about any day care centers and early childhood programs that [this child] attends. This does not include care provided in a private home.

35. Is [this child] now attending a day care center, preschool, or prekindergarten not in a private home?

No (Go to question 54)
Yes

36. Does [this child] go to a day care center, preschool, or prekindergarten, at least once each week?

No(Go to question 54)
Yes

37. The next questions ask about the program where [this child] spends the most time. Is [this child]’s current program a day care program, a preschool program, or a prekindergarten program?

38. Is this program a Head Start or Early Head Start program? (Head Start and Early Head Start are federally sponsored preschool programs primarily for children from low-income families.)

39. Where is this program located?

In a church, synagogue, or other place of worship
In a public elementary or secondary school
In a private elementary or secondary school
At a college or university
At a community center
At a public library
In its own building, office space, or storefront

Some other place— Specify:

If respondent selects “Some other place”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Some other place” option, please note any reaction to the greyed out box.

40. Is this program run by a church, synagogue, or other religious group?

41. Is this program located at your workplace or [this child]’s other parent’s workplace?

42. How many days each week does [this child] go to this program?

____ days each week

[ERROR will occur if respondent enters anything but a whole number or a number out of range: OBSERVE how respondent handles error]

43. How many hours each week does [this child] go to this program?

____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] [NOTE/PROBE hours per week versus hours per day]

44. How old was [this child] in years and months when [he/she] started going to this particular program?

____ years _____months

[ERROR will occur if respondent enters age older than the one reported in the screener, anything but a whole number or a number out of range: OBSERVE how respondent handles error]

45. What language does [this child]’s main care provider or teacher at this program speak most when caring for [him/her]?

46. Would you recommend this program to another parent?

47. Does this program provide any of the following services to [this child] or your family? (Select one response for EACH ITEM below.)

a. Hearing, speech, or vision testing

b. Physical examinations

c. Dental examinations

d. Formal testing for developmental or learning problems

e. Sick child care when this child is sick but does not have a fever

f. Sick child care when this child is sick and has a fever

48. Is there any charge or fee for this program, paid either by you or some other person or agency?

No (Go to question 52)
Yes

49. Do any of the following people, programs, or organizations help pay for [this child] to go to this program? (Select one response for EACH ITEM below.)

a. A relative of [this child] outside your household who provides money specifically for that care, not including general child support

b. Temporary Assistance for Needy Families, or TANF

c. Another social service, welfare, or child care agency

d. An employer, not including a tax-free spending account for child care

e. Someone else

50. How much does your household pay for [this child] to go to this program, not counting any money that you may receive from others to help pay for care? (Enter ’0’ if your household does not pay this non-relative for care.)

____

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or does not enter a value.

Is that amount per…

Hour
Day
Week
Month
Year
Every 2 weeks
Other — Specify:

51. How many children from your household is this amount for, including [this child]?

52. Does [this child] have any other care arrangements at a day care center or preschool on a regular basis?

No (Go to question 54)
Yes

53. How many total hours each week does [this child] spend at those day care centers or preschools?

_____ hours each week

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error] [NOTE/PROBE hours per week versus hours per day]

Task 3. Hypothetical Scenario – LOG OUT, LOG IN

[Task 3 will occur for all respondents in Round 1, all bilingual respondents in Round 2, and removed in Round 3 as long as issues do not seem to remain]

[INTERVIEWER: Stop respondent prior to question 54.]

Let’s pretend that you didn’t have time to finish the survey right now and wanted to exit the survey and return to finish it later. What would you do?

Observe how the respondent exits the survey.

Does user have any trouble finding the “Save & Continue Later” button?

If…

Respondent finds the “Save & Continue Later” button:

How easy or difficult was it to find the log out button?

Respondent closes the browser without using the “Save & Continue Later” button: Did you see that there was a logout button on the screen?

Why did you choose to close the browser instead of using the logout button?

Interviewer: Now, please go back to finish the survey

Observe how the respondent goes back to finish the survey.

If respondent did not write down or memorize PIN: I see that you don’t have your PIN. Do you think you will still be able to complete the survey?

If respondent says “Yes”: What do you think you will have to do to complete the survey?

Would you attempt to contact someone to try to get your PIN?

Was it clear that you would not be able to return to the survey if you don’t have your PIN?

[Give respondent their PIN to reenter if they don’t have it]

Interviewer: Please continue answering the questions.

Finding and Choosing Care for Your Child

54. Has [this child] ever attended a Head Start or Early Head Start program? (Head Start and Early Head Start are federally sponsored preschool programs primarily for children from low-income families.)

55. What is the main reason your household wanted a care program for [this child] in the past year?

56. Do you feel there are good choices for child care or early childhood programs where you live?

57. How much difficulty did you have finding the type of child care or early childhood program you wanted for [this child]?

Have not tried to find care (Go to question 60)
No difficulty (Go to question 59)
A little difficulty
Some difficulty
A lot of difficulty
Did not find the child care program you wanted

58. What was the primary reason for the difficulty finding child care or early childhood programs?

Cost
Location
Quality
Lack of open slots for new children
Needed a program for children with special needs
Other- Specify

59. How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time? (Select one response for EACH ITEM below.)

a. The location of the arrangement?

b. The cost of the arrangement?

c. The reliability of the arrangement?

d. The learning activities at the arrangement?

e. [The child] spending time with other kids [his/her] age?

f. The times during the day that this caregiver is able to provide care?

g. The number of other children in the child’s care group?"

h. Ratings on a website?

I. Recommendations from friends and family?

j. The religious orientation of the program?

Family Activities

The next questions ask about this child’s activities with family members in the past week or month.

60. About how many books does [this child] have of [his/her] own, including those shared with brothers or sisters?

____ number of books

[ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error]

61. How many times have you or someone in your family read to [this child] in the past week?

Not at all (Go to question 63)

____ times

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error]

62. About how many minutes on each of those times did you or someone in your family read to [this child]?

____minutes

[ERROR will occur if respondent enters anything but a whole number or a zero: OBSERVE how respondent handles error]

63. In the past week, how many times has anyone in your family done the following things with [this child]? (Select one response for EACH ITEM below.)

a. Told [this child] a story? (Do not include reading to [him/her].)"

b. Taught [this child] letters, words, or numbers?"

c. Sang songs with [this child]?

d. Worked on arts and crafts with [this child]?

64. In the past week, how many days has your family eaten the evening meal together?

Enter ‘0’ if none.

_____

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or does not enter a value.

65. In the past month, have you or someone in your family visited a library with [this child]?

66. In the past month, have you or someone in your family visited a bookstore with [this child]?

Things Your Child May be Learning

These next questions ask about things that different children do at different ages. These things may or may not be true for [this child].

67. Is [this child] under 2 years old or is [he/she] 2 years old or older?

Under 2 years (Go to Question 75)
2 years or older

68. Can [this child] identify the colors red, yellow, blue, and green by name?

69. Can [this child] recognize the letters of the alphabet?

70. How high can [this child] count?

71. Can [this child] write [his/her] first name, even if some of the letters are backwards?

72. Does [this child] ever read or pretend to read storybooks on [his/her] own?

No (Go to question 75)
Yes

73. Does [this child] actually read the words written in the book, or does [he/she] look at the book and pretend to read?

Pretends to read (Go to question 74)

Actually reads the written words(Go to question 75)
Does both(Go to question 75)

74. When [this child] pretends to read a book, does it sound like a connected story, or does [he/she] tell what is in each picture without much connection between them?

Child’s Health

75. In general, how would you describe [this child]’s health?

76. Has a health or education professional told you that [this child] has any of the following conditions? (Select one response for EACH ITEM below.)

a. An intellectual disability (mental retardation)

b. A speech or language impairment

c. A serious emotional disturbance

d. Deafness or another hearing impairment

e. Blindness or another visual impairment not corrected with glasses

f. An orthopedic impairment

g. Autism

h. Pervasive Developmental Disorder (PDD)

i. Attention Deficit Disorder, ADD or ADHD

j. A specific learning disability

k. A developmental delay

l. Traumatic brain injury

m. Another health impairment lasting 6 months or more

77. (If child is under 3 years old) Has a health, education, or early intervention professional told you this child is “at-risk” for a substantial developmental delay?

78. [Automatic Skip based on response to Q76. If no “Yes” marked questionnaire skips to Question 86. If any condition marked “Yes” Go to Q79]

79. Is [this child] receiving services for [his/her] condition?

No (Go to question 84)
Yes

80. Are these services provided by any of the following sources? (Select one response for EACH ITEM below.)

a. Your local school district

b. A state or local health or social service agency

c. A doctor, clinic, or other health care provider

d. This child's private school

81. Are any of these services provided through an Individualized Family Service Plan (IFSP), Individualized Education Program (IEP) or services plan?

No (Go to question 84)
Yes

82. Did any adult in your household work with the service provider or school to develop or change [this child]’s IFSP, IEP or services plan?

83. During this school year, how satisfied or dissatisfied have you been with the following aspects of [this child]’s IFSP, IEP or services plan? (Select one response for EACH ITEM below.)

a. The service provider’s or school’s communication with your family?

b. [The child]’s special needs teacher or therapist?

c. The service provider’s or school’s ability to accommodate [this child]’s special needs?

d. The service provider’s or school’s commitment to help [this child] learn?

84. Is [this child] currently enrolled in any special education classes or services?

85. Does [this child]’s condition interfere with [his/her] ability to do any of the following things? (Select one response for EACH ITEM below.)

a. Learn

b. Participate in play with other children

c. Go on outings

d. Make friends

Child’s Background

86. Based on your household's previous response, [this child] was born in [month] of [year]. Is that correct?

87. In what month and year was [this child] born?

Month ________ Year ________ [drop down]

OBSERVE if respondent has difficulty with the drop down menu

88. Where was [this child] born?

One of the 50 United States or the District of Columbia (Go to question 90)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

89. How old was [this child] when [he/she] first moved to the 50 United States or the

District of Columbia?

____ Age

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number greater than 18: OBSERVE how respondent handles error]

90. Is [this child] of Hispanic, Latino, or Spanish origin?

91. What is [this child]’s race? You may mark one or more races.

92. Based on your household's previous response, [this child] is [gender]. Is that correct?

No (Go to Question 93)
Yes

93. What is this child’s sex?

94. Does [this child] live at this address and another address (for example, because of a joint custody arrangement)? (Do not include vacation properties.)

No (Go to question 96)
Yes

95. Does [this child]…

96. What language does [this child] speak most at home?

Child has not started to speak (Go to question 98)
English(Go to question 98)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

97. Is [this child] currently enrolled in English as a second language, bilingual education, or an English immersion program?

Household Members

98. Based on your household's previous response, there are [total number of people in HH] people living in this household. Is that correct?

No (Go to question 99)
Yes

OBSERVE if the respondent has trouble remembering who or how many people they entered previously.

99. How many people live in this household? (Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home.)

____

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

100. How many of the following people live in this household with [this child]? (Example: Brother(s) 2)

This child’s	Enter ’0’ if none
Brother(s)
Sister(s)
Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent's girlfriend/boyfriend/partner
Other relative(s)
Other non-relative(s)

[ERROR will occur if respondent enters anything but a whole number for any of the options above. OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or does not enter a value.

101. How are you related to [this child]?

Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt
Uncle
Grandmother
Grandfather
Parent’s girlfriend/boyfriend/partner
Other relationship – Specify:

If respondent selects “Other relationship”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other relationship” option, please note any reaction to the greyed out box.

102. Which language(s) are spoken at home by the adults in this household? (Select all that apply.)

English
Spanish or Spanish Creole
French (including Patois, Creole, Cajun)
Chinese
Other languages—Specify:

If respondent selects “Other languages”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other languages” option, please note any reaction to the greyed out box.

OBSERVE if the respondent has any difficulty understanding “select all the apply”

Child’s Family

PARENT 1 LIVING IN HOUSEHOLD

Answer these questions about yourself if you are [the child]’s parent or guardian.

If you are not [the child]’s parent or guardian, answer these questions about one of [this child]’s parents or guardians living in the household.

103. [Are you] Is this parent or guardian the child’s…

104. [Are you] Is this person male or female?

105. What is [your] this person’s current marital status?

Now married (Go to question 107)
Widowed
Divorced
Separated
Never married

106. [Are you] Is this person currently living with a boyfriend/girlfriend or partner in this household?

107. What was the first language [you] this parent or guardian learned to speak?

English (Go to question 109)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

108. What language [do you] does this person speak most at home now?

109. Where [were you] was this parent or guardian born?

One of the 50 United States or the District of Columbia (Go to question 111)
One of the U.S. territories ((Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

110. How old [were you] was this person when [you] he or she first moved to the 50 United States or the District of Columbia?

____age

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

111. [Are you] Is this person of Hispanic, Latino, or Spanish origin?

112. What is [your] this person’s race? You may mark one or more races.

113. What is the highest grade or level of school that [you] this parent or guardian completed?

114. [Are you] Is he or she currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

115. Which of the following best describes [your] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (Go to question 117)
Full-time student(Go to question 118)
Stay at home parent(Go to question 118)
Retired(Go to question 118)
Disabled or unable to work(Go to question 118)

116. About how many hours per week [do you] does he or she usually work for pay or income, counting all jobs?

____hours (Go to question 118)

[ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error]

117. [Have you] Has this parent or guardian been actively looking for work in the past 4 weeks?

118. In the past 12 months, how many months (if any) [have you] has this person worked for pay or income?

____months

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error]

119. How old [are you] is this person?

____age

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error]

120. How old [were you] was this person when [you] he or she first became a parent to any child?

____age

Don’t know

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error]

PARENT 2 LIVING IN HOUSEHOLD

Answer these questions about a second parent or guardian living in the household.

121. Is there a second parent or guardian living in this household?

No (Go to 140)
Yes

122. Is this person the child’s…

123. Is this person male or female?

124. What is [his/her] this person’s current marital status?

Now married (Go to question 126)
Widowed
Divorced
Separated
Never married

125. Is [he/she] this person currently living with a boyfriend/girlfriend or partner in this household?

126. What was the first language [he/she] this parent or guardian learned to speak?

127. What language does [he/she] this person speak most at home now?

English (Go to question 128)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

128. Where was [he/she] this parent or guardian born?

One of the 50 United States or the District of Columbia (Go to question 130)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

129. How old was [he/she] this person when [he or she] first moved to the 50 United States or the District of Columbia?

____age

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

130. Is [he/she] this person of Hispanic, Latino, or Spanish origin?

131. What is [his/her] this person’s race? You may select one or more races.

132. What is the highest grade or level of school that [he/she] this parent or guardian completed?

133. Is [he or she] currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

134. Which of the following best describes [his/her] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (Go to question 136)
Full-time student(Go to question 137)
Stay at home parent(Go to question 137)
Retired(Go to question 137)
Disabled or unable to work(Go to question 137)

135. About how many hours per week does [he or she] usually work for pay or income, counting all jobs?

____ hours (Go to question 137)

[ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error]

136. Has [he/she] this parent or guardian been actively looking for work in the past 4 weeks?

137. In the past 12 months, how many months (if any) has [he/she] this person worked for pay or income?

____months

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error]

138. How old is [he/she] this person?

____age

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error]

139. How old was [he/she] this person when [he or she] first became a parent to any child?

____age

Don’t know

[ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error]

Your Household

140. In the past 12 months, did your family ever receive benefits from any of the following programs? (Select one response for EACH ITEM below.)

a. Temporary Assistance for Needy Families, or TANF

b. Your state welfare or family assistance program

c. Women, Infants, and Children, or WIC

d. Food Stamps

e. Medicaid

f. Child Health Insurance Program (CHIP)

g. Section 8 housing assistance

141. Which category best fits the total income of all persons in your household over the past 12 months? (Include your own income. Include money from jobs or other earnings, pensions, interest, rent, Social Security payments, and so on.)

142. How many years have you lived at this address?

Enter ’0’ if less than 1 year.

____years at this address

[ERROR will occur if respondent enters anything but a whole number. OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or does not enter a value.

143. Is this house…

144. Do you have Internet access on a cell phone?

145. Do you have Internet access at home on a computer or tablet?

146. How often do you use the Internet?

Thank you/submit page

STEP 4: DEBRIEFING

[Give respondent the QUIS survey]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ________________ as a _______?

Ask the following questions:

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Do you have anything else you would like to mention about the survey that we haven’t had a chance to talk about?

If survey was conducted on a mobile device (tablet or cell):

Overall, what was your impression of conducting the survey on a mobile device?

STEP 5: THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this website better.

[Interviewer Instructions: Collect materials and escort respondent out of the interviewing room and to the lobby or street. Please give the respondent incentive envelope [$40 gift card], and thank them for their time and effort.]

Testing Protocol: Usability Evaluation of NHES PFI-Enrolled

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us out with a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. This survey asks about students’ and families experience with their schools.

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know. Would you like to be part of this study?

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

Use your internet browser to find the address for the Department of Education

Please tell me what you are thinking while you are figuring out how to find the address.

[Probe as needed for detail: Where do you plan to click first? Can you tell me more about why you clicked [xxx]?]

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

Please keep talking.
Tell me what you are thinking.
What are you planning to do next?
Is there anything else you can do?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Can you tell me if there is anything confusing on the page?
[If you pick up on a visual cue that indicates confusion like going to previous pages or rereading instructions]: “Tell me what just happened. Tell me about what you were just doing.” [In response to R actions or comments]: What does the feature mean to you?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

Observations to keep in mind:

Does user try to use the enter button to move to the next page?
Does user try to use the browser back button to move backward in the survey, instead of the “Previous” link?
How does the user respond to error messages?
How does the user feel about the overall look of the site?
Does the user have a hard time finding the “Next” button or “Previous” link?
Does the user go back and forth between pages (follow up on why if not obvious)?
Does the user try to go back multiple pages, only to get stuck? How do they react?
Does the user skip any questions? Accidentally or on purpose?
Note if any difficulty is content or usability related (when expressed verbally)
Note if any difficulty with skips - are there places where user receives questions that are not applicable to them? When this happens, can they find their way out?
[If user is using mobile device or tablet: can they see what they need to see? Can they navigate? How challenging is it?
Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

[Hand respondent invitation letter (which includes the User ID)]

Task 1: Log into the survey

OBSERVE if respondent has any difficulty using the invitation letter or finding their username

[INSTRUCTIONS: Not all Respondents will proceed through the NHES SCREENER:

If the screener has already been completed (by AIR staff), they will automatically SKIP to the PFI-EN survey (TASK 2).
If the screener has not been completed, respondent will automatically be directed to the SCREENER (below).]

Screener:

How many people live in this household?

OBSERVE if the respondent has any difficulty with “writing in” the answer

2. What is your first name, initials, or nickname?

3. What is your month and year of birth?

OBSERVE if the respondent has any difficulty with the drop down menu

4. What is your sex?

OBSERVE if the respondent has any difficulty with the radio buttons

5. Are you currently in…?

6. What is your current grade or equivalent?

[Screener will repeat QUESTIONS 2-6 for as many household members as entered in QUESTION 1 then will automatically go to the additional PFI-E screener questions]

7. Are you a parent or guardian in this household who knows about (SAMPLED CHILD)'s education?

OBSERVE how respondent handles the automatically filled in information “(SAMPLE CHILD)’s”

8. How are you related to [this child]?

9. Before we take you to the questions about (SAMPLED CHILD)'s education, would you please give us your email address in case we need to contact you further?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

10. Who is the person in this household who knows the most about (SAMPLED CHILD)'s education?

OBSERVE if respondent has any difficulty with automatically tabulated list

[OBSERVE if the respondent has trouble remembering who or how many people they entered about previously

After respondent has completed the screener, STOP and ASK these debriefing questions:

What were your first impressions of the survey when you began it and got into the first several questions?

Do you think there are questions some people would not want to answer online? Which question(s)? Why?

PFI-EN Survey

TASK 2:

INTERVIEWER: [After logging on] before respondent begins the survey, ask these questions:

If you had a question about the survey, what would you do? (OBSERVE: Does respondent have any difficulties finding the FAQ section?)

If you can’t find the answer to your question, what would you do?

If respondent is bilingual or Spanish-speaking:

If you wanted to see an item or page in Spanish [or English], what would you do?

INTERVIEWER: Thank you! Please continue.

Child’s Schooling

1. Based on your household's previous response, [this child] is currently in [grade], is that correct?

No (SKIPS TO QUESTION 3)
Yes (If kindergarten, SKIPS TO QUESTION 2; otherwise, QUESTION 4)

If respondent did not provide child’s name, OBSERVE how respondent handles the automatically filled in information

OBSERVE if the respondent has any difficulty with the radio buttons

2. Is [this child] in full-day or partial-day kindergarten?

Full-day kindergarten (SKIPS TO QUESTION 4)
Partial-day kindergarten (SKIPS TO QUESTION 4)

3. What is [this child]’s current grade or year of school? (If [this child] is not assigned a specific grade, select the grade [he/she] would be in at a school with regular grades.)

Child has not yet started kindergarten

ERROR will occur if chosen: Observe how respondent handles error

Full-day kindergarten
Partial-day kindergarten
1st grade
2nd grade
3rd grade
4th grade
5th grade

6th grade
7th grade
8th grade
9th grade
10th grade
11th grade
12th grade

4. What type of school does [this child] attend?

Private, Catholic (SKIPS TO QUESTION 5)
Private, religious but not Catholic (SKIPS TO QUESTION 5)
Private, not religious (SKIPS TO QUESTION 5)
Public school

3. Is it his/her district-assigned school?

4. Is this school a charter school?

5. Did you move to your current neighborhood so that [this child] could attend [his/her] current school?

6. Does your public school district let you choose which public school you want [this child] to attend?

7. Did you consider other schools for [this child]?

No (SKIPS TO QUESTION 9)
Yes

8. In deciding between schools, did you seek information on the performance of the schools you were considering, like test scores, dropout rates, and so on?

9. Is the school [this child] attends your first choice, that is, the school you wanted most for [him/her] to attend?

10. Since the beginning of this school year, has [this child] been in the same school?

11. In which month did [this child] start at [his/her] current school this school year?

_____________ (month) [Drop Down]

OBSERVE if the respondent has any difficulty with the drop down menu

12. How much do you agree or disagree with the following statement:

"[This child] enjoys school."

13. Please tell us about [this child]’s grades during this school year. Overall, across all subjects, what grades does [this child] get?

14. Is [he/she] currently enrolled in high school Advanced Placement (AP) classes?

15. Since the beginning of this school year, how many times have any of [this child]’s teachers or school staff contacted your household about… (Enter ’0’ if none. Please provide a response for EACH ITEM below.)

a. Behavior problems this child is having in school ____

b. Problems this child is having with school work ____

c. Very good behavior ____

d. Very good school work ____

ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error

OBSERVE if the respondents provide a response for each item or if they only respond to the ones that apply

OBSERVE if respondent enters a 0 or blank if there is none.

16. Since the beginning of this school year, how many days has [this child] been absent from school?

_____ days

ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error

17. Since starting kindergarten, has [this child] repeated any grades?

No (SKIPS TO QUESTION 19)
Yes

18. What grade or grades did [he/she] repeat?

19. Has [this child] ever had the following experiences? (Select one response for EACH ITEM below.)

a. An out-of-school suspension

b. An in-school suspension not counting detentions

c. Been expelled from school

20. How far do you expect [this child] to go in [his/her] education?

21. How would you describe [his/her] work at school?

22. Is [this child] taking any school-related courses online instead of in-person with the teacher?

Do not include courses that use the Internet only for selected assignments.

No (SKIPS TO QUESTION 30)
Yes

23. Is that instruction provided by any of the following places?

Select all that apply.

Your local public school
Your state
A charter school
Another public school
A private school
A college, community college, or university
Someplace else, Specify _________________ (SKIPS TO QUESTION 24)

If respondent selects the “Someplace else” option, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Someplace else” option, please note any reaction to the greyed out box.

OBSERVE if the respondent has any difficulty understanding “select all that apply”

24. Is there a charge or fee for that instruction?

No (SKIPS TO QUESTION 30)
Yes

25. Some parents decide to educate their children at home rather than send them to a public or private school. Is [this child] being schooled at home instead of at school for at least some classes or subjects?

No (SKIPS TO QUESTION 30)
Yes

26. Which of the following statements best describes your homeschooling arrangement for [this child]?

This child is homeschooled for all classes or subject areas.
This child is homeschooled for some classes or subject areas and also attends a public or private school.
This child is not homeschooled. This child attends a public or private school for all classes or subject areas. (SKIPS TO QUESTION 30)

27. How many hours each week does [this child] usually go to a public or private school for instruction? Do not include time spent in extracurricular activities.

Enter '0' if this child is being schooled at home instead of at school for all classes or subject areas.

____ hours

ERROR will occur if respondent enters anything but a whole number: OBSERVE how respondent handles error

28. There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool [this child] because: (Select one response for EACH ITEM below.)

a. You are concerned about the school environment, such as safety, drugs, or negative peer pressure?	No	Yes
b. You are dissatisfied with the academic instruction at other schools?
c. You prefer to teach this child at home so that you can provide religious instruction?
d. You prefer to teach this child at home so that you can provide moral instruction?
e. This child has a physical or mental health problem that has lasted six months or more?
f. This child has a temporary illness that prevents him/her from going to school?
g. This child has other special needs that you feel the school can’t or won’t meet?
h. You are interested in a nontraditional approach to children’s education?
i. You prefer the flexibility homeschooling provides to this child during frequent moves?
j. You have another reason for homeschooling this child?

Specify ___________________________ (SKIPS TO QUESTION 29)

If respondent selects “yes” for option j, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “another reason” option, please note any reaction to the greyed out box.

29. Of the reasons your family chose to homeschool this child, which one would you say is the most important to you? [Automatic response options based on QUESTION 28; ALL NOs SKIPS TO QUESTION 30]

OBSERVE if respondent has any difficulty with automatically tabulated list

Task 3. Hypothetical Scenario – LOG OUT, LOG IN

[Task 3 will occur for all respondents in Round 1, all bilingual respondents in Round 2, and removed in Round 3 as long as issues do not seem to remain]

INTERVIEWER: Stop respondent prior to question 30. Let’s pretend that you didn’t have time to finish the survey right now and wanted to exit the survey and return to finish it later. What would you do?

Observe how the respondent exits the survey.

Does user have any trouble finding the “Save & Continue Later” button?

If…

Respondent finds the “Save & Continue Later” button:

How easy or difficult was it to find the log out link?

Respondent closes the browser without using the “Save & Continue Later” button: Did you see that there was a logout button on the screen?

Why did you choose to close the browser instead of using the logout button?

Interviewer: Now, please go back to finish the survey

Observe how the respondent goes back to finish the survey.

If respondent did not write down or memorize PIN: I see that you don’t have your PIN. Do you think you will still be able to complete the survey?

If respondent says “Yes”: What do you think you will have to do to complete the survey?

Would you attempt to contact someone to try to get your PIN?

Was it clear that you would not be able to return to the survey if you don’t have your PIN?

[Give respondent their PIN to reenter if they don’t have it]

Interviewer: Please continue answering the questions.

Families & School

30. Since the beginning of this school year, has any adult in [this child]’s household done any of the following things at this child’s school? (Select one response for EACH ITEM below.)

a. Attended a school or class event, such as a play, dance, sports event, or science fair

b. Served as a volunteer in this child’s classroom or elsewhere in the school

c. Attended a general school meeting, for example, an open house, or a back-to-school night

d. Attended a meeting of the parent-teacher organization or association

e. Gone to a regularly scheduled parent-teacher conference with this child’s teacher

f. Participated in fundraising for the school

g. Served on a school committee

h. Met with a guidance counselor in person

31. During this school year, how many times has any adult in the household gone to meetings or participated in activities at [this child]’s school?

_____________ number of times.

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

32. During this school year, has your family received any of the following: (Select one response for EACH ITEM below.)

a. Notes or emails specifically about [this child] from [his/her] teachers or school administrators?

b. Newsletters, memos, emails, or notices addressed to all parents?

c. Phone calls specifically about this child from [his/her] teachers or school administrators?

33. How well has [this child]’s school been doing the following things during this school year? (Select one response for EACH ITEM below.)

a. Letting you know how [this child] is doing in school between report cards.

b. Providing information about how to help [this child] with homework.

c. Providing information about why [this child] is placed in particular groups or classes.

d. Providing information on your expected role at [this child]’s school.

e. Providing information on how to help [this child] plan for college or vocational school.

34. How satisfied or dissatisfied are you with each of the following: (Select one response for EACH ITEM below.)

a. The school [this child] attends this year?

b. The teachers [this child] has this year?

c. The academic standards of the school?

d. The order and discipline at the school?

e. The way that school staff interacts with parents?

Homework

35. How often does this child do homework at home, at an after-school program, or somewhere else outside of school?

Less than once a week
1 to 2 days a week
3 to 4 days a week
5 or more days a week
Never (SKIPS TO QUESTION 42)
Child does not have homework (SKIPS TO QUESTION 42)

36. In an average week, how many hours does [this child] spend on homework outside of school?

____ number of hours per week

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

37. How do you feel about the amount of homework [this child] is assigned?

38. How does [this child] feel about the amount of homework [he/she] is assigned?

39. Is there a place in your home that is set aside for [this child] to do homework?

40. How often does any adult in your household check to see that [this child]’s homework is done?

41. During this school year, about how many days in an average week does anyone in your household help [this child] with [his/her] homework?

Family Activities

42. In the past week, has anyone in your family done the following things with this child? (Select one response for EACH ITEM below.)

a. Told [him/her] a story (Do not include reading to this child.)

b. Done activities like arts and crafts, coloring, painting, pasting, or using clay

c. Played board games or did puzzles with [him/her]

d. Worked on a project like building, making, or fixing something

e. Played sports, active games, or exercised together

f. Discussed with [him/her] how to manage time

g. Talked with [him/her] about the family’s history or ethnic heritage

43. In the past week, how many days has your family eaten the evening meal together? (Enter ‘0’ if none.)

____ Days

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

OBSERVE if respondent enters a 0 or blank if there is none.

44. In the past month, has anyone in your family done the following things with this child? (Select one response for EACH ITEM below.)

a. Visited a library

b. Visited a bookstore

c. Gone to a play, concert, or other live show

d. Visited an art gallery, museum, or historical site

e. Visited a zoo or aquarium

f. Attended an event sponsored by a community, religious, or ethnic group

g. Attended an athletic or sporting event outside of school in which this child was not a player

Child’s Health

45. In general, how would you describe this child’s health?

46. Has a health or education professional told you that this child has any of the following conditions? (Select one response for EACH ITEM below.)

a. An intellectual disability (mental retardation)

b. A speech or language impairment

c. A serious emotional disturbance

d. Deafness or another hearing impairment

e. Blindness or another visual impairment not corrected with glasses

f. An orthopedic impairment

g. Autism

h. Pervasive Developmental Disorder (PDD)

i. Attention Deficit Disorder, ADD or ADHD

j. A specific learning disability

k. A developmental delay

l. Traumatic brain injury

m. Another health impairment lasting 6 months or more

47. [Automatic Skip based on response to QUESTION 46: if all of QUESTION 46 was no, SKIPS TO QUESTION 55]

48. Is this child receiving services for [his/her] condition?

No (SKIPS TO QUESTION 53)
Yes

49. Are these services provided by any of the following sources? (Select one response for EACH ITEM below.)

a. Your local school district

b. A state or local health or social service agency

c. A doctor, clinic, or other health care provider

d. [This child]’s private school

50. Are any of these services provided through an Individualized Education Program (IEP) or services plan?

No (SKIPS TO QUESTION 53)
Yes

51. Did any adult in your household work with the service provider or school to develop or change [this child]’s IEP or services plan?

52. During this school year, how satisfied or dissatisfied have you been with the following aspects of [this child]’s IEP or services plan? (Select one response for EACH ITEM below.)

a. The service provider’s or school’s communication with your family?

b. [This child]’s special needs teacher or therapist?

c. The service provider’s or school’s ability to accommodate [this child]’s special needs?

d. The service provider’s or school’s commitment to help [this child] learn?

53. Is [this child] currently enrolled in any special education classes or services?

54. Does [this child]’s condition interfere with [his/her] ability to do any of the following things? (Select one response for EACH ITEM below.)

a. Learn

b. Participate in sports, clubs, or other organized activities

c. Attend school on a regular basis

d. Make friends

Child’s Background

55. Based on your household's previous response, [this child] was born in [month] of [year]. Is that correct?

No
Yes (SKIPS TO QUESTION 57)

56. In what month and year was this child born?

Month ___________ Year of birth _________ [Drop Down]

OBSERVE if the respondent has any difficulty with the drop down menu

57. Where was this child born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESTION 59)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

58. How old was [this child] when [he/she] first moved to the 50 United States or the District of Columbia?

_____ age

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

59. Is [this child] of Hispanic, Latino, or Spanish origin?

60. What is [this child]’s race? You may select one or more races.

61. Based on your household's previous response, [this child] is [gender]. Is that correct?

No
Yes (SKIPS TO QUESTION 63)

62. What is [this child]’s sex?

63. Does this child live at this address and another address (for example, because of a joint custody arrangement)? Do not include vacation properties.

No (SKIPS TO QUESTION 65)
Yes

64. Does this child...

65. What language does this child speak most at home?

Child has not started to speak (SKIPS TO QUESTION 67)
English (SKIPS TO QUESTION 67)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

66. Is [this child] currently enrolled in English as a second language, bilingual education, or an English immersion program?

Household Members

67. Based on your household's previous response, there are # people living in this household. Is that correct?

No
Yes (SKIPS TO QUESTION 69)

OBSERVE if the respondent has trouble remembering who or how many people they entered about previously.

68. How many people live in this household? (Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home.)

__________ people

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

69. How many of the following people live in this household with this child? (Do not include this child in your answer (Example: Brother(s) 2; ….))

This child’s…	Enter ’0’ if none Number
Brother(s)
Sister(s)
Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent's girlfriend/boyfriend/partner
Other relative(s)
Other non-relative(s)

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

OBSERVE if respondent enters a 0 or blank if there is none.

70. How are you related to this child?

Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt
Uncle
Grandmother
Grandfather
Parent’s girlfriend/boyfriend/partner
Other relationship – Specify: ___________________

If respondent selects “Other relationship”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other Relationship” option, please note any reaction to the greyed out box.

71. Which language(s) are spoken at home by the adults in this household?

Select all that apply.

English
Spanish or Spanish Creole
French (including Patois, Creole, Cajun)
Chinese
Other languages—Specify: ___________________

Child’s Family

▶ PARENT 1 LIVING IN HOUSEHOLD
Answer these questions about yourself if you are the child’s parent or guardian. If you are not this child’s parent or guardian, answer these questions about one of this child’s parents or guardians living in the household.

72. [Are you] Is this parent or guardian [the child]’s……

73. [Are you] Is this person male or female?

74. What is [your] this person’s current marital status?

Now married (SKIPS TO QUESTION 76)
Widowed
Divorced
Separated
Never married

75. [Are you] Is this person currently living with a boyfriend/girlfriend or partner in this household?

76. What was the first language [you] this parent or guardian learned to speak?

English (SKIPS TO QUESTION 78)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

77. What language [do you] does this person speak most at home now?

English (SKIPS TO QUESTION 78)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

78. How difficult is it for [you] this person to participate in activities at [this child]’s school because [you speak] [he/she] speaks a language other than English?

79. Does the school have interpreters who speak [your] this person’s native language for meetings or parent-teacher conferences?

77. Does the school have written materials, such as newsletters or school notices, that are translated into [your] this person’s native language?

78. Where [were you] was this parent or guardian born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESTION 80)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

79. How old [were you] was this person when [you] he or she first moved to the 50 United States or the District of Columbia?

________ age

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

80. [Are you] Is this person of Hispanic, Latino, or Spanish origin?

81. What is [your] this person’s race? You may mark one or more races.

82. What is the highest grade or level of school that [you] this parent or guardian completed?

83. [Are you] Is [he or she] currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

84. Which of the following best describes [your] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (SKIPS TO QUESTION 86)
Full-time student (SKIPS TO QUESTION 87)
Stay at home parent (SKIPS TO QUESTION 87)
Retired (SKIPS TO QUESTION 87)
Disabled or unable to work (SKIPS TO QUESTION 87)

85. About how many hours per week [do you] does he or she usually work for pay or income, counting all jobs?

___________ hours (SKIPS TO QUESTION 87)

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error

86. [Have you] Has this parent or guardian been actively looking for work in the past 4 weeks?

87. In the past 12 months, how many months (if any) [have you] has this person worked for pay or income?

___________ months

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

88. How old [are you] is this person?

___________ age

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error

89. How old [were you] was this person when [you] he or she first became a parent to any child?

___________ age

Don’t know

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error

▶ PARENT 2 LIVING IN HOUSEHOLD
Answer these questions about a second parent or guardian living in the household.90.

90. Is there a second parent or guardian living in this household?

No (SKIPS TO QUESTION 112)
Yes

91. Is this person the child’s…

92. Is this person male or female?

93. What is [his/her] this person’s current marital status?

Now married (SKIPS TO QUESTION 95)
Widowed
Divorced
Separated
Never married

94. Is [he/she] this person currently living with a boyfriend/girlfriend or partner in this household?

95. What was the first language [he/she] this parent or guardian learned to speak?

English (SKIPS TO QUESTION 100)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

96. What language does [he/she] this person speak most at home now?

English (SKIPS TO QUESTION 100)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

97. How difficult is it for this person to participate in activities at this child’s school because [he/she] speaks a language other than English?

98. Does the school have interpreters who speak [his/her] this person’s native language for meetings or parent-teacher conferences?

99. Does the school have written materials, such as newsletters or school notices, that are translated into [his/her] this person’s native language?

100. Where was [he/she] this parent or guardian born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESTION 102)
One of the U.S. territories

(Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)

Another country

101. How old was [he/she] this person when [he or she] first moved to the 50 United States or the District of Columbia?

___________ age

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

102. Is [he/she] this person of Hispanic, Latino, or Spanish origin?

103. What is [his/her] this person’s race? You may select one or more races.

104. What is the highest grade or level of school that [he/she] this parent or guardian completed?

105. Is [he or she] currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

106. Which of the following best describes [his/her] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (SKIPS TO QUESTION 108)
Full-time student (SKIPS TO QUESTION 109)
Stay at home parent (SKIPS TO QUESTION 109)
Retired (SKIPS TO QUESTION 109)
Disabled or unable to work (SKIPS TO QUESTION 109)

107. About how many hours per week does [he or she] usually work for pay or income, counting all jobs?

___________ hours

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

108. Has [he/she] this parent or guardian been actively looking for work in the past 4 weeks?

109. In the past 12 months, how many months (if any) has [he/she] this person worked for pay or income?

___________ months

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error

110. How old is [he/she] this person?

___________ age

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error

111. How old was [he/she] this person when [he or she] first became a parent to any child?

___________ age

Don’t know

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a number less than 12: OBSERVE how respondent handles error

Your Household

112. In the past 12 months, did your family ever receive benefits from any of the following programs? (Select one response for EACH ITEM below.)

a. Temporary Assistance for Needy Families, or TANF

b. Your state welfare or family assistance program

c. Women, Infants, and Children, or WIC

d. Food Stamps

e. Medicaid

f. Child Health Insurance Program (CHIP)

g. Section 8 housing assistance

113. Which category best fits the total income of all persons in your household over the past 12 months?

114. How many years have you lived at this address? (Enter ’0’ if less than 1 year.)

___________ years at this address

ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error

OBSERVE if respondent enters a 0 or blank if less than 1 year.

115. Is this house…?

116. Do you have Internet access on a cell phone?

117. Do you have Internet access at home on a computer or tablet?

118. How often do you use the Internet?

119. We would like to identify [this child]’s school so we can include information about the school in our study. Using the list of schools below, select the school [this child] attends.

__________________________________ [DROP DOWN] (SKIPS TO END OF SURVEY)

[This child’s] school not listed above.

OBSERVE if respondent has any difficulty using the interactive tool to choose the school attended by the child

120. Please start typing the name of [this child's] school. A list of matches will pop up. (If school pops up, SKIPS TO END OF SURVEY)

121. To help us identify the school [this child] attends, please enter the name and address in the spaces below.

Thank you/ end survey page, submit button

STEP 4: DEBRIEFING

[Give respondent the QUIS survey]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ________________ as a _______?

Ask the following questions:

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Do you have anything else you would like to mention about the survey that we haven’t had a chance to talk about?

If survey was conducted on a mobile device (tablet or cell):

Overall, what was your impression of conducting the survey on a mobile device?

STEP 5: THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this website better.

Testing Protocol: Usability Evaluation of NHES PFI-Homeschool

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us out with a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. This survey asks about students’ and families experience with homeschooling.

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know. Would you like to be part of this study?

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

Use your internet browser to find the address for the Department of Education

Please tell me what you are thinking while you are figuring out how to find the address.

[Probe as needed for detail: Where do you plan to click first? Can you tell me more about why you clicked [xxx]?]

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

Please keep talking.
Tell me what you are thinking.
What are you planning to do next?
Is there anything else you can do?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Can you tell me if there is anything confusing on the page?
[If you pick up on a visual cue that indicates confusion like going to previous pages or rereading instructions]: “Tell me what just happened. Tell me about what you were just doing.” [In response to R actions or comments]: What does the feature mean to you?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

Observations to keep in mind:

Does user try to use the enter button to move to the next page?
Does user try to use the browser back button to move backward in the survey, instead of the “Previous” link?
How does the user respond to error messages?
How does the user feel about the overall look of the site?
Does the user have a hard time finding the “Next” button or “Previous” link?
Does the user go back and forth between pages (follow up on why, if not obvious)?
Does the user try to go back multiple pages, only to get stuck? How do they react?
Does the user skip any questions? Accidentally or on purpose?
Note if any difficulty is content or usability related (when expressed verbally)
Note if any difficulty with skips - are there places where user receives questions that are not applicable to them? When this happens, can they find their way out?
[If user is using mobile device or tablet] can they see what they need to see? Can they navigate? How challenging is it?
Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

[Hand respondent invitation letter (which includes the User ID)]

Task 1: Log into the survey

OBSERVE if respondent has any difficulty using the invitation letter or finding their username

[INSTRUCTIONS: Not all Respondents will proceed through the NHES SCREENER:

If the screener has already been completed (by AIR staff), they will automatically SKIP to the PFI-HS survey (TASK 2).
If the screener has not been completed, respondent will automatically be directed to the SCREENER (below).]

Screener:

1. How many people live in this household?

OBSERVE if the respondent has any difficulty with “writing in” the answer

2. What is your first name, initials, or nickname?

3. What is your month and year of birth?

OBSERVE if the respondent has any difficulty with the drop down menu

4. What is your sex?

OBSERVE if the respondent has any difficulty with the radio buttons

5. Are you currently in…?

6. What is your current grade or equivalent?

[Screener will repeat QUESTIONS 2-6 for as many household members as entered in QUESTION 1 then will automatically go to the additional PFI-H screener questions]

7. Are you a parent or guardian in this household who knows about (SAMPLED CHILD)'s education?

OBSERVE how respondent handles the automatically filled in information “(SAMPLE CHILD)’s”

8. How are you related to [this child]?

9. Before we take you to the questions about (SAMPLED CHILD)'s education, would you please give us your email address in case we need to contact you further?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

10. Who is the person in this household who knows the most about (SAMPLED CHILD)'s education?

OBSERVE if respondent has any difficulty with automatically tabulated list

------------------------------------------------------------------------------------

[OBSERVE if the respondent has trouble remembering who or how many people they entered previously.

After respondent has completed the screener, STOP and ASK these debriefing questions:

What were your first impressions of the survey when you began it and got into the first several questions?

Do you think there are questions some people would not want to answer online? Which question(s)? Why?

PFI-HS Survey

TASK 2:

INTERVIEWER: [After logging on] before respondent begins the survey, ask these questions:

If you had a question about the survey, what would you do? (OBSERVE: Does respondent have any difficulties finding the FAQ section?)

If you can’t find the answer to your question, what would you do?

If respondent is bilingual or Spanish-speaking:

If you wanted to see an item or page in Spanish [or English], what would you do?

INTERVIEWER: Thank you! Please continue.

Child's Homeschooling

1. Who is the person that mainly provides [this child]’s home instruction?

Mother
Father
Grandparent
Brother/sister
Another person

Who is that? _______________

If respondent selects the “Another person” option, OBSERVE how the respondent handles the “Who is that?” portion. Or if respondent does not select the “Another person” option, please note any reaction to the greyed out box.

If respondent did not provide child’s name, OBSERVE how respondent handles the automatically filled in information

OBSERVE if the respondent has any difficulty with the radio buttons

2. Is any of [this child]’s home instruction provided by a private tutor or teacher?

3. Is any of [this child]’s instruction provided by a local homeschooling group or co-op?

4. Does [this child] attend a public or private school or a college or university for instruction?

No (SKIPS TO QUESTION 7)
Yes

5. What type of school(s) does [this child] attend? (Select all that apply)

Public school (K–12)
Private school (K–12)
College, community college, or university

OBSERVE if the respondent has any difficulty with the select all the apply feature

6. How many hours each week does [this child] usually go to a school for instruction? Do not include time spent in extracurricular activities.

_______ hours

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error

OBSERVE if the respondent has any difficulty with “writing in” the answer

7. Based on your household's previous response, [this child’s] grade or grade equivalent is [grade], is that correct?

No (SEE BELOW)
Yes (SKIPS TO QUESTION 8)

What grade or year would [this child] be in if [he/she] was attending school?

8. These next questions ask you to estimate the amount of time you homeschool [this child].

How many days each week is [this child] homeschooled?

_______ days each week

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error

About how many total hours each week is [he/she] homeschooled?

_______ hours per week

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range: OBSERVE how respondent handles error

9. Since September, has [this child] participated in activities with other children who are homeschooled?

10. Which of the following statements best describes the teaching style used to homeschool [this child]?

11. Thinking about sources of curriculum or books you use to homeschool [this child], please tell us about all the sources that apply to you. Since September, have you used materials from... (Select one response for EACH ITEM below.)

	No	Yes
a. A public library?
b. A homeschooling catalog, publisher, or individual who specializes in homeschooling materials?
c. Another educational publisher?
d. A homeschooling organization?
e. A church, synagogue, or other religious organization?
f. Your local public school or school district?
g. A private school?
h. A bookstore or other store (including online)?
i. Websites, excluding retailers?
j. Other source—Specify _________________

If respondent selects the “Other source” option, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other source” option, please note any reaction to the greyed out box.

OBSERVE if the respondents click a response for each item or if they only select the ones that apply

12. In the past year, have you or another family member taken any courses, either online or in-person, to help you prepare your child’s home instruction?

13. Some homeschooled children take courses over the Internet taught by people outside the household. Is [this child] receiving any instruction this way?

No (SKIPS TO QUESTION 16)
Yes

14. Is that instruction provided by any of the following places?

Select all that apply

Your local public school
Offered by your state
A charter school
Another public school
A private school
A college, community college, or university
Someplace else-Specify _________________

If respondent selects the “Someplace else” option, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Someplace else” option, please note any reaction to the greyed out box.

15. Is there a charge or fee for that instruction?

16. Thinking about typical grade levels, for which grades was [this child] schooled at home for at least some classes or subjects? (Select all that apply. Include the current year.)

17. There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool [this child] because: (Select one response for EACH ITEM below.)

	No	Yes
a. You are concerned about the school environment, such as safety, drugs, or negative peer pressure?
b. You are dissatisfied with the academic instruction at other schools?
c. You prefer to teach [this child] at home so that you can provide religious instruction?
d. You prefer to teach [this child] at home so that you can provide moral instruction?
e. [This child] has a physical or mental health problem that has lasted six months or more?
f. [This child] has a temporary illness that prevents [him/her] from going to school?
g. [This child] has other special needs that you feel the school can’t or won’t meet?
h. You are interested in a nontraditional approach to children’s education?
i. You have another reason for homeschooling your child? Specify: _______________

If respondent selects option “j”, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “another reason” options, please note any reaction to the greyed out box.

18. Of the reasons your family chose to homeschool [this child], which one would you say is the most important to you? [Automatic response options based on QUESTION 17, if QUESTION 17 was all no, SKIPS TO QUESTION 19]

OBSERVE if respondent has any difficulty with automatically tabulated list

19. How far do you expect [this child] to go in [his/her] education?

20. Thinking about all years [this child] has been homeschooled, which of the following subject areas has [this child] been taught during his or her home instruction? (Select all that apply.)

Art
Music
Arithmetic
Basic algebra (Algebra I)
Advanced algebra (Algebra II)
Geometry
Calculus
Probability
Scientific inquiry or experiments
Earth sciences or geology
Biology

Chemistry or physics
Geography
Basic reading/ reading skills
Spelling
English or literature
Computer science (e.g., computer programming)"
Social science, history, social studies
Foreign language
Physical education or gym
Health

21. Which of the following subject areas are being taught to [this child] now? (Select all that apply.)

Art
Music
Arithmetic
Basic algebra (Algebra I)
Advanced algebra (Algebra II)
Geometry
Calculus
Probability
Scientific inquiry or experiments
Earth sciences or geology
Biology

Chemistry or physics
Geography
Basic reading/ reading skills
Spelling
English or literature
Computer science (e.g., computer programming)"
Social science, history, social studies
Foreign language
Physical education or gym
Health

QUESTIONS 20 and 21. The “correct” method of response is to not include any subjects in the 2nd question that were not also included in the first, but we think that may not always be how people respond. OBSERVE how the respondent answers. STOP and PROBE if necessary.

Family Activities

22. In the past week, has anyone in your family done the following things with [this child]?

(Select one response for EACH ITEM below.)

a. Told [him/her] a story (Do not include reading to [this child])

b. Done activities like arts and crafts, coloring, painting, pasting, or using clay

c. Played board games or did puzzles with [him/her]

d. Worked on a project like building, making, or fixing something

e. Played sports, active games, or exercised together

f. Discussed with [him/her] how to manage time

g. Talked with [him/her] about the family’s history or ethnic heritage

23. In the past week, how many days has your family eaten the evening meal together? (Enter ‘0’ if none.)

____ days

OBSERVE if respondent enters a 0 or does not enter a value.

24. In the past month, has anyone in your family done the following things with [this child]?

(Select one response for EACH ITEM below.)

a. Visited a library

b. Visited a bookstore

c. Gone to a play, concert, or other live show

d. Visited an art gallery, museum, or historical site

e. Visited a zoo or aquarium

f. Attended an event sponsored by a community, religious, or ethnic group

g. Attended an athletic or sporting event outside of school in which [this child] was not a player

25. Does your family participate in the activities or meetings of a local homeschooling association, co-op, or other local homeschool group?

No (SKIPS TO QUESTION 27)
Yes

26. Since September, how many times has your family gone to meetings or participated in the activities of a local homeschooling association, co-op, or other local homeschool group?

_______ number of times

[ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error]

27. Is your family or someone in your household a member of a national homeschooling organization?

Task 3. Hypothetical Scenario – LOG OUT, LOG IN

[Task 3 will occur for all respondents in Round 1, all bilingual respondents in Round 2, and removed in Round 3 as long as issues do not seem to remain]

INTERVIEWER: Stop respondent prior to question 28.

Let’s pretend that you didn’t have time to finish the survey right now and wanted to exit the survey and return to finish it later. What would you do?

Observe how the respondent exits the survey.

Does user have any trouble finding the “Save & Continue Later” button?

If…

Respondent finds the “Save & Continue Later” button:

How easy or difficult was it to find the log out button?

Respondent closes the browser without using the “Save & Continue Later” button: Did you see that there was a logout button on the screen?

Why did you choose to close the browser instead of using the logout button?

Interviewer: Now, please go back to finish the survey

Observe how the respondent goes back to finish the survey.

If respondent did not write down or memorize PIN: I see that you don’t have your PIN. Do you think you will still be able to complete the survey?

If respondent says “Yes”: What do you think you will have to do to complete the survey?

Would you attempt to contact someone to try to get your PIN?

Was it clear that you would not be able to return to the survey if you don’t have your PIN?

[Give respondent their PIN to reenter if they don’t have it]

Interviewer: Please continue answering the questions.

Child's Health

28.In general, how would you describe [this child]’s health?

29. Has a health or education professional told you that [this child] has any of the following conditions? (Select one response for EACH ITEM below.)

a. An intellectual disability (mental retardation)

b. A speech or language impairment

c. A serious emotional disturbance

d. Deafness or another hearing impairment

e. Blindness or another visual impairment not corrected with glasses

f. An orthopedic impairment

g. Autism

h. Pervasive Developmental Disorder (PDD)

i Attention Deficit Disorder, ADD or ADHD

j. A specific learning disability

k. A developmental delay

l. Traumatic brain injury

m. Another health impairment lasting 6 months or more

30. Did you mark yes to any condition in question 29? [Automatic Skip based on response to Q29 (SKIPS TO QUESTION 38 IF ALL ANSWERS ARE NO)]

31. Is [this child] receiving services for [his/her] condition?

No (SKIPS TO QUESTION 36)
Yes

32. Are these services provided by any of the following sources? (Select one response for EACH ITEM below.)

a. Your local school district

b. A state or local health or social service agency

c. A doctor, clinic, or other health care provider

d. This child's private school

33. Are any of these services provided through an Individualized Education Program (IEP) or services plan?

No (SKIPS TO QUESTION 36)
Yes

34. Did any adult in your household work with the service provider or school to develop or change [this child]’s IEP or services plan?

35. During this school year, how satisfied or dissatisfied have you been with the following aspects of [this child]’s IEP or services plan? (Select one response for EACH ITEM below.)

a. The service provider’s or school’s communication with your family?

b. [The child]’s special needs teacher or therapist?

c. The service provider’s or school’s ability to accommodate [this child]’s special needs?

d. The service provider’s or school’s commitment to help [this child] learn?

36. Is [this child] currently enrolled in any special education classes or services?

37. Does [this child]’s condition interfere with [his/her] ability to do any of the following things? (Select one response for EACH ITEM below.)

a. Learn

b. Participate in play with other children

c. Go on outings

d. Make friends

Child's Background

38. Based on your household's previous response, [this child] was born in [month] of [year]. Is that correct?

No
Yes (SKIPS TO QUESTION 40)

39. In what month and year was [this child] born?

Month _____ Year _____ [drop down]

OBSERVE if respondent has difficulty with the drop down menu

40. Where was [this child] born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESTION 42)
One of the U.S. territories

(Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)

Another country

41. How old was [this child] when [he/she] first moved to the 50 United States or the District of Columbia?

_______ Age

[ERROR will occur if respondent tries to enter anything but a whole number or the respondent enters a number greater than 18: OBSERVE how respondent handles error]

42. Is [this child] of Hispanic, Latino, or Spanish origin?

43. What is [this child]’s race? You may mark one or more races.

44. Based on your household's previous response, [this child] is [gender]. Is that correct?

No (SKIPS TO QUESTION 46)
Yes

45. What is this child’s sex?

46. Does [this child] live at this address and another address (for example, because of a joint custody arrangement)? (Do not include vacation properties.)

No (SKIPS TO QUESTION 48)
Yes

47. Does [this child]…

48. What language does [this child] speak most at home?

Child has not started to speak (SKIPS TO QUESTION 50)
English (SKIPS TO QUESTION 50)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

49. Is [this child] currently enrolled in English as a second language, bilingual education, or an English immersion program?

Household Members

50. Based on your household's previous response, there are [total number of people in HH] people living in this household. Is that correct?

No
Yes (SKIPS TO QUESTION 52)

OBSERVE if the respondent has trouble remembering who or how many people they entered previously.

51. How many people live in this household? (Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home.)

_______

ERROR will occur if respondent tries to enter anything but a whole number or if the respondent enters a zero: OBSERVE how respondent handles error

52. How many of the following people live in this household with [this child]? Do not include this child in your answer; Example: Brother(s) 2

[This child]’s…	Enter ’0’ if none
Brother(s)
Sister(s)
Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent's girlfriend/boyfriend/partner
Other relative(s)
Other non-relative(s)

[ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or blank if there is none.

53. How are you related to [this child]?

Mother (birth, adoptive, step, or foster)
Father (birth, adoptive, step, or foster)
Aunt
Uncle
Grandmother
Grandfather
Parent’s girlfriend/boyfriend/partner
Other relationship – Specify: _______________

54. Which language(s) are spoken at home by the adults in this household?

Select all that apply.

English
Spanish or Spanish Creole
French (including Patois, Creole, Cajun)
Chinese
Other languages—Specify: _______________

Child's family

PARENT 1 LIVING IN HOUSEHOLD

Answer these questions about yourself if you are the child’s parent or guardian.

If you are not the child’s parent or guardian, answer these questions about one of [this child]’s parents or guardians living in the household.

55. [Are you] Is this parent or guardian the child’s…

56. [Are you] Is this person male or female?

57. What is [your] this person’s current marital status?

Now married (SKIPS TO QUESTION 59)
Widowed
Divorced
Separated
Never married

58. [Are you] Is this person currently living with a boyfriend/girlfriend or partner in this household?

59. What was the first language [you] this parent or guardian learned to speak?

English (SKIPS TO QUESTION 61)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

60. What language [do you] does this person speak most at home now?

61. Where [were you] was this parent or guardian born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESTION 63)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

62. How old [were you] was this person when [you] he or she first moved to the 50 United States or the District of Columbia?

________ age

[ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error]

63. [Are you] Is this person of Hispanic, Latino, or Spanish origin?

64. What is [your] this person’s race? You may mark one or more races.

65. What is the highest grade or level of school that [you] this parent or guardian completed?

66. [Are you] Is he or she currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

67. Which of the following best describes [your] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (SKIPS TO QUESTION 69)
Full-time student (SKIPS TO QUESTION 70)
Stay at home parent (SKIPS TO QUESTION 70)
Retired (SKIPS TO QUESTION 70)
Disabled or unable to work (SKIPS TO QUESTION 70)

68. About how many hours per week [do you] does he or she usually work for pay or income, counting all jobs?

________ hours (SKIPS TO QUESTION 70)

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters zero: OBSERVE how respondent handles error]

69. [Have you] Has this parent or guardian been actively looking for work in the past 4 weeks?

70. In the past 12 months, how many months (if any) [have you] has this person worked for pay or income?

________ months

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number out of range: OBSERVE how respondent handles error]

71. How old [are you] is this person?

________ age

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number less than 12: OBSERVE how respondent handles error]

72. How old [were you] was this person when [you] he or she first became a parent to any child?

________ age

Don’t know

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number less than 12: OBSERVE how respondent handles error]

PARENT 2 LIVING IN HOUSEHOLD

Answer these questions about a second parent or guardian living in the household.

73. Is there a second parent or guardian living in this household?

No (SKIPS TO QUESTION 92)
Yes

74. Is this person the child’s…

75. Is this person male or female?

76. What is [his/her] this person’s current marital status?

Now married (SKIPS TO QUESTION 78)
Widowed
Divorced
Separated
Never married

77. Is [he/she] this person currently living with a boyfriend/girlfriend or partner in this household?

78. What was the first language [he/she] this parent or guardian learned to speak?

English (SKIPS TO QUESTION 80)
Spanish
A language other than English or Spanish
English and Spanish equally
English and another language equally

79. What language does [he/she] this person speak most at home now?

80. Where was [he/she] this parent or guardian born?

One of the 50 United States or the District of Columbia (SKIPS TO QUESITON 82)
One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands)
Another country

81. How old was [he/she] this person when [he or she] first moved to the 50 United States or the District of Columbia?

________ age

[ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error]

82. Is [he/she] this person of Hispanic, Latino, or Spanish origin?

83. What is [his/her] this person’s race? You may select one or more races.

84. What is the highest grade or level of school that [he/she] this parent or guardian completed?

85. Is [he or she] currently attending or enrolled in a school, college, university, or adult learning center, or receiving vocational education or job training?

86. Which of the following best describes [his/her] this person’s employment status?

Employed for pay or income
Self-employed
Unemployed or out of work (SKIPS TO QUESTION 88)
Full-time student (SKIPS TO QUESTION 89)
Stay at home parent (SKIPS TO QUESTION 89)
Retired (SKIPS TO QUESTION 89)
Disabled or unable to work (SKIPS TO QUESTION 89)

87. About how many hours per week does [he or she] usually work for pay or income, counting all jobs?

________ hours (SKIPS TO QUESTION 89)

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a zero: OBSERVE how respondent handles error]

88. Has [he/she] this parent or guardian been actively looking for work in the past 4 weeks?

89. In the past 12 months, how many months (if any) has [he/she] this person worked for pay or income?

________ months

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number out of range: OBSERVE how respondent handles error]

90. How old is [he/she] this person?

________ age

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number less than 12: OBSERVE how respondent handles error]

91. How old was [he/she] this person when [he or she] first became a parent to any child?

________ age

Don’t know

[ERROR will occur if respondent tries to enter anything but a whole number or if respondent enters a number less than 12: OBSERVE how respondent handles error]

Your Household

92. In the past 12 months, did your family ever receive benefits from any of the following programs? (Select one response for EACH ITEM below.)

a. Temporary Assistance for Needy Families, or TANF

b. Your state welfare or family assistance program

c. Women, Infants, and Children, or WIC

d. Food Stamps

e. Medicaid

f. Child Health Insurance Program (CHIP)

g. Section 8 housing assistance

93. Which category best fits the total income of all persons in your household over the past 12 months? (Include your own income. Include money from jobs or other earnings, pensions, interest, rent, Social Security payments, and so on.)

94. How many years have you lived at this address? Enter ’0’ if less than 1 year.

________ years at this address

[ERROR will occur if respondent tries to enter anything but a whole number: OBSERVE how respondent handles error]

OBSERVE if respondent enters a 0 or blank if less than 1 year.

95. Is this house…

96. Do you have Internet access on a cell phone?

97. Do you have Internet access at home on a computer or tablet?

98. How often do you use the Internet?

Thank you/submit page

STEP 4: DEBRIEFING

[Give respondent the QUIS survey]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ________________ as a _______?

Ask the following questions:

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Do you have anything else you would like to mention about the survey that we haven’t had a chance to talk about?

If survey was conducted on a mobile device (tablet or cell):

Overall, what was your impression of conducting the survey on a mobile device?

STEP 5: THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this website better.

Testing Protocol: Usability Evaluation of NHES ATES

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us out with a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Household Education Survey. This survey asks about the education, training, and credentials that adults get for work.

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know. Would you like to be part of this study?

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

□ SIGNED CONSENT FORM COLLECTED

□ COPY OF CONSENT FORM GIVEN TO RESPONDENT

□ IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

Use your internet browser to find the address for the Department of Education

Please tell me what you are thinking while you are figuring out how to find the address.

[Probe as needed for detail: Where do you plan to click first? Can you tell me more about why you clicked [xxx]?]

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about how you will complete the assigned tasks.

Probe Bank:

Please keep talking.
Tell me what you are thinking.
What are you planning to do next?
Is there anything else you can do?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Can you tell me if there is anything confusing on the page?
[If you pick up on a visual cue that indicates confusion like going to previous pages or rereading instructions]: “Tell me what just happened. Tell me about what you were just doing.” [In response to R actions or comments]: What does the feature mean to you?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

Observations to keep in mind:

Does user try to use the enter button to move to the next page?
Does user try to use the browser back button to move backward in the survey, instead of the “Previous” link?
How does the user respond to error messages?
How does the user feel about the overall look of the site?
Does the user have a hard time finding the “Next” button or “Previous” link?
Does the user go back and forth between pages (follow up on why, if not obvious)?
Does the user try to go back multiple pages, only to get stuck? How do they react?
Does the user skip any questions? Accidentally or on purpose?
Note if any difficulty is content or usability related (when expressed verbally)
Note if any difficulty with skips - are there places where user receives questions that are not applicable to them? When this happens, can they find their way out?
[If user is using mobile device or tablet: can they see what they need to see? Can they navigate? How challenging is it?
Note if users are delayed in responding to an item.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

[Hand respondent invitation letter (which includes the User ID)]

Task 1: Log into the survey

OBSERVE if respondent has any difficulty using the invitation letter or finding their username

[INSTRUCTIONS: Not all Respondents will proceed through the NHES SCREENER:

If the screener has already been completed (by AIR staff), they will automatically SKIP to the ATES survey (TASK 2).
If the screener has not been completed, respondent will automatically be directed to the SCREENER (below).]

Screener:

1. How many people live in this household?

OBSERVE if the respondent has any difficulty with “writing in” the answer

2. What is your first name, initials, or nickname?

3. What is your month and year of birth?

OBSERVE if the respondent has any difficulty with the drop down menu

4. What is your sex?

OBSERVE if the respondent has any difficulty with the radio buttons

5. Are you currently in…?

6. What is your current grade or equivalent?

[Screener will repeat QUESTIONS 2-6 for as many household members as entered in QUESTION 1 then will automatically go to the additional ATES screener questions]

7. We would like to ask some additional questions about (SAMPLED PERSON)'s education and job training. Are you (SAMPLED PERSON)?

8. Before we take you to the questions about education and job training, would you please give us your email address in case we need to contact you further?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

9. Would you please give us (SAMPLED PERSON)'s email address so that we may contact (him/her) to do a brief interview about (his/her) education and job training?

OBSERVE how respondent handles the error if they filled in the email addresses incorrectly

OBSERVE if the respondent has trouble remembering who or how many people they entered about previously.

After respondent has completed the screener, STOP and ASK these debriefing questions:

What were your first impressions of the survey when you began it and got into the first several questions?

Do you think there are questions some people would not want to answer online? Which question(s)? Why?

ATES Survey

TASK 2:

[After logging on] before respondent begins the survey, ask these questions:

If you had a question about the survey, what would you do? (OBSERVE: Does respondent have any difficulties finding the FAQ section?)

If you can’t find the answer to your question, what would you do?

If respondent is bilingual or Spanish-speaking:

If you wanted to see an item or page in Spanish [or English], what would you do?

INTERVIEWER: Thank you! Please continue.

Education

1. What is the highest degree or level of school you have completed?

Elementary or high school, but no high school diploma or GED® (Go to question 3)
High school diploma GED® or alternative high school credential (Go to question 3)
Some college credit but less than one year of college credit (Go to question 3)
1 or more years of college credit, no degree
Associate’s degree (for example, AA, AS) Bachelor’s degree (for example, BA, BS)
Master’s degree (for example, MA, MS, MEng, MEd, MSW, MBA)
Professional degree beyond a bachelor’s degree (for example, MD, DDS, DVM, LLB, JD)
Doctorate degree (for example, PhD, EdD)

OBSERVE if the respondent has any difficulty with the radio buttons

2. Which one of the following best describes the field of study for the highest level of school you have completed? (If there was more than one, please choose the one you consider most important.)

General studies, no major, or undeclared major
Accounting, finance, insurance, or real estate
Administrative support Agriculture
Audio, broadcasting, multimedia, or graphic technologies
Business management, administration, or marketing
Communications or journalism
Computer science or information technology
Construction, repair, manufacturing, or transportation
Cosmetology Education
Engineering or architecture
English language or literature
Fine arts or music
Healthcare
Law or legal studies
Law enforcement, security, or firefighting
Liberal arts
Psychology
Religious vocations or theology
Science or mathematics
Social or human services or public administration
Social sciences, political science, economics, or history
Other — Specify:

If respondent selects the “Other” option, OBSERVE how the respondent handles the “Specify” portion. Or if respondent does not select the “Other” option, please note any reaction to the greyed out box.

3. Are you currently enrolled at a college, university, technical or trade school, or other school?

4. Since leaving high school, have you taken any classes to learn English as a second language, sometimes called ESL or ESOL classes?

5. Since leaving high school, have you taken any literacy classes to help improve your reading? (Do not include college-level classes.)

Certifications and Licenses

6. Do you have a currently active professional certification or a state or industry license? Do not include business licenses, such as a liquor license or vending license. (A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teacher, or an IT certification.)

No (Go to Question 30)
Yes

7. How many currently active certifications and licenses do you have? (If you had to get a certification in order to get a license, count each certification and license separately.)

____ Number of certifications and licenses

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a zero: OBSERVE how respondent handles error

OBSERVE if the respondent has any difficulty with “writing in” the answer

8. The next few questions ask about the certification or license that you consider to be your most important. What is the name of your most important certification or license?

9. What kind of work is your [CNNAME1W credential/ most important certification or license] for?

OBSERVE how respondent handles the automatically filled in information

10. Is your [CNNAME1W credential/ most important certification or license] required by a federal, state, or local government agency in order to do that kind of work?

11. Can your [CNNAME1W credential/ most important certification or license] be revoked or suspended for any reason?

12. In what year did you first get your [CNNAME1W credential/ most important certification or license]?

________ year

OBSERVE if respondent has difficulty with the drop down menu

13. Did you prepare for getting your [CNNAME1W credential/ most important certification or license] by… (Select one response for EACH ITEM below.)

a. taking classes from a college, technical school, or trade school?

b. taking classes or training from a company, association, union, or private instructor?

c. studying on my own using textbooks or online resources?

OBSERVE if the respondents click a response for each item or if they only select the ones that apply

14. Is your [CNNAME1W credential/ most important certification or license] for your current job?

15. How useful has your [CNNAME1W credential/ most important certification or license] been for each of the following? (Select one response for EACH ITEM below.)

a. Getting a job

b. Keeping a job

c. Keeping you marketable to employers or clients

d. Improving your work skills

16. Do you have another currently active certification or license?

No (Go to Question 30)
Yes

17. What is the name of your second-most important certification or license?

18. What kind of work is your [CNNAME2W credential/second-most important certification or license] for?

19. Is your [CNNAME2W credential/second-most important certification or license] required by a federal, state, or local government agency in order to do that kind of work?

20. Can your [CNNAME2W credential/second-most important certification or license] be revoked or suspended for any reason?

21. In what year did you first get your [CNNAME2W credential /second-most important certification or license]?

22. Did you prepare for getting your [CNNAME2W credential /second-most important certification or license] by… (Select one response for EACH ITEM below.)

a. taking classes from a college, technical school, or trade school?

b. taking classes or training from a company, association, union, or private instructor?

c. studying on my own using textbooks or online resources?

23. Is your [CNNAME2W credential/second-most important certification or license] for your current job?

24. How useful has your [CNNAME2W credential/second-most important certification or license] been for each of the following? (Select one response for EACH ITEM below.)

a. Getting a job

b. Keeping a job

c. Keeping you marketable to employers or clients

d. Improving your work skills

25. Do you have another currently active certification or license?

No (Go to question 30)
Yes

26. What is the name of your third-most important certification or license?

27. What kind of work is your [CNNAME3W credential/third-most important certification or license] for?

28. Is your [CNNAME3W credential/third-most important certification or license] required by a federal, state, or local government agency in order to do that kind of work?

29. Can your [CNNAME3W credential/third-most important certification or license] be revoked or suspended for any reason?

Task 3. Hypothetical Scenario – LOG OUT, LOG IN

[Task 3 will occur for all respondents in Round 1, all bilingual respondents in Round 2, and removed in Round 3 as long as issues do not seem to remain]

INTERVIEWER: Stop respondent prior to question 30.

Let’s pretend that you didn’t have time to finish the survey right now and wanted to exit the survey and return to finish it later. What would you do?

Observe how the respondent exits the survey.

Does user have any trouble finding the “Save & Continue Later” button?

If…

Respondent finds the “Save & Continue Later” button:

How easy or difficult was it to find the logout button?

Respondent closes the browser without using the “Save & Continue Later” button: Did you see that there was a logout button on the screen?

Why did you choose to close the browser instead of using the logout button?

Interviewer: Now, please go back to finish the survey

Observe how the respondent goes back to finish the survey.

If respondent did not write down or memorize PIN: I see that you don’t have your PIN. Do you think you will still be able to complete the survey?

If respondent says “Yes”: What do you think you will have to do to complete the survey?

Would you attempt to contact someone to try to get your PIN?

Was it clear that you would not be able to return to the survey if you don’t have your PIN?

[Give respondent their PIN to reenter if they don’t have it]

Interviewer: Please continue answering the questions.

Certificates

30. People sometimes earn certificates from an education or training program. These are different from certifications or licenses. Do not include certifications or licenses here. Have you ever earned any of the following types of certificates? (Select one response for EACH ITEM below.)

a. A certificate for completing a training program from an employer, employment agency, union, software or equipment manufacturer, or other training provider.

b. A certificate for completing a vocational program at a high school.

c. A high school equivalency certificate, such as a GED®.

d. A certificate—not a degree—for completing a program at a community or technical college, or other school after high school. Do not include teaching certificates or college degrees. (Go to Question 39)

31. We will refer to certificates earned for completing a program at a community or technical college, or other school after high school as “post-secondary certificates". What was the field of study for your last post-secondary certificate?"

Accounting, finance, insurance or real estate
Administrative support
Agriculture
Audio, broadcasting, multimedia, or graphic technologies
Business management, administration, or marketing
Computer science or information technology
Construction trades
Cosmetology
Culinary arts
Education
Engineering technologies or drafting
Fine arts or music
Funeral service or mortuary science
Healthcare
Law enforcement, security, or firefighting
Law or legal studies
Liberal arts
Manufacturing or production (for example machinist, welder, boilermaker)
Mechanic or repair technologies
Transportation

Other — Specify:

32. Who gave you your [PSFOS_2/last post-secondary] certificate?

A community college
A vocational, technical, trade, or business school
Another college or university
Someplace else— Specify:

33. About how many hours of instruction did you complete in order to earn your [PSFOS_2/last post-secondary] certificate?

34. Which one of the following was required for enrolling in your [PSFOS_2/last post-secondary] certificate program?

35. To earn your [PSFOS_2/last post-secondary] certificate did you have to complete…

(Select one response for EACH ITEM below.)

a. a minimum number of credits?

b. a minimum number of instructional hours?

36. Was your [PSFOS_2/last post-secondary] certificate part of the training you took for a professional certification or license?

37. Is your current job related to your [PSFOS_2/last post-secondary] certificate?

38. How useful has your [PSFOS_2/last post-secondary] certificate been for each of the following? (Select one response for EACH ITEM below.)

Getting a job

Increasing your pay

Improving your work skills

Work Experience Programs

39. Have you ever completed an internship, co‐op, practicum, clerkship, externship, residency, clinical experience, apprenticeship, or similar program?

No, and I am not in one now (Go to question 50)
No, but I am in one now (Go to question 50)
Yes, I have completed this type of program (Go to question 40)

We will refer to these as "work experience programs." Please answer the following questions about the last work experience program you completed.

40. What type of work was your last work experience program for?

Building or construction trades:

Carpenter
Electrician
Plumber or pipefitter
Sheet metal worker or structural steel worker
Other building and construction trades

Healthcare:

Medical doctor
Nursing or nursing assistant
Other healthcare

Other types of work:

Accounting, finance, insurance, or real estate
Chef, cook, or food preparation
Computer networking or information technology
Cosmetology
Driving, piloting, or other transportation
Engineering or architecture
Funeral service or mortuary science
Law enforcement, security, or firefighting
Legal practice
Machinist or tool and die maker
Management or administration
Mechanic or repair work
Printing
Social work, counseling, or religious vocations
Teaching
Utility or telecommunications technician
Other— Specify:_________________________________

41. How long did your [WEFOLP_2/last work experience] program last?

42. What wage did you earn as part of your [WEFOLP_2/last work experience] program?

43. As a part of your [WEFOLP_2/last work experience] program did you… (Select one response for EACH ITEM below.)

a. have instruction or training from a co-worker or supervisor?

b. take classes from a college, technical school, or trade school?

c. take classes or training from a company, association, union, or private instructor?

44. Do the following statements describe your [WEFOLP_2/last work experience] program?

(Select one response for EACH ITEM below.)

a. I was evaluated by a co-worker or supervisor

b. I got college credit

c. I received journeyman status at the end of an apprenticeship

d. I got a state or federal apprenticeship number

45. Which one of the following best describes your [WEFOLP_2/last work experience] program?

46. Did (or will) your [WEFOLP_2/last work experience] program help you earn a professional certification or license?

47. Is your current job related to your [WEFOLP_2/last work experience] program?

48. In your current job, how often do you use the skills or knowledge that you learned during your [WEFOLP_2/last work experience] program?

49. How useful was your [WEFOLP_2/last work experience] program for each of the following? (Select one response for EACH ITEM below.)

Getting a job

Increasing your pay

Improving your work skills

Employment

50. Last week, were you employed for pay at a job or business?

If you were temporarily absent from a job or business (on vacation, temporarily ill, on maternity leave, etc.), answer “Yes.”

No (Go to question 56)
Yes

51. For the job or business you were in last week, were you a member of a labor union or an employee association similar to a union (for example, AFL-CIO, Change to Win Federation, NEA)?

52. Last week, how many jobs did you have?

__________________

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a zero: OBSERVE how respondent handles error

53. Last week, did you work at a full-time job (a job where you work 35 hours or more per week)?

54. Last week, did you work at a part-time job (a job where you work fewer than 35 hours per week)?

No (Go to question 60)
Yes (Go to question 55)

55. Would you have preferred for your part-time job to be a full-time job?

No (Go to question 60)
Yes (Go to question 60)

56. Last week, were you on layoff from a job?

57. During the last 4 weeks, have you been actively looking for work?

No
Yes (Go to question 59)

58. Do you intend to look for work within the next 5 years?

59. When did you last work, even for a few days?

Never worked for pay (Go to question 71)
Over 12 months ago (Go to question 63)
Within the past 12 months (Go to question 60)

60. During the past 12 months (52 weeks), how many weeks did you work, including paid vacation, paid sick leave, and military service?

61. During the past 12 months, in the weeks you worked, how many hours did you usually work each WEEK?

_______ Usual hours worked each WEEK

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a zero: OBSERVE how respondent handles error

62. Which category best fits your earnings from wages, salary, commissions, bonuses, or tips, from all jobs over the past 12 months? (Report amount before deductions for taxes, bonds, dues, or other items.)

63. The next few questions ask about your current or last job. If you had more than one job, describe the one at which you worked the most hours. In your current or last job, for whom did you work?

___ (If now on active duty in the Armed Forces, select this box and enter the branch of the Armed Forces below.)

Name of company, business, or other employer: ______________

64. What kind of business or industry was this? (For example: hospital, newspaper publishing, mail order house, auto engine manufacturing, bank)

65. Which one of the following were you?

66. What kind of work were you doing? (For example: registered nurse, personnel manager, supervisor of order department, secretary, accountant)

67. What were your most important activities or duties? (For example: patient care, directing hiring policies, supervising order clerks, typing and filing, reconciling financial records)

68. Did you have a license that was required by a federal, state, or local government agency to do this job?

69. What kind of position did you hold?

Permanent (Go to question 71)
Temporary (Go to question 70)

70. Would you have preferred to work at a permanent job?

Background

71. Have you ever served on active duty in the U.S. Armed Forces, Reserves, or National Guard?

No, never served in the military (Go to question 73)
Yes, but only on active duty for training in the Reserve or National Guard (Go to question 72)
Yes, on active duty now or in past (Go to question 72)

72. Have you served on active duty since September 2001?

73. Based on your household's previous response, you are [gender]. Is that correct?

No (Go to question 74)
Yes (Go to question 75)

74. Are you male or female?

75. What is your current marital status?

Now married (Go to question 77)
Widowed
Divorced
Separated
Never married

76. Are you currently living with a boyfriend/girlfriend or partner?

77. Do you speak a language other than English at home?

No (Go to question 79)
Yes

78. How well do you speak English?

79. Based on your household's previous response, you are [AGE] years old. Is that correct?

No (Go to question 80)
Yes

80. How old are you?

____ years old

ERROR will occur if respondent enters anything but a whole number or if the respondent enters a number out of range (less than 15, older than 66): OBSERVE how respondent handles error

81. Are you of Hispanic, Latino, or Spanish origin?

82. What is your race? You may select one or more races.

83. Do you have Internet access on a cell phone?

84. Do you have Internet access at home on a computer or tablet?

85. How often do you use the Internet?

Thank you/ end survey page, submit button

STEP 4: DEBRIEFING

[Give respondent the QUIS survey]

Review QUIS responses – probe any items rated less than 5

Could you tell me more about why you rated ________________ as a _______?

Ask the following questions:

Overall, what was your impression of the survey?

What features did you find the most helpful to you?

What features did you find the least helpful to you?

Are there any other features you would like to see in the platform?

Do you have anything else you would like to mention about the survey that we haven’t had a chance to talk about?

If survey was conducted on a mobile device (tablet or cell):

Overall, what was your impression of conducting the survey on a mobile device?

STEP 5: THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this website better.

Questionnaire for User Interaction Satisfaction (QUIS)

Please read each statement carefully. Rate each statement using the words listed above the endpoints of the scale that most appropriately reflect your impressions about responding to this web survey. (Not applicable = NA)

1.	Overall reactions to the interface:	difficult	easy
		1 2 3 4 5 6 7 8 9		NA

		rigid	flexible
		1 2 3 4 5 6 7 8 9			NA

2.	Characters on the computer screen	hard to read	easy to read
		1 2 3 4 5 6 7 8 9			NA

3.	Getting started	difficult	easy
		1 2 3 4 5 6 7 8 9			NA

4.	Screen layouts were helpful	never	always
		1 2 3 4 5 6 7 8 9			NA

5.	Arrangement of information on screen	illogical	logical
		1 2 3 4 5 6 7 8 9			NA

6.	Sequence of screens	confusing	clear
		1 2 3 4 5 6 7 8 9			NA

7.	Going back to the previous screen	impossible	easy
		1 2 3 4 5 6 7 8 9			NA

8.	Use of terminology throughout system	inconsistent	consistent
		1 2 3 4 5 6 7 8 9			NA

9.	Messages which appear on screen	confusing	clear
		1 2 3 4 5 6 7 8 9			NA

10.	Error messages	unhelpful	helpful
		1 2 3 4 5 6 7 8 9			NA

11.	Error messages clarify the problem	never	always
		1 2 3 4 5 6 7 8 9			NA

12.	Finding specific information using the on-line help	difficult	easy
		1 2 3 4 5 6 7 8 9			NA

Testing Protocol: Usability Evaluation of NHES Screener (Spanish)

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA

Instrucciones para el entrevistador: El siguiente protocolo NO debe ser leído textualmente. El texto en letra cursiva solo es contenido con el cual el entrevistador debe familiarizarse antes de la entrevista. El texto entre paréntesis [] contiene instrucciones para el entrevistador.

Hola, mi nombre es <su nombre> y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. Gracias por venir a ayudarnos el día de hoy.

[Platique de algún tema casual para establecer una buena relación con el participante.]:

Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta acerca de las experiencias educativas o de entrenamiento para el trabajo de niños y adultos de todas las edades.

Estoamos interesados en conocer sus opiniones acerca de la página web que vamos a utilizar para realizar la encuesta. Le voy a pedir que complete una encuesta como si estuviera completando una encuesta real en su casa, pero le voy a parar de vez en cuando para hacerle preguntas o pedirle que complete una tarea específica que puede que usted haga o no de manera habitual. Cuando le pida que complete una determinada tarea, es posible que le pregunte que me explique cómo está realizando dicha tarea. Esto no es una evaluación de su conocimiento de internet o computadoras. No existen respuestas correctas o incorrectas y está bien si usted no sabe algunas de las respuestas. Esta simulación nos ayudará para mejorar la página web y hacerla más clara para otras personas como usted.

Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va a observar nuestra sesión de hoy].

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan revisar la grabación más tarde. Cualquier información que se refiera a usted o su familia personalmente, tal como su nombre o la escuela de su hijo(a), no será incluida en nuestro informe y sus respuestas serán utilizadas para fines de investigación únicamente.

Recuerde que usted no está tomando un examen, simplemente queremos saber si la encuesta funciona bien.

Si en algún momento usted desea detenerse, solamente déjemelo saber. ¿Le gustaría participar en este estudio?

[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y termine la entrevista.]

Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a cualquier pregunta que pueda hacer el participante]

[Dele al encuestado una copia del formulario de consentimiento. Pídale que lea la forma (o léasela, dependiendo de la preferencia del encuestado), responda cualquier pregunta y pídale que firme la forma. Dele una copia del formulario de consentimiento al encuestado.]

□ RECOGER FORMULARIO DE CONSENTIMIENTO FIRMADO

□ ENTREGAR COPIA DEL FORMULARIO DE CONSENTIMIENTO AL PARTICIPANTE

□ SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACIÓN

PASO 2: PRACTICA DE PENSAR EN VOZ ALTA

Mientras esté completando la encuesta, habrá veces en las que quiero que me hable sobre lo que está pensando mientras completa una tarea. Escucharlo hablar sobre cómo llega a sus respuestas nos ayudará a entender cómo mejorar la encuesta. Estoy más interesado en ver cómo completa las distintas tareas que en sus respuestas específicas a la encuesta. Para ayudarle a empezar a pensar en voz alta vamos a hacer un ejercicio de práctica. Le voy a dar una tarea relacionada con su [aparato/computadora/celular/tableta] –no la encuesta todavía- y le voy a preguntar que me diga en voz alta lo que se le pasa por la cabeza mientras la completa.

Use su explorador de Internet para encontrar la página web del Departamento de Educación.

Por favor dígame qué está pensando mientras intenta.

[Pregunte cuanto sea necesario para obtener detalles: ¿Dónde está pensando en apretar primero?¿Podría decirme más acerca de por qué ha apretado [xxx]?]

Una vez completado: Muchas gracias; esos son los tipos de detalles que estoy buscando cuando usted me habla de cómo completar las tareas asignadas.

Preguntas Útiles:

Por favor continúe con el proceso de pensar en voz alta.
Dígame lo que está pensando.
¿Qué está pensando en hacer a continuación?
¿Hay algo más que pueda hacer?
[Si el entrevistador nota gestos como el ceño fruncido o cara de perplejidad]: ¿Me podría decir si hay algo confuso en esta página?
[Si el entrevistador nota que el participante parece confundido porque vuelve a páginas anteriores o lee las instrucciones varias]: “Dígame por favor lo que acaba de pasar, lo que estaba usted haciendo.” [En respuesta a las acciones o comentarios de R]: ¿Qué significa esta característica para usted?
[En respuesta a las acciones o comentarios de R]: ¿Qué tal fácil o difícil le resulsta encontrar el botón/link?
[En respuesta a las acciones o comentarios de R]: Dígame, ¿qué ha pasado aquí? ¿Podría decirme lo que estaba haciendo?

Observaciones a tener en cuenta:

¿Intenta el usuario apretar la tecla “enter” para pasar a la página siguiente?
¿Intenta el usuario usar la tecla “atrás” del explorador para ir hacia atrás en la encuesta en vez de apretar el botón “anterior”?
¿Cómo responde el usuario a los mensajes de error?
¿Qué piensa el usuario de la organización del sitio web?
¿Le cuesta trabajo al usuario encontrar el botón de “siguiente” o “anterior”?
¿Navega el usuario hacia delante y hacia atrás entre las distintas pantallas?
¿Intenta el usuario ir varias páginas atrás y se queda atascado? ¿Cómo reacciona?
Facilidad/dificultad del usuario para navegar dentro y entre las preguntas de la encuesta y las diferentes pantallas del explorador
¿Se salta el usuario alguna pregunta? ¿Accidentalmente o a propósito?
Anote si las dificultades están relacionadas con el contenido o al usabilidad (siempre que el usuario las exprese verbalmente)
Note si hay dificultades con los “skip patterns” – ¿hay momentos en los que el usuario recibe preguntas que no le corresponden? Cuando esto ocurre, ¿encuentran una manera de salir de la situación?
[Si el usuario está usando un celular o tableta]: ¿pueden ver todo lo que necesitan ver? ¿Qué tan difícil les resulta?

PASO 3: RESPUESTAS A LA ENCUESTA

[RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL CUANDO PIENSA EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON DE GRAN UTILIDAD.]

Por favor recuerde pensar en voz alta y responder a las preguntas de la manera más precisa posible basándose en su situación. Estoy interesado(a) en escucharle hablar acerca de lo que está pensando mientras decide sus respuestas a las preguntas.

[Entréguele al encuestado la carta de invitación (que contiene el nombre de usuario)]

Ahora es el momento de completar la encuesta en línea. Por favor use la información de esta carta para acceder a la encuesta. Recuerde que estoy muy interesado(a) en escucharle hablar acerca de lo que está pensando mientras decide cómo completar la tarea.

Tarea 1: Acceder a la encuesta

OBSERVE si el encuestado tiene alguna dificultad usando la carta de invitación o encontrando su nombre de usuario

TAREA 2:

[Tras acceder a la encuesta] antes de que el encuestado empiece la encuesta, hágale las siguientes preguntas:

¿Si tuviera una pregunta sobre la encuesta, qué haría? (OBSERVE: ¿Tiene el entrevistado alguna dificultad para encontrar la sección de Preguntas Frecuentes?

Si no encuentra la respuesta a su pregunta, ¿qué haría?

Si el encuestado es bilingüe en español e inglés:

Si quisiera ver una pregunta o página en inglés [o español], ¿qué haría? [Después de que cambien al inglés diga a todos los participantes bilingües en la ronda 2 que continúen en inglés]

ENTREVISTADOR: ¡Gracias! Continúe por favor.

Screener:

1. ¿Cuántas personas viven en este hogar? Incluya a los adultos y niños que se encuentren fuera del hogar temporalmente (Por ejemplo, los que viven en residencias universitarias) si no tienen otro hogar permanente.

_______

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero: OBSERVE cómo el encuestado se arregla con el error]

OBSERVE si el encuestado tiene alguna dificultad con “escribir” la respuesta

Continúe contestando las siguientes preguntas para cada persona que viva en este hogar. Incluya a todos los niños y adultos. Empiece con usted.

2. ¿Cuál es su nombre, iniciales o apodo? Los nombres se usarán solo para preguntar sobre la educación de una persona específica.

_________________________

OBSERVE si el encuestado se siente incómodo con alguna de las preguntas del screener y si tiene que ver con no querer introducir información personal online.

3. ¿Cuál es el mes y el año de nacimiento de esta persona?

Mes _________ Año de nacimiento _________ [drop down]

OBSERVE si el encuestado tiene alguna dificultad con el menú desplegable

4. ¿Cuál es su sexo?

Masculino
Femenino

OBSERVE si el encuestado tiene alguna dificultad con los botones de marcar las opciones

Tarea 3. Escenario hipotético – SALIR, ACCEDER

[La Tarea 3 tendrá lugar para todos los participantes en la Ronda 1, todos los participantes bilingües en la Ronda 2 y se eliminará de la Ronda 3 tanto en cuanto los problemas no prevalezcan]

[ENTREVISTADOR: Pare al encuestado antes de la pregunta 5.]

Imaginemos ahora que no le dio tiempo a terminar la encuesta en este momento y quiere salir de la encuesta y volver para terminarla más adelante. ¿Qué haría?

Observe cómo el encuestado sale de la encuesta.

¿Tiene el usuario algún problema para encontrar el botón de “Guardar y Continuar después”?

Si…

El encuestado encuentra el botón de “Guardar y Continuar después”:

¿Qué tan fácil o difícil le resultó encontrar el botón de salir?

El encuestado cierra el explorador sin usar el botón de “Guardar y Continuar después” ¿Vio usted que había un botón de salir en la pantalla?

¿Por qué decidió cerrar el explorador en lugar de usar el botón de salir?

Entrevistador: Ahora por favor regrese para finalizar la encuesta

Observe cómo el encuestado regresa para finalizar la encuesta.

Si el encuestado no se apuntó o memorizó su número de PIN: Veo que no tiene su número de PIN. ¿Cree que todavía sería capaz de completar la encuesta?

Si el encuestado dice “Sí”: ¿Qué cree que tendría que hacer para completar la encuesta?

¿Contactaría con alguien para intentar obtener su número de PIN?

¿Estaba claro que usted no podría volver a la encuesta si no tenía su número de PIN?

[Dele al encuestado su número de PIN para entrar de nuevo si no lo tiene]

Entrevistador: Por favor continúe respondiendo a las preguntas.

5. ¿Asiste usted actualmente a…

Escuela en el hogar (homeschool) en lugar de asistir a una escuela pública o privada para algunas o todas las clases,
Escuela pública o privada, o preescolar,
Universidad o escuela vocacional, o (IF Question 1 = 1, END SURVEY; = 2+, SKIP TO QUESTION 7)
Ninguna? (IF Question 1 = 1, END SURVEY; = 2+, SKIP TO QUESTION 7)

6. ¿Cuál es su grado actual, o su equivalente?

Preescolar (END SURVEY)
Kindergarten
Grado 1
Grado 2
Grado 3
Grado 4
Grado 5
Grado 6
Grado 7
Grado 8
Grado 9
Grado 10
Grado 11
Grado 12
Universidad o escuela vocacional
Ninguna de éstas

7. ¿Cuál es el nombre, iniciales, o apodo de la siguiente persona?

_________________________

OBSERVE si el encuestado tiene dificultad recordando sobre quiénes o cuántas personas introdujeron información al principio.

8. ¿Cuál es el mes y el año de nacimiento de [P2NAME]?

9. ¿Cuál es el sexo de [P2NAME]?

10. ¿Asiste [P2NAME] actualmente a…

Escuela en el hogar (homeschool) en lugar de asistir a una escuela pública o privada para algunas o todas las clases,
Escuela pública o privada, o preescolar,
Universidad o escuela vocacional, o [IF Question 1 = 2, END SURVEY; = 3+, SKIP TO QUESTION 7]
Ninguna? [IF Question 1 = 2, END SURVEY; = 3+, SKIP TO QUESTION 7]

11. ¿Cuál es el grado actual de [esta persona/P2NAME], o su equivalente??

Preescolar (END SURVEY)
Kindergarten
Grado 1
Grado 2
Grado 3
Grado 4
Grado 5
Grado 6
Grado 7
Grado 8
Grado 9
Grado 10
Grado 11
Grado 12
Universidad o escuela vocacional
Ninguna de éstas

[Las Preguntas 7-11 SE REPITEN para tantas personas (máximo 10) como haya en el hogar]

[EN ESTE MOMENTO EL ENCUESTADO SERÁ DIRIGIDO AUTOMÁTICAMENTE A PREGUNTAS DE MUESTREO ADICIONALES ESPECÍFICAS DE LAS DIFERENTES ENCUESTAS TEMÁTICAS DEPENDIENDO DE LA ECUACIÓN DE MUESTREO. POR FAVOR PRESTE ESPECIAL ATENCIÓN A LAS SIGUIENTES PREGUNTAS.]

Preguntas de muestreo de PFI (Enrolled y Homeschool)

Nos gustaría preguntarle más acerca de la educación de (SAMPLED CHILD).

12. ¿Es usted un padre, madre o tutor legal en este hogar que sabe sobre la educación de (SAMPLED CHILD)?

No
Sí (SKIP TO QUESTION 14)

Si el encuestado no dio el nombre de su niño(a), OBSERVE como el encuestado se arregla con la información autocompletada de “(SAMPLED CHILD)”

13. ¿Cuál es su relación o parentesco con (SAMPLED CHILD)?

Madre (biológica, adoptiva, madrastra o sustituta) (SKIP TO QUESTION 15)
2 = Padre (biológico, adoptivo, padrastro o sustituto) (SKIP TO QUESTION 15)
Tía (SKIP TO QUESTION 15)
Tío (SKIP TO QUESTION 15)
Abuela (SKIP TO QUESTION 15)
Abuelo (SKIP TO QUESTION 15)
Novia/Novio o pareja de alguno de los padres (SKIP TO QUESTION 15)
Otra relación o parentesco – Especifique: _______________ (SKIP TO QUESTION 15)

Si el encuestado selecciona la opción “Otra relación o parentesco”, OBSERVE cómo el encuestado se arregla con la sección de “Especifique”. O si el encuestado no selecciona la opción “Otra relación o parentesco”, por favor anote cualquier otra reacción al área en gris.

14. Antes de pasar a las siguientes preguntas sobre la educación de (SAMPLED CHILD), ¿podría por favor darnos su dirección de correo electrónico en caso de que tengamos que contactarle en un futuro?

___________________ ________________________ (SKIPS TO PFI-EN or PFI-HS SURVEY)

Esta pregunta tiene 2 espacios de respuesta. El primer espacio es para escribir la dirección de correo electrónico y el segundo espacio es para confirmarla. Un mensaje de ERROR aparecerá si el encuestado no entra una dirección de correo electrónico válida o si las direcciones no coinciden: OBSERVE cómo el encuestado reacciona al mensaje de error o a introducir la dirección de correo electrónico dos veces.

15. ¿Quién es la persona en este hogar que sabe más acerca de la educación de (SAMPLED CHILD)?

[UNA LISTA DE NOMBRES DE LAS PREGUNTAS 1 a 11 aparecerá]

OBSERVE si el encuestado tiene alguna dificultad con la lista automática

16. ¿Podría por favor darnos la dirección de correo electrónico de [PFIMOST] para que podamos contactar con (él/ella) para una corta entrevista sobre la educación de (SAMPLED CHILD)?

____________________ _______________________ (END SURVEY)

OBSERVE si el encuestado se siente incómodo dando la dirección de correo electrónico de otra persona

Preguntas de muestreo de ECPP

17. ¿Es usted un padre, madre o tutor legal en este hogar que sabe sobre la educación y el cuidado de (SAMPLED CHILD)?

No
Sí (SKIP TO QUESTION 19)

18. ¿Cuál es su relación o parentesco con (SAMPLED CHILD)?

Madre (biológica, adoptiva, madrastra o sustituta) (SKIP TO QUESTION 20)
Padre (biológico, adoptivo, padrastro o sustituto) (SKIP TO QUESTION 20)
Tía (SKIP TO QUESTION 20)
Tío (SKIP TO QUESTION 20)
Abuela (SKIP TO QUESTION 20)
Abuelo (SKIP TO QUESTION 20)
Novia/Novio o pareja de alguno de los padres (SKIP TO QUESTION 20)
Otra relación o parentesco – Especifique: _______________ (SKIP TO QUESTION 20)
Si el encuestado selecciona la opción “Otra relación o parentesco”, OBSERVE cómo el encuestado se arregla con la sección de “Especifique”. O si el encuestado no selecciona la opción “Otra relación o parentesco”, por favor anote cualquier otra reacción al área en gris.

19. Antes de pasar a las siguientes preguntas sobre la educación de (SAMPLED CHILD), ¿podría por favor darnos su dirección de correo electrónico en caso de que tengamos que contactarle en un futuro?

___________________ ________________________ (GO TO ECPP SURVEY)

20. ¿Quién es la persona en este hogar que sabe más acerca de la educación y el cuidado de (SAMPLED CHILD)?

[UNA LISTA DE NOMBRES DE LAS PREGUNTAS 1 a 11 aparecerá]

OBSERVE si el encuestado tiene alguna dificultad con la lista automática

21. ¿Podría por favor darnos la dirección de correo electrónico de [ECPPMOST] para que podamos contactar con (él/ella) para una corta entrevista sobre la educación de (SAMPLED CHILD)?

__________________ _________________________ (END SURVEY)

OBSERVE si el encuestado se siente incómodo dando la dirección de correo electrónico de otra persona

Preguntas de muestreo de ATES

22. Nos gustaría preguntarle más acerca de la educación y el entrenamiento para el trabajo de (SAMPLED PERSON). ¿Es usted (SAMPLED PERSON)?

No (SKIP TO QUESTION 24)
Sí

23. Antes de pasar a las siguientes preguntas sobre educación y entrenamiento para el trabajo, ¿podría por favor darnos su dirección de correo electrónico en caso de que tengamos que contactarle en un futuro?

_____________________ ______________________ (GO TO ATES SURVEY)

24. ¿Podría por favor darnos la dirección de correo electrónico de (SAMPLED PERSON) para que podamos contactar con (él/ella) para una corta entrevista sobre su educación y entrenamiento para el trabajo?

___________________________________________ (END SURVEY)

OBSERVE si el encuestado se siente incómodo dando la dirección de correo electrónico de otra persona

Página de Gracias/no elegible/no seleccionado

OBSERVE cómo los encuestados reaccionan a no ser elegibles o seleccionados

STEP 4: DEBRIEFING

Una vez que el encuestado ha completado el SCREENER, PARE y PREGUNTE las siguientes preguntas de debriefing:

¿Cuáles fueron sus primeras impresiones de la encuesta cuando empezó y contestó a las primeras preguntas?

¿Cree usted que hay preguntas que algunas personas pueden tener dificultad para contestar? ¿Qué pregunta(s)? ¿Por qué?

Nos gustaría que usted llene una breve encuesta acerca de sus experiencias con esta encuesta web que acaba de terminar. No hay respuestas correctas o incorrectas. Sus respuestas son importantes para nosotros y sólo serán utilizados para nuestra información. Podemos hacerle algunas preguntas de seguimiento de acuerdo a sus respuestas para que podamos recibir sus comentarios sobre cómo mejorar nuestra encuesta web.

[Dele al participante la encuesta QUIS]

Revise las respuestas al QUIS – pregunte acerca de cualquier ítem con menos de un 5

Podría decirme un poco más acerca de por qué ha puntuado __________ con un _______?

Haga las siguientes preguntas:

En general, ¿cuál es su impresión de la encuesta?

¿Diría usted que las preguntas de la encuesta fueron fáciles o difíciles de responder? ¿Por qué?

¿Qué características de la encuesta le han resultado de mayor ayuda?

¿Qué tal fácil o difícil le ha resultado completar la encuesta?

¿Hay alguna otra característica que le gustaría ver en la plataforma en línea?

¿Hay algo más que le gustaría mencionar sobre la encuesta de lo que no hayamos hablado?

Si la encuesta tuvo lugar en un celular o tableta:

En general, ¿cuál es su impresión al realizar la encuesta en un celular/tableta?

PASO 5: GRACIAS POR PARTICIPAR

Hemos terminado. Muchas gracias por ayudarnos. La información que nos ha proporcionado nos será de gran ayuda para mejorar esta página web.

[Instrucciones para el entrevistador: recoja todos los materiales y acompañe al participante fuera de la sala de entrevista al lobby o a la calle. Por favor entregue al participante el sobre con el incentivo [tarjeta regalo de $40], y agradézcale por su tiempo y esfuerzo.]

Testing Protocol: Usability Evaluation of NHES ECPP (Spanish)

Interview ID:
Date of Interview:
Interviewee:
Your Name:

MATERIALS NEEDED FOR INTERVIEW

□ INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)

□ INVITATION LETTER WITH USERNAME FOR CORRECT VERSION OF THE SURVEY

□ QUIS Survey

□ COMPUTER, TABLET, OR MOBILE PHONE

□ CONSENT FORM (TWO COPIES)

□ INCENTIVE

□ PAYMENT RECEIPT

□ DIGITAL RECORDER AND EXTRA BATTERIES

□ PENS AND PAPER

□ AIR GUEST WIRELESS USERNAME AND PASSWORD

□ EXTRA LAPTOP AND CHARGING CORD

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA

Hola, mi nombre es <su nombre> y trabajo para [COMPAÑIA]. Mucho gusto de conocerle. Gracias por venir a ayudarnos el día de hoy.

[Platique de algún tema casual para establecer una buena relación con el participante.]:

Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta acerca de la educación temprana de los niños, los programas en los que los niños participan y cómo las familias preparan a los niños para la escuela].

Estamos interesados en conocer sus opiniones acerca de la página web que vamos a utilizar para realizar la encuesta. Le voy a pedir que complete una encuesta como si estuviera completando una encuesta real en su casa, pero le voy a parar de vez en cuando para hacerle preguntas o pedirle que complete una tarea específica que puede que usted haga o no de manera habitual. Cuando le pida que complete una determinada tarea, es posible que le pregunte que me explique cómo está realizando dicha tarea. Esto no es una evaluación de su conocimiento de internet o computadoras. No existen respuestas correctas o incorrectas y está bien si usted no sabe algunas de las respuestas. Esta simulación nos ayudará para mejorar la página web y hacerla más clara para otras personas como usted.

Recuerde que usted no está tomando un examen, simplemente queremos saber si la encuesta funciona bien.

Si en algún momento usted desea detenerse, solamente déjemelo saber. ¿Le gustaría participar en este estudio?

[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y termine la entrevista.]

Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a cualquier pregunta que pueda hacer el participante]

□ RECOGER FORMULARIO DE CONSENTIMIENTO FIRMADO

□ ENTREGAR COPIA DEL FORMULARIO DE CONSENTIMIENTO AL PARTICIPANTE

□ SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACIÓN

PASO 2: PRACTICA DE PENSAR EN VOZ ALTA

Use su explorador de Internet para encontrar la página web del Departamento de Educación.

Por favor dígame qué está pensando mientras intenta encontrar la dirección web.

[Pregunte cuanto sea necesario para obtener detalles: ¿Dónde está pensando en apretar primero?¿Podría decirme más acerca de por qué ha apretado [xxx]?]

Una vez completado: Muchas gracias; esos son los tipos de detalles que estoy buscando cuando usted me habla de cómo completar las tareas asignadas.

Preguntas Útiles:

Por favor continúe con el proceso de pensar en voz alta.
Dígame lo que está pensando.
¿Qué está pensando en hacer a continuación?
¿Hay algo más que pueda hacer?
[Si el entrevistador nota gestos como el ceño fruncido o cara de perplejidad]: ¿Me podría decir si hay algo confuso en esta página?
[Si el entrevistador nota que el participante parece confundido porque vuelve a páginas anteriores o lee las instrucciones varias]: “Dígame por favor lo que acaba de pasar, lo que estaba usted haciendo.” [En respuesta a las acciones o comentarios de R]: ¿Qué significa esta característica para usted?
[En respuesta a las acciones o comentarios de R]: ¿Qué tal fácil o difícil le resulta encontrar el botón/link?
[En respuesta a las acciones o comentarios de R]: Dígame, ¿qué ha pasado aquí? ¿Podría decirme lo que estaba haciendo?

Observaciones a tener en cuenta:

¿Intenta el usuario apretar la tecla “enter” para pasar a la página siguiente?
¿Intenta el usuario usar la tecla “atrás” del explorador para ir hacia atrás en la encuesta en vez de apretar el botón “anterior”?
¿Cómo responde el usuario a los mensajes de error?
¿Qué piensa el usuario de la organización del sitio web?
¿Le cuesta trabajo al usuario encontrar el botón de “siguiente” o “anterior”?
¿Navega el usuario hacia delante y hacia atrás entre las distintas pantallas?
¿Intenta el usuario ir varias páginas atrás y se queda atascado? ¿Cómo reacciona?
Facilidad/dificultad del usuario para navegar dentro y entre las preguntas de la encuesta y las diferentes pantallas del explorador
¿Se salta el usuario alguna pregunta? ¿Accidentalmente o a propósito?
Anote si las dificultades están relacionadas con el contenido o al usabilidad (siempre que el usuario las exprese verbalmente)
Note si hay dificultades con los “skip patterns” – ¿hay momentos en los que el usuario recibe preguntas que no le corresponden? Cuando esto ocurre, ¿encuentran una manera de salir de la situación?
[Si el usuario está usando un celular o tableta]: ¿pueden ver todo lo que necesitan ver? ¿Qué tan difícil les resulta?

PASO 3: RESPUESTAS A LA ENCUESTA

[RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL CUANDO PIENSA EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON DE GRAN UTILIDAD.]

[Entréguele al encuestado la carta de invitación (que contiene el nombre de usuario)]

Tarea 1: Acceder a la encuesta

OBSERVE si el encuestado tiene alguna dificultad usando la carta de invitación o encontrando su nombre de usuario

[INSTRUCCIONES: No todos los Encuestados tendrán que completar el SCREENER de NHES:

Si el Screener ha sido completado previamente (por el personal de AIR), el encuestado PASARÁ automáticamente a completar la encuesta ECPP (TAREA 2).
Si el screener no ha sido completado previamente, el encuestado será dirigido directamente al SCREENER (a continuación).]

Screener:

1. ¿Cuántas personas viven en este hogar?

OBSERVE si el encuestado tiene alguna dificultad con “escribir” la respuesta

2. ¿Cuál es su nombre, iniciales o apodo?

OBSERVE si el encuestado tiene alguna dificultad con el menú desplegable

3. ¿Cuál es su mes y año de nacimiento?

4. ¿Cuál es su sexo?

OBSERVE si el encuestado tiene alguna dificultad con los botones de seleccionar una opción

5. ¿Asiste usted actualmente a…

6. ¿Cuál es su grado actual, o su equivalente?

[El Screener repetirá las PREGUNTAS 2-6 para tantos miembros del hogar como se hayan introducido en la PREGUNTA 1 y a continuación pasará automáticamente a las cuestiones adicionales para el screener del ECPP]

7. ¿Es usted un padre, madre o tutor legal en este hogar que sabe sobre la educación y el cuidado de (SAMPLED CHILD)?

Si el encuestado no dio el nombre de su niño(a), OBSERVE como el encuestado se arregla con la información autocompletada de “(SAMPLED CHILD)”

8. ¿Cuál es su relación o parentesco con (este niño(a))? Otra especifique

9. Antes de pasar a las siguientes preguntas sobre la educación de (SAMPLED CHILD), ¿podría por favor darnos su dirección de correo electrónico en caso de que tengamos que contactarle en un futuro?

OBSERVE cómo el encuestado se maneja con el error si trata de completar la dirección de correo electrónico incorrectamente

10. ¿Quién es la persona en este hogar que sabe más acerca de la educación de (SAMPLED CHILD)?

OBSERVE si el encuestado tiene alguna dificultad con la lista automática

11. ¿Podría por favor darnos la dirección de correo electrónico de [PFIMOST] para que podamos contactar con (él/ella) para una corta entrevista sobre la educación de (SAMPLED CHILD)?

OBSERVE cómo el encuestado se maneja con el error si trata de completar la dirección de correo electrónico incorrectamente

[OBSERVE si el encuestado tiene problemas para recordar sobre quién o cuántas personas introdujeron información al principio.

Una vez que el encuestado haya completado el screener, PARE y PREGUNTE las siguientes cuestiones de aclaración/debriefing:

¿Cuáles fueron sus primeras impresiones de la encuesta cuando empezó y contestó a las primeras preguntas?

¿Cree usted que hay preguntas que algunas personas pueden tener dificultad para contestar? ¿Qué pregunta(s)? ¿Por qué?

Encuesta ECPP

TAREA 2:

[Tras acceder a la encuesta] antes de que el encuestado empiece la encuesta, hágale las siguientes preguntas:

¿Si tuviera una pregunta sobre la encuesta, qué haría? (OBSERVE: ¿Tiene el entrevistado alguna dificultad para encontrar la sección de Preguntas Frecuentes?

Si no encuentra la respuesta a su pregunta, ¿qué haría?

Si el encuestado es bilingüe en español e inglés:

ENTREVISTADOR: ¡Gracias! Continúe por favor.

Cuidado Infantil y Programas para la Niñez

1. Actualmente, ¿cuida a [este niño(a)] con regularidad un pariente, distinto a uno de los padres o tutor legales? Por ejemplo, los abuelos, hermanos, hermanas u otro pariente.

No (Go to Question 17)
Sí

Si el encuestado no proporcionó el nombre del niño, OBSERVE cómo se arregla con la información autocompletada

OBSERVE si el encuestado tiene alguna dificultad con los botones de marcar las opciones

2. ¿Se planifica con regularidad alguno de estos arreglos de cuidado infantil por lo menos una vez a la semana?

No (Go to Question 17)
Sí

3. Las siguientes preguntas son acerca del cuidado que [este niño(a)] recibe por parte del pariente que lo cuida la mayor parte del tiempo.

¿Cuál es la relación o el parentesco de ese pariente con este niño(a)?

4. ¿Qué edad tiene el [RCTYPE_ALT/pariente] que cuida a [este niño(a)] la mayor parte del tiempo?

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o un número menor que 12: OBSERVE cómo el encuestado se arregla con el error]

5. ¿Cuida al niño(a) en su casa o en otra casa?

6. ¿Cuántos días a la semana cuida [RCTYPE_ALT/este pariente] a [este niño(a)]?

_____ días a la semana

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o un valor fuera de rango: OBSERVE cómo el encuestado se arregla con el error]

7. ¿Cuántas horas a la semana cuida [RCTYPE_ALT/este pariente] a [este niño(a)]?

______ horas a la semana

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o 0: OBSERVE cómo el encuestado se arregla con el error] NOTE/PREGUNTE para ver si el encuestado está respondiendo horas a la semana u horas al día

8. ¿Qué edad tenía [este niño(a)], en años y meses, cuando comenzó este arreglo habitual específico con [RCTYPE_ALT/este pariente]?

_____ años ______meses

[Un mensaje de ERROR aparecerá si el encuestado introduce una edad superior a la que proporcionó en el screener, otra cosa que no sea un número entero o un valor fuera de rango: OBSERVE cómo el encuestado se arregla con el error]

9. ¿Qué idioma habla [RCTYPE_ALT/este pariente] la mayor parte del tiempo cuando cuida a [este niño(a)]?

10. ¿Cuida [RCTYPE_ALT/este pariente] a [este niño(a)] si el niño(a) está…
Seleccione una respuesta para CADA OPCIÓN a continuación

a. Enfermo pero no tiene fiebre?

b. Enfermo y tiene fiebre?

OBSERVE si el encuestado selecciona una respuesta para cada ítem o si selecciona solo las que aplican

11. ¿Hay alguna tarifa o pago por el cuidado que [este niño(a)] recibe por parte de [RCTYPE_ALT/este pariente], ya sea que lo pague usted u otra persona o agencia?

No (Go to Question 15)
Sí

12. ¿Le ayudan algunas de las siguientes personas, programas u organizaciones a pagarle a [RCTYPE_ALT/este pariente] por el cuidado de [este niño(a)]?
Seleccione una respuesta para CADA OPCIÓN a continuación

a. Un pariente de este niño(a) que no vive en su hogar y da el dinero para pagar específicamente ese cuidado, sin incluir manutención para niños(as)

b. Asistencia Temporal a Familias Necesitadas (TANF)

c. Otra agencia de servicios
sociales, bienestar público
o cuidado infantil

d. Un empleador, sin incluir cuentas especiales para pagos de cuidado infantil, libres de impuestos

e. Alguien más

13. ¿Cuánto paga su hogar a [RCTYPE_ALT/este pariente] por el cuidado de [este niño(a)], sin incluir dinero que reciba de otras personas para ayudarle a pagar el cuidado del niño(a)?
Introduzca '0' si su hogar no le paga a este pariente por el cuidado del niño(a).

_____

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o 0: OBSERVE cómo el encuestado se arregla con el error]

OBSERVE si el encuestado introduce un 0 o no introduce ningún valor.

¿Es esa cantidad por…

Hora
Día
Semana
Mes
Año
Quincena (cada 2 semanas)
Otra – Especifique:

Si el encuestado selecciona la opción “Otra”, OBSERVE cómo el encuestado se arregla con la sección de “Especifique”. O si el encuestado no selecciona la opción “Otra”, por favor anote cualquier otra reacción al área en gris.

14. ¿Para cuántos de los niños(as) de su hogar, incluyendo a [este niño(a)], es esta cantidad?

15. ¿Tiene [este niño(a)] algún otro arreglo de cuidado infantil con un pariente con regularidad?

No (Go to Question 17)
Sí

16. ¿Cuántas horas a la semana en total pasa [este niño(a)] al cuidado de esos otros parientes?

____ horas a la semana

Cuidado que el niño(a) recibe por parte de personas que no son sus parientes

Las siguientes preguntas son acerca del cuidado que [este niño(a)] recibe por parte de personas que no son sus parientes, ya sea en su casa o en la casa de otra persona. Esto incluye a personas que cuidan niños(as) en sus casas o vecinos, pero no incluye guarderías ni programas preescolares

17. Actualmente, ¿cuida a [este niño(a)], con regularidad, una persona que no es su pariente, ya sea en casa o en otra casa?

No (Go to Question 35)
Sí

18. ¿Se planifica con regularidad alguno de estos arreglos de cuidado infantil por lo menos una vez a la semana?

No (Go to Question 35)
Sí

19. Las siguientes preguntas son acerca del cuidado que [este niño(a)] recibe por parte de una persona que no es su pariente y que lo cuida la mayor parte del tiempo.

¿Cuida al niño(a) en su casa o en otra casa?

20. ¿Vive en el hogar de usted esta persona que cuida a [este niño(a)]?

21. ¿Cuántos días a la semana cuida esta persona a [este niño(a)]?

____ días a la semana

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o un valor fuera de rango: OBSERVE cómo el encuestado se arregla con el error]

22. ¿Cuántas horas a la semana cuida esta persona a [este niño(a)]?

_____ horas a la semana

23. ¿Qué edad tenía [este niño(a)], en años y meses, cuando comenzó este arreglo habitual específico con esta persona?

____ años ____ meses

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o un valor fuera de rango: OBSERVE cómo el encuestado se arregla con el error]

24. ¿Era esta persona que cuida niños(as) alguien a quien usted ya conocía?

25. ¿Es esta persona que cuida [este niño(a)] mayor de 18 años?

26. ¿Qué idioma habla esta persona que cuida niños(as) la mayor parte del tiempo cuando cuida a [este niño(a)]?

27. ¿Cuida esta persona a [este niño(a)] si [el niño(a)] está…
Seleccione una respuesta para CADA OPCIÓN a continuación

a. Enfermo pero no tiene fiebre?

b. Enfermo y tiene fiebre?

28. ¿Le recomendaría esta persona a otros padres?

29. ¿Hay alguna tarifa o pago por el cuidado que [este niño(a)] recibe por parte de esta persona, ya sea que lo pague usted u otra persona o agencia?

No (Go to Question 33)
Sí

30. ¿Le ayudan algunas de las siguientes personas, programas u organizaciones a pagarle a esta persona por el cuidado de [este niño(a)]?
Seleccione una respuesta para CADA OPCIÓN a continuación.

a. Un pariente de [este niño(a)] que no vive en su hogar y da el dinero para pagar específicamente ese cuidado, sin incluir manutención para niños(as)

b. Asistencia Temporal a Familias Necesitadas (TANF)

c. Otra agencia de servicios sociales, bienestar público o cuidado infantil

d. Un empleador, sin incluir cuentas especiales para pagos de cuidado infantil, libres de impuestos

e. Alguien más

31. ¿Cuánto paga su hogar a esta persona por el cuidado de [este niño(a)], sin incluir dinero que reciba de otras personas para ayudarle a pagar el cuidado del niño(a)?
Introduzca '0' si su hogar no le paga a esta persona por el cuidado del niño(a)

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o 0: OBSERVE cómo el encuestado se arregla con el error]

¿Es esa cantidad por…

Hora
Día
Semana
Mes
Año
Quincena (cada 2 semanas)
Otra—Especifique:

32. ¿Para cuántos de los niños(as) de su hogar, incluyendo a [este niño(a)], es esta cantidad?

No (Go to Question 33)
Sí

33. ¿Alguna otra persona que no es pariente cuida a [este niño(a)] con regularidad? No incluya el cuidado en guarderías o programas preescolares.

No (Go to Question 35)
Sí

34. ¿Cuántas horas en total a la semana pasa [este niño(a)] al cuidado de esas otras personas que no son parientes?

_____ horas a la semana

Guarderías y programas preescolares a los que su niño(a) asiste

Las siguientes preguntas son acerca de las guarderías y los programas de educación temprana a los que su este [niño(a)] asiste. Esto no incluye el cuidado que el niño(a) recibe en una casa.

35. Actualmente, ¿asiste [este niño(a)] a una guardería, a un programa preescolar o a un prekindergarten, que no sea una casa?

No (Go to Question 54)
Sí

36. ¿Va [este niño(a)] a una guardería, a un programa preescolar o a un prekindergarten por lo menos una vez a la semana?

No (Go to Question 54)
Sí

37. Las siguientes preguntas son acerca del programa en el que [este niño(a)] pasa la mayor parte del tiempo.

¿Es el programa actual de [este niño(a)] un programa de guardería, preescolar o de prekindergarten?

38. ¿Es este tipo de programa de Head Start o de Early Head Start?
Los programas de Head Start y de Early Head Start son programas preescolares patrocinados por el gobierno federal y orientados principalmente a niños(as) de familias de bajos recursos económicos.

39. ¿Dónde queda este programa?

En una iglesia, sinagoga o en otro lugar de culto religioso
En una escuela primaria o secundaria pública
En una escuela primaria o secundaria privada
En una universidad
En un centro comunitario
En una biblioteca pública
Tiene su propio edificio, oficina o local
Algún otro lugar—Especifique:

Si el encuestado selecciona la opción “Algún otro lugar”, OBSERVE cómo el encuestado se arregla con la sección de “Especifique”. O si el encuestado no selecciona la opción “Algún otro lugar”, por favor anote cualquier otra reacción al área en gris.

40. ¿Es este programa dirigido por una iglesia, sinagoga u otro grupo religioso?

41. ¿Está este programa en su lugar de trabajo o en el lugar de trabajo del otro padre de [este niño(a)]?

42. ¿Cuántos días a la semana asiste [este niño(a)] a este programa?

_____ días a la semana

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o un valor fuera de rango: OBSERVE cómo el encuestado se arregla con el error]

43. ¿Cuántas horas a la semana asiste [este niño(a)] a este programa?

_____ horas a la semana

44. ¿Qué edad tenía [este niño(a)], en años y meses, cuando comenzó a asistir a este programa específicamente?

_____ años _____ meses

45. ¿Qué idioma habla la mayor parte del tiempo el maestro o la persona a cargo de [este niño(a)] en este programa cuando cuida [al niño(a)]?

46. ¿Le recomendaría este programa a otros padres?

47. ¿Ofrece este programa alguno de los siguientes servicios a [este niño(a)] o a su familia?
Seleccione una respuesta para CADA OPCIÓN a continuación.

a. Evaluaciones auditivas, del habla o de la vista

b. Exámenes físicos

c. Exámenes dentales

d. Evaluaciones formales para detectar problemas de desarrollo o de aprendizaje

e. Cuidado infantil cuando el niño(a) está enfermo pero no tiene fiebre

f. Cuidado infantil cuando el niño(a) está enfermo y tiene fiebre

48. ¿Hay alguna tarifa o pago por este programa, ya sea que lo pague usted u otra persona o agencia?

No (Go to Question 52)
Sí

49. ¿Le ayudan algunas de las siguientes personas, programas u organizaciones a pagar para que [este niño(a)] vaya a este programa?
Seleccione una respuesta para CADA OPCIÓN a continuación.

a. Un pariente de este niño que no vive en su hogar y da el dinero para pagar específicamente ese cuidado, sin incluir manutención para niños(as)

b. Asistencia Temporal a Familias Necesitadas (TANF)

c. Otra agencia de servicios sociales, bienestar público o cuidado infantil

d. Un empleador, sin incluir cuentas especiales para pagos de cuidado infantil, libres de impuestos

e. Alguien más

50. ¿Cuánto paga su hogar para que [este niño(a)] vaya a este programa, sin incluir dinero que reciba de otras personas para ayudarle a pagar el cuidado del niño(a)?
Introduzca '0' si su hogar no paga nada por este programa.

[Un mensaje de ERROR aparecerá si el encuestado introduce otra cosa que no sea un número entero o 0: OBSERVE cómo el encuestado se arregla con el error]

OBSERVE si el encuestado introduce un 0 o no introduce ningún valor.

¿Es esa cantidad por…

Hora
Día
Semana
Mes
Año
Quincena (cada 2 semanas)
Otra—Especifique:

51. ¿Para cuántos de los niños(as) de su hogar, incluyendo a [este niño(a)], es esta cantidad?

52. ¿Va [este niño(a)] a alguna otra guardería o programa preescolar con regularidad?

No (Go to Question 54)
Sí

53. ¿Cuántas horas en total a la semana pasa [este niño(a)] en esas guarderías o programas preescolares?

______ horas a la semana

Tarea 3. Escenario hipotético – SALIR, ACCEDER

[La Tarea 3 tendrá lugar para todos los participantes en la Ronda 1, todos los participantes bilingües en la Ronda 2 y se eliminará de la Ronda 3 tanto en cuanto los problemas no prevalezcan]

[ENTREVISTADOR: Pare al encuestado antes de la pregunta 54.]