Appendix 4 Cognitive Interview Results

Appendix 4 NHES 2016 Cognitive Interview Results.pdf

National Household Education Survey 2016 (NHES:2016) Full-scale Data Collection

Appendix 4 Cognitive Interview Results

OMB: 1850-0768

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0768 can be found here:

Document [pdf]

Download: pdf | pdf

Appendix 4
Cognitive Interview Results
This appendix contains the following reports and memos that provide results from the
NHES:2016 cognitive interviews:
Cognitive Interview Findings for the Parent and Family Involvement and Early
Childhood Program Participation Surveys 2016: Final Report
Cognitive Interview Findings for the Adult Training and Education Survey (ATES) 2016:
Draft Report
Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews: Memo
Preliminary Findings from the 2015 Spanish Cognitive Interviews: Memo

Cognitive Interview Findings
for the Parent and Family
Involvement and Early
Childhood Program
Participation Surveys 2016:
Final Report
Prepared for the National Center for
Education Statistics (NCES) by
Susan Cole, Raquel Gonzalez, Kaitlin Fronberg, and
Suzette Chavez

May 2015

Contents
1. Introduction ..........................................................................................................4
1.1 Cognitive Interview Methodology .......................................................................... 5
1.2 Sample ....................................................................................................................... 5
1.3 Data Collection and Analytic Approach ................................................................ 8
Data Collection .............................................................................................................................. 8
Data Analysis Methods ................................................................................................................. 9

2. Findings .................................................................................................................9
2.1 Screener Items .......................................................................................................... 9
2.2 PFI Items ................................................................................................................. 10
2.2.1 Child’s Schooling ......................................................................................................... 10
School Assignment ......................................................................................................................... 10
Advanced Placement ....................................................................................................................... 11
Internet Courses .............................................................................................................................. 12
Homeschooling ............................................................................................................................... 14
Other Items in the Child’s Schooling Section ................................................................................. 17

2.2.2 Child’s Health............................................................................................................... 18
Services ........................................................................................................................................... 18
Other Items in the Child’s Health Section ...................................................................................... 20

2.2.3 Child’s Background ...................................................................................................... 20
Joint Custody .................................................................................................................................. 20
Language at Home ........................................................................................................................... 21
Other Items in the Child’s Background Section .............................................................................. 22

2.2.4 Your Household ........................................................................................................... 22
Other Items in the Your Household Section ................................................................................... 24

2.2.5 Child’s Family .............................................................................................................. 25

2.3 ECPP Items ............................................................................................................. 25
2.3.1 Finding and Choosing Care for Your Child ................................................................. 25
Difficulty Finding Child Care ......................................................................................................... 25
Importance of Reasons for Choosing Child Care ........................................................................... 26
Ratings from a Website ................................................................................................................... 26
Early Childhood Program Curriculum ............................................................................................ 28
Other ECPP Items ........................................................................................................................... 29

Appendix A……………………………………………………………………………...................... 30

1. Introduction
In 2016, the National Household Education Surveys Program (NHES) will field the Parent and
Family Involvement in Education Survey (PFI) and the Early Childhood Program Participation
Survey (ECPP). This report summarizes the findings from cognitive interviews conducted to test
and revise a subset of items from the PFI survey based on potential issues with a few items that
were observed in the 2012 collection and the 2014 Feasibility Study. This report also
summarizes findings from a series of new and revised ECPP items that cover topics suggested by
the Office of Nonpublic Education (ONPE) and the Policy and Program Studies Service (PPSS)
in consultation with the Office of Early Learning. The report also includes findings from
cognitive testing of the household screener.
In order to reduce the respondent’s cognitive and time burden, the interviews included only the
sections of the NHES questionnaires that contained items requiring testing. The objective of the
cognitive interviews in 2015 was to identify and correct problems of ambiguity or
misunderstanding in item wording that were evident from NHES:2012 data. The research was
iterative, in that item wording and format design changed during the testing period in response to
early findings. The cognitive interviews should result in a set of questionnaires that are easier to
understand and therefore less burdensome for respondents, while also yielding more accurate
information.
A total of 67 items underwent cognitive testing, including 60 items from the PFI survey and 7
items from the ECPP survey. The items from the PFI survey that were tested included items
about the following topics:









School assignment
Advanced Placement courses
Internet courses
Homeschooling
Joint child-custody arrangements
Language spoken by the child at home
Your household
Child’s family

The items from the ECPP questionnaire that were tested in cognitive interviews included items
about the following topics:




Difficulty finding child care
Importance of reasons for choosing child care
Early childhood program curriculum

The selected survey items were tested with members of the appropriate target respondent group
to determine their understandability, the need for revision, and for retention in or exclusion from
the surveys. The target respondent groups are described in section 1.2.
Interviews were conducted between January 20 and March 2, 2015.

1.1 Cognitive Interview Methodology
Each cognitive interview lasted approximately one hour and was conducted using primarily a
“think-aloud” approach with concurrent probing as needed, based on a structured protocol. The
methodology was developed by a senior researcher at AIR in consultation with NCES and
drawing on best practices and methods from cognitive science (see Appendix A for the protocol).
These interviews were designed to identify problems of ambiguity or misunderstanding in item
wording. The interviews were conducted by interviewers who received training in the cognitive
interview technique and in the interview protocol by a senior researcher at AIR. Researchers
from AIR and NCES listened to a subset of interviews and provided guidance and feedback
throughout the study.
During each interview, the study participant was welcomed by the interviewer and led to an
interview room. The participant was then told that the purpose of the interview was to help
answer questions about how people respond to items on schooling. All participants were assured
that their participation was voluntary and that their answers would be used only for research
purposes and not disclosed, or used, in identifiable form for any other purpose except as required
by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the
think-aloud process (see below) and modeled a think-aloud practice with a sample question.
Participants were then given a paper copy of the household screener and questionnaire to fill out.
The interviews were conducted using “concurrent probing,” a technique in which interviewers
asked respondents to “think aloud” as they answer the survey items. The cognitive interviews
also included a verbal probing component, conducted after completion of the think-aloud portion
for a given questionnaire. The verbal probes included a combination of pre-planned, itemspecific questions identified before the session as important and ad hoc questions that the
interviewer identified as important from observations during the interview, such as clarifications
or expansions on points raised by the participant.

1.2 Sample
Cognitive interview respondents were recruited through local recruitment firms in San Mateo,
CA, the Washington, DC metro area, and Waltham, MA and through additional interviewer
outreach to homeschooling parents. Respondents were given a $40 American Express gift card
for their participation. The recruitment criteria included:


15 parents or guardians of a child enrolled in public school and 7 in private school;









12 parents or guardians of a child enrolled in grades K-5 and 7 enrolled in grades 6-12;
12 parents or guardians of a child who is in a joint custody arrangement;
12 parents or guardians for whom English is their second language but who can read
English fluently;
6 parents or guardians of a child who is homeschooled;
9 parents or guardians who also have a child or children ages 0 to 5 and not yet in
kindergarten and also have a care arrangement for the child;
6 parents or guardians of a child who receives instruction from school over the internet;
and
10 parents or guardians with a high school completion or less education.

Note: The sum of the categories listed is greater than the total because several respondents are expected to
fall into multiple categories.

A total of 51cognitive interviews were conducted across the three locations. We sought to recruit
a diverse set of respondents; however the respondent characteristics reported during recruitment
did not always match the respondent’s reported characteristics during the cognitive interviews.
Table 1 provides more detailed information on the recruitment and interview reported
characteristics of the respondents.

Table 1. Number of participants in PFI/ECPP cognitive interviews, by selected recruitment
and interview characteristics

Total Respondents1
Child’s Grade
K ‐ 5
6 ‐ 12
Type of School2
Public
Private
Homeschool3,4
English as a Second Language1
Yes
No
Highest Degree
High School Diploma or Less
More than just High School
Joint Custody Arrangement5
Yes
No
Internet Instruction for Credit6
Yes
No
Second Child Under 5
Yes
No
Location
DC
CA
MA

Reported Characteristics
from Recruitment
51

14
32

Reported Characteristics
from Cognitive Interviews
51

14
36

36
12
5

38
11
6

8
43

7
43

5
46

6
44

11
40

10
43

10
41

8
42

9
42

16
25
10

9
42

16
25
10

1One respondent only filled out the ECPP items on the questionnaire as he/she did not have a second child K‐
12; n=50 for the PFI items
2 Four respondents also indicated that their child attended a public or private school.
3 Two respondents were former homeschoolers and filled out the screener and questionnaire as if they were
still homeschooling their child.
4 One respondent believed that “tutoring” was a form of homeschooling.
5 Some respondents were confused about the meaning of joint custody.
6 For Internet courses, there was initially a discrepancy between the language on the recruitment screener
and the survey.

Table 2 provides information on the total number of respondents who were scheduled for an
interview, the number of interviews that were completed, and the cancellations and no shows by
demographic characteristics.
Table 2. Number of interviews scheduled, completed, cancellations and no shows, by
demographics
Interviews
Interviews
Cancellations No Shows
Demographics
Scheduled
Completed
Child’s Grade
29
14
8
K ‐ 5
59
36
10
6 ‐ 12
Type of School
69
38
12
Public
16
11
3
Private

11
5
6
Homeschool
10
7
1
English as a Second Language
6
1
High School Diploma or
Less
21
10
3
Joint Custody Arrangement
16
8
4
Internet Instruction for Credit
16
9
5
Second Child Under 5
Location
49
16
18
DC
34
25
3
CA
12
10
1
MA
95
51
22
TOTAL

7
13
19
2
0
2
7
8
4
2
15
6
1
22

Note: The sum of the categories listed is greater than the total because several respondents fall into multiple
categories.

1.3 Data Collection and Analytic Approach
Data Collection

Respondents were asked to complete a paper version of the screener and questionnaire; while
doing this, they were asked to “think aloud” when responding to the items. After the respondent
completed an item of interest, the interviewer asked the respondent to stop and asked probing
questions about the item as needed. If the respondent had a problem or issue with items other
than the items of interest, the interviewer asked probing questions about those items. While the
respondent filled out the screener and questionnaire, the interviewer took handwritten notes in an
interview protocol that contains the items and probes for the cognitive interview. The screener
and questionnaire were kept by the interviewer as a record of the respondent’s responses to the
items. Once the interview was completed, the interviewer entered notes in an electronic note
template. This template is a standardized format for qualitative data entries, which allowed the
analysis team to process and analyze the data in NVivo, a qualitative data processing software

system. The interviewer then saved the interview notes and an audio recording of the interview
in a secure server for analysis.
Data Analysis Methods

Once the interview notes were entered, a senior researcher checked the data to ensure that the
most accurate information was included about the errors, issues, and concerns that respondents
had about the items in the questionnaire and screener. After the interview notes were checked for
quality, all items of interest as well as any items that respondents had difficulty answering were
coded into the following categories:
a.
b.
c.
d.
e.
f.
g.

No problems for questions of interest
Respondent was hesitant in answering the question
Respondent self-corrected his or her answer
Respondent made an item or survey suggestion
Skip problem (questionnaire navigation)
How the respondent interpreted an item term
Special situation (when a respondent responds to an item differently than expected based
on a special situation )
h. Response problem or issue
i. Respondent had difficulty understanding the survey formatting
ii. Respondent was unable to find a response that matched his or her experience
iii. Respondent had difficulty understanding the question
iv. Respondent did not include a response
v.
Respondent misinterpreted what the question was asking
vi. Respondent had trouble understanding what a specific term meant
vii. Other problems or issues
These categories allowed the analysis team to determine the types of problems that each item of
interest has. Once we were able to determine the item-by-item issues, we were able to do a
higher level analysis of topic-level problems.

2. Findings
2.1 Screener Items
At the beginning of the cognitive interviews, all respondents completed the National Household
Education Survey household screener. Below we report on two items of interest (month and year
of birth and school enrollment) and discuss a common skip error.
For the item on the month and year of birth, the majority (n=45) of respondents did not have any
difficulty reporting the month and year of birth for household members. Only a few respondents
struggled to remember the year of birth for members of the household. In most of these cases, the
respondent had some difficulty recalling the year of birth for a child, but subtracted his/her

child’s age from the current year to identify the year of birth. Only two respondents were unable
to recall the year of birth for adult members of the household (i.e., a mother-in-law’s and
respondent’s parents’ year of birth).
For the item that asks if household members are currently homeschooled or enrolled in school
(including college, university or vocational school), most participants were able to answer
without any problems. In a couple of instances respondents were unsure what option to select for
their young children (i.e., a one year old and 10 month old) who are in child care. One
respondent selected “public or private school, or preschool” and the other respondent left the
item blank.
The most common problem in the household screener was a skip error. If the respondent reports
that a household member is “not in school”, the survey instructs the respondent to go to the next
household member. If a household member is in school, respondents should proceed to the next
item that asks for the person’s current grade or equivalent. Half of the respondents failed to skip
out of the current grade or equivalent item for household members not currently enrolled in
school. These respondents selected either “none of these” for current grade or equivalent or the
household member’s highest level of schooling.
None of the items on the screener were changed during cognitive interview testing.

2.2 PFI Items
2.2.1 Child’s Schooling
School Assignment

The item on a child’s school assignment was to be completed only by respondents whose
children attend public schools. Table 3 lists the three versions of this item that were tested in
cognitive interviews.
Table 3. Wording changes for item on school assignment
Original Wording
Is it his/her regularly‐
assigned school? (No/Yes)

Wording Change
Is it his/her district‐assigned
school? (No/Yes)

Final Wording
Is it his/her district‐assigned
school?
A district‐assigned school is the
school that your local public
school district told you that this
child can attend, based on the
location of your residence.
(No/Yes)

The original item that asks about “his/her regularly-assigned school” was tested by 12
respondents. After respondents answered the item, the interviewer presented the respondent with
an alternative item (“his/her district assigned school”). During testing, the two items were

switched and 13 respondents responded to the item on “his/her district-assigned” school and
were presented with the item on “his/her regularly-assigned school” as the alternative. For the
remainder of the cognitive interviews (n=26), the item was changed to include a definition of
“district-assigned” and the item with the original wording was no longer tested.
Although many respondents understood the initial wording (“regularly-assigned school”), some
respondents misinterpreted “regularly” to mean daily – a school the student goes to everyday
during the week. Most respondents did not have a problem with the term “district-assigned,” and
seemed to prefer this over “regularly-assigned” because they were more familiar with the term.
They described it as the closest school to the home or the school assigned to a student based on
address/geographic zone.
For at least one respondent, interpretation of “district-assigned” and “regularly-assigned” was
informed by local policy. One respondent, based in California, responded “Yes,” to “districtassigned school,” and reported that California education code permits parents to indicate a
preference for which school his/her child will attend within the district regardless of where the
family resides within the district. The respondent applied to a preferred high school and was
allowed to enroll the child in that high school, and therefore, reported that the school was
district-assigned. However, when asked if the respondent’s child attended the “regularly assigned
school,” the respondent reported ”No” – this school was not the child’s regularly assigned
school, but rather the preferred school. Adding a definition seemed to be helpful for respondents.
Five respondents completed this item even though they should have skipped out of it since their
child attends a private school.
Advanced Placement

One item on Advanced Placement (AP) enrollment was tested in cognitive interviews. This item
was tested by all 50 respondents who completed the PFI items. The response option “Does not
apply” was removed during testing (See Table 4 below).

Table 4. Wording changes for Advanced Placement (AP) item
2012 Wording
Is he/she currently
enrolled in advanced
placement classes?
(No/Yes/Does not
apply)

Original Wording on
2016 Survey
Is he/she currently
enrolled in any high
school Advanced
Placement (AP) classes?
Advanced Placement is a
program that offers
college‐level courses to
high school students,
with the option for
students to take AP
exams to earn college
credit. (No/Yes/Does
not apply)

Wording Change

Final Wording

Is he/she currently
enrolled in any high
school Advanced
Placement (AP)
classes?
Advanced Placement is
a program that offers
college‐level courses to
high school students,
with the option for
students to take AP
exams to earn college
credit. (No/Yes)

Of the 12 respondents who responded to the original wording of the item, seven did not have any
difficulty answering the item. Some respondents whose children attend high school were
confused as to whether they should respond “No” or “Does not apply.” Examples include one
respondent who responded “No” stated that her child took an AP course the year before, but that
the child’s school did not offer AP courses in 11th grade. Another respondent who marked “Does
not apply,” misinterpreted the item and believed that elementary school students could take AP
classes. Another respondent could not tell the difference between AP and IB courses, and, lastly,
one respondent did not understand the meaning of Advanced Placement.
“Does not apply,” was removed as a response option as it was determined that sufficient data
would be captured for respondents who responded “No” for children not in high school.
Additionally, respondents with high school students at schools that do not offer AP might select
“Does not apply.”
Of the total 38 respondents who responded to the final wording of this item, the majority did not
have any difficulty answering it, although a few were confused as to how to respond. One
respondent who incorrectly marked “Yes” for this item ended up revising the answer to “No”
when she reread the item and realized there was a difference between advanced classes and
Advanced Placement. Another respondent who responded “No” said that the term was “foreign”
to him or her, although the respondent attributed that to the child being in 9th grade.
Internet Courses

Three items of interest on courses taken over the internet were initially tested by respondents.
The first item in the table below was tested by all 50 respondents, and the other two items were
tested by 11 respondents who marked “Yes” in the first item. The following wording changes
were made to two of the items and one item was deleted during testing.

Table 5. Wording changes/deletions for items on courses taken over the internet
Wording in 2012

Original Wording on
2016 Survey
Some students take
Some students take
school‐related courses school‐related courses
over the Internet. Is
for credit over the
this child receiving
Internet. Is this child
any instruction this
receiving any
way?
instruction this way?
(No/Yes)

Wording Change

Is this child taking any
school‐related courses
online instead of in‐
person with the
teacher?
Do not include courses
that use the Internet
only for selected
assignments? (No/Yes)
Is that instruction
Is that instruction
Is that instruction
provided by any of the provided by any of the provided by any of the
following places?
following places?
following places?

Mark all that apply.
Mark all that apply.
Mark all that apply.
Your local public
Your local public
Your local public
school
school
school
A charter school
Another public
school
A private school
A college,
community college, or
university
Someplace else—
Specify:

Final Wording
Is this child taking any
school‐related courses
online instead of in‐
person with the
teacher?
Do not include courses
that use the Internet
only for selected
assignments? (No/Yes)
Is that instruction
provided by any of the
following places?

Mark all that apply.
Your local public
school

Your state

A charter school

Another public
school
A private school

A college,
A college, community
community college, or college, or university
university
Someplace else—
A website
Specify:

Someplace else—
Specify:

What is the main
Deleted
reason this child took
school‐related courses
over the internet?

Another public
school
A private school
A college, community
college, or university
Someplace else—
Specify:

Deleted

Although most respondents understood the initial wording “school-related courses for credit over
the Internet” in the first item, seven respondents misinterpreted the item, including four who
incorrectly responded “Yes” on the questionnaire. These responses included materials, tests, and
homework at the child’s school that are accessible on the Internet, extra credit for courses done
at home, extra coursework done at school for additional grades, or the completion of a math
course on-line over the summer. One respondent who responded “No,” interpreted “Internet

courses” to mean reading, spelling, or alphabet games, and said: “it could be anything, it could
be listening to some songs over the Internet.”
The wording was changed on this item because of the misinterpretations and also because the
intent of the item. After the wording was changed and the definition was added to the initial
item, many respondents correctly responded, although three respondents misinterpreted the item,
even when they correctly responded “No.” One thought the item was asking if the student is
advanced enough to be taking online college courses, and another referred to courses outside of
school, for example, tutoring for a calculus class. A third respondent showed some confusion in
his or her reasoning for not including an informal online typing class as an online course: “there
is no human being on the other end, the entire class is completely electronic, and he [the student]
does not receive a certificate or credit for it.”
Respondents suggested the following changes to this item to make it more comprehensible: 1)
Restructuring the sentence to read, “all-school related courses in person with the teacher” or, “are
any [courses] online,” and 2) replacing the wording “instead of” to “in addition to,” which might
lessen the chance of glossing over the question.”
On the second item of interest that asks who provided the internet courses, of the 11 respondents
who responded to the item, four respondents, hesitated to respond or misinterpreted the item
because they did not understand what the item was asking, or because they were not sure
whether to respond about the institution that pays for the course or the website used to facilitate
the instruction. Based on these cognitive interviews, it was decided to delete “the website” as a
response option for the remainder of the respondents.
After “the website” was deleted on this item, a couple of respondents were not sure how to
respond because courses were through a free on-line system or an academy affiliated with
homeschooling, and they were not sure of the funding source.
On the third item that asks what the main reason is that the child took school-related courses over
the internet, three respondents misinterpreted the item based on their misinterpretation of the first
item, for example, two respondents selected “improve basic reading, writing, math, or science
skills,” although the courses were either not for credit or for extra credit, and one respondent
selected “Other,” which corresponds to the completion of a math course on-line over the
summer. This item was eliminated because of conflict with the first item on internet courses.
Homeschooling

Initially, six items of interest on homeschooling were tested in cognitive interviews. On the 2012
item, there was an overreporting of homeschooling and it was suspected that some respondents
did not understand that the item was asking about formal homeschooling programs, and instead
reported on casual instruction at home. In the 2014 feasibility study, the number of hours the
child attended public or private school was added to the item, which allowed NCES to evaluate
the number of respondents who indicated that the child was homeschooled but who likely

misinterpreted the item. A majority of 2014 respondents who indicated that their child was
homeschooled also reported that the child was attending a public or private school full-time.
Based on these issues, it was decided to test a series of homeschool items on the 2016 PFIEnrolled survey.
The items tested in this section on homeschooling were:









Some parents decide to educate their child at home rather than send them to a public or
private school. Is this child being schooled at home instead of at school for at least some
classes or subjects? (No, Yes)
Which of the following statements best describes your homeschooling arrangement for
this child?
o This child is homeschooled for all classes or subject areas
o This child is homescooled for some classes and subject areas and also attends a
public or private school
o This child is not homeschooled. This child attends a public or private school for
all classes or subject areas
How many hours each week does this child usually go to a public or private school for
instruction? Do not include time spent in extracurricular activities. [__] Hours
There are many different reasons that parents choose to homeschool their children. Did
your family choose to homeschool this child because: (Mark X ONE box for each item
below.) (Responses a-j)
Of the reasons your family chose to homeschool this child, which one would you say is
the most important to you? Write the letter from question [above] for the most important
reason you chose to homeschool your child. [__]
Thinking about all years this child has been homeschooled, which of the following
subject areas has this child been taught during his or her home instruction Mark X all
that apply .(list of subject areas)
[New item added] Which of the following subject areas are being taught to this child
now? Mark X all that apply. (list of subject areas)

On the first item that asks if the child is being schooled at home, one respondent initially
responded “No,” and then changed the response to “Yes” after the probing questions, and said
that he or she is providing tutoring for the child and considers it to be homeschooling. This
respondent incorrectly completed all of the remaining items on homeschooling. On the item on
description of the homeschooling arrangement this respondent chose the response option: “this
child is homeschooled for some classes and subject areas and also attends a public or private
school.” On the item on the number of hours each week the child usually goes to school for
instruction, this respondent misread “hours each week” and wrote 9 hours and explained he or
she drops off the child at school at 7 a.m. and picks the child up at 3:30 p.m. On the item about
the subject areas the child has been taught during home instruction, this respondent selected

“Calculus” and “Chemistry or physics” because those are the subject areas for which the child
was receiving tutoring.
The majority of the other 49 respondents did not have any problems or issues with the first item
that asks if the child is being schooled at home, although a couple of respondents had difficulty
with key vocabulary in the item. One of these respondents defined homeschooling as educating
the child after school, although correctly responded “No,” and another respondent defined
“homeschool” as “a supplement to what [the child] is learning in school,” although also correctly
responded “No.” One homeschooling respondent made a skip error and did not continue to the
next item on homeschooling. At the end of the cognitive testing a decision was made to add a
downward arrow, similar to other arrows on the questionnaire, to direct the respondent to
continue to the rest of the items on homeschooling if they responded “Yes” to the first
homeschooling item.
The remaining homeschooling items were completed by five respondents: three current and two
former homeschooling parents, and a respondent who tutors the child and considers it to be
homeschooling. A couple of respondents had problems with the item on the description of the
homeschooling arrangement. The item on the number of hours each week the child usually goes
to school for instruction presented problems for two respondents. Two respondents were
confused on how to respond to this item, one because the online homeschooling program is
considered a private school and the respondent thought the item on the questionnaire was
referring to a “brick and mortar” school, and another respondent was not sure how to respond
about a homeschooling academy that was not traditionally public or private.
There were no problems on the item on the different reasons that parents choose to homeschool
their children. There were also no problems on the item on the most important reason the
respondent chose to homeschool the child.
There were no problems on the item about the subject areas the child has been taught during
home instruction. Four respondents recommended adding “physical education” and “health” to
the list of subject areas. Based on these recommendations, at the end of cognitive testing these
were added to the list of subject areas, although these items were not tested during this phase of
cognitive interview testing (see Table 6 below).
An additional item on subject areas being taught “now” was added during the cognitive interview
testing because there is not enough space to obtain every subject the child was taught in every
year that they were homeschooled. Overall, this item did not present any problems for
respondents. One respondent who had only started to homeschool the child this year inquired
whether the item is asking about today, this year, this semester, or this month, although
responded with the same answer as the previous item.

Table 6. Wording changes and new item on subject areas taught during home instruction
Original Wording
Thinking about all years the
child has been homeschooled,
which of the following subject
areas has this child been taught
during her home instruction?

Wording Change
Thinking about all years the
child has been homeschooled,
which of the following subject
areas has this child been taught
during her home instruction?

Final Wording
Thinking about all years the
child has been homeschooled,
which of the following subject
areas has this child been taught
during her home instruction?

Mark X all that apply.

[List of subject areas]

Physical education or gym

Health
Wording Change
Which of the following subject
areas are being taught to this
child now?

Mark X all that apply.

[List of subject areas]

Physical education or gym

Health

[List of subject areas]

Physical education or gym

Health
Final Wording
Which of the following subject
areas are being taught to this
child now?

Mark X all that apply.

[List of subject areas]

Physical education or gym

Health

New Item
Which of the following subject
areas are being taught to this
child now?

Mark X all that apply.

[List of subject areas]

Other Items in the Child’s Schooling Section

In this section, there were 19-21 other items tested by respondents. The majority of these items
were tested by all 50 respondents except for a few skip patterns. Of these items, respondents had
varying degrees of difficulty responding to the items as well as the instructions. Only 13 of the
50 respondents had no difficulty when completing these items.
Five respondents were confused with the formatting on the item asking the child’s current grade
or year of school, which has the response boxes for grade 1 through 12 under the kindergarten
response options. All five said that they expected the boxes designating their response to be
placed directly under the item text and were initially confused when it was the last response
option.
On the item asking respondents what type of school the child attends, three respondents who all
responded “private school,” continued to fill out the next two items relating to public school,
when they should have skipped to a later item.

Three respondents had difficulty responding to the item asking if the child attends a charter
school, and noted that they did not understand the meaning of “charter school.” Additionally
some respondents were confused about the difference between a charter school and a magnet
school when reading the definition in the item asking whether or not a respondent’s school
district let them choose what public school they want the child to attend. On this same item
several respondents were uncertain as to how to answer and noted that they have the option to
transfer schools in their district, however the district did not guarantee the transfer.
Three respondents did not follow the skip pattern on the item asking whether or not they had
considered other schools for the child. One of these respondents ignored the skip pattern and
completed the following item, one incorrectly followed the skip pattern, and the third respondent
turned the page and skipped three items instead of the item they were directed to.
On the item asking respondents about how many times teachers or school staff had contacted
their household, a couple of respondents referred to contact they had made with teachers at an
open house, and some could not understand why “very good behavior” and “very good school
work,” were included in the questionnaire when teachers only contact them about problems with
the child’s behavior or school work.
Several respondents had difficulty responding to the item asking about their child’s overall
grades across subjects. Several respondents selected responses other than “This child’s school
does not give these grades,” and reported that their child’s school had a 1-4 point system and did
not use traditional grading systems. Similarly, several respondents had difficulty determining an
average grade. One respondent wrote on the questionnaire, “all over the map” and did not select
one of the response options.
On the item asking about the child’s suspension and expulsion history, a few respondents did not
know how to interpret the meaning of “in-school suspension.” Two respondents inquired about
the meaning and suggested that it means that the student was to attend school but not attend
classes at all during school hours, and another respondent reported that it means that the child
could not attend certain activities such as Chorus.
Several respondents who had difficulty determining an average grade also had difficulty
responding to the item asking how the respondent would describe their child’s work at school
because their child excelled in some subjects and did poorly in others, and it was difficult for
them to mark only one response option, e.g., “above average,” across all subjects. .
2.2.2 Child’s Health
Services

Two items of interest on child’s health were tested in cognitive interviews. Only respondents
whose children received services for health conditions responded to these items. The following

wording change was made to one of the sub-items on “sources for where services were
provided” to be more parallel with other sub-items (see Table 7).
Table 7. Wording changes for item on sources for services provided
Original Wording
Are these services provided by
any of the following sources?
Mark X ONE box for each item
below.

Wording Change
Final Wording
Are these services provided by Are these services provided by
any of the following sources?
any of the following sources?
Mark X ONE box for each item
Mark X ONE box for each item
below.
below.

No/Yes
No/Yes

No/Yes

a. Your local school
a. Your local school
a. Your local school
district……………
district……………
district……………
b. A state or local health or
b. A state or local health or
b. A state or local health or
social service
social service
social service
agency………………………
agency………………………
agency………………………
c. A doctor, clinic, or other
c. A doctor, clinic, or other
c. A doctor, clinic, or other
health care
health care
health care
provider……………………………
provider……………………………
provider…………………………… d. This child’s private
d. This child’s private
d. A teacher or health provider
school………………………
school………………………
from a private
school………………………

Of the 12 respondents who responded to the item on the sources where services were provided,
two respondents were unsure whether to mark “Yes” for the sub-item “your local school
district,” reasoning that either the school or the district provided the service, or they were unsure
altogether whether the local school district provided the service their child received. Two
respondents indicated that the “Yes” and “No” response options tend to be reversed on other
surveys (with “Yes” before “No”). One of these respondent accidentally marked “No” on all of
the sub-items when he or she intended to mark “Yes.” One respondent incorrectly marked
“doctor or clinic” for the person who was providing services for a condition that was not listed
on the previous item.
On the second item about whether the services are provided through an IEP or services plan,
respondents provided varying responses for the meaning of IEP and services plan, or did not
understand the meaning of the terms. Two respondents defined “IEP” or “services plan” as
instances when children receive extra time to complete coursework or their assignments. One
respondent indicated uncertainly about the meaning of an IEP, but said that if he or she was
answering the survey at home, would look up the meaning online to confirm. One respondent
responded “No” to this item, but his or her explanation for the “504 plan” his or her child
received largely aligned with the definition of a services plan.

Other Items in the Child’s Health Section

Two other items on child’s health presented some problems, although they were not prevalent.
Of the 50 respondents who responded to the item on “conditions told by health and education
professionals,” two respondents noted issues with the item’s wording, citing that professionals do
not “tell” you that your child has one of the conditions; instead the “health or education
professional” recommends that you seek testing for your child to see if they have one of the
conditions listed on the item. Two respondents had difficulty responding to this item because
they were uncertain about the timeframe. One respondent initially marked sub-item “a specific
learning disability” then went down the list and identified sub-item “attention deficit disorder,
ADD or ADHD” as more fitting for the child’s condition and removed selection of “a specific
learning disability.” It was decided to move the sub-item “a specific learning disability” from the
top of the response options to just after ADD/ADHD.
There were also comprehension issues on this item. Two respondents confused sub-item
“orthopedic” with orthodontic matters. One respondent did not know the definition of
“orthopedic” altogether. Additionally, four respondents could not define or did not recognize
“Pervasive Development Disorder (PDD).
For the item “if the child is receiving services for his/her condition,” one respondent indicated
confusion about how to respond since the child no longer received services for the condition
marked on the previous item. Two respondents had problems with the skip patterns. One
respondent marked “Yes” and missed the skip pattern, citing the arrow as less conspicuous than
the box indicating a skip to the next set of questions. One respondent marked “No,” but read the
following items and marked “Yes” for the “IEP and services plan” item.
2.2.3 Child’s Background
Joint Custody

Two items of interest on joint custody were tested in cognitive interviews. No changes were
made to these items. The first item that asks if the child “lives at this address and another address
(for example, because of a joint custody arrangement)” was tested by all 50 respondents. The
second item of interest on where the child spends his/her time (“at this address,” “at another
address,” or “equal time at both addresses”) was asked only of 11 respondents who marked
“Yes” to the first item.
Although many respondents did not have any comprehension problems with the first item on
joint custody, some respondents had to re-read the item a few times before responding, and some
changed their response either from “No” to “Yes” or vice versa. Four respondents were not clear
on the meaning of “this” address, although assumed it meant their home address – this response
confusion could be a function of the cognitive interview setting outside of the respondent’s
home. Some were not clear on the meaning of “joint custody,” or whether they were asking
about one or two addresses. For example, one respondent who responded “Yes” said that she and
her husband are married and are the two people who are the parents of that child that have joint-

custody or joint decision making. Another respondent who responded “Yes” said that her child
goes to the father’s house on the weekends, although when asked what the item was asking, the
respondent said, “it doesn’t necessarily mean divorce; children could go over to grandparents. It
is asking if children spend time in another home.”
Four respondents described special situations where they had physical custody of a child, but
they also had legal custody arrangements. For example, where the child is supposed to spend
time with one parent but prefers not to go to that parent’s house, so instead spends all of the time
with the respondent, and the respondents responded “No” to the item. In a couple of instances,
respondents stated that their response would be different for the first part of the question, “Does
this child live at this address and another address” than the second part, “for example, because of
a joint custody arrangement.”
Two suggestions were made by respondents for the first item on joint custody: 1) to move the
item to the screener; and 2) to simplify the item to read: “Does your child have more than one
residency?” This respondent said that since a respondent might answer the item quickly, he or
she might “skip the residency component altogether.””
Of the nine respondents who responded to the second item of interest on joint custody that asks
where the child spends his/her time, a few responded incorrectly because they did not understand
the first item described above, and one responded “No” to the first item and should have skipped
the second item.
Language at Home
One item on what language the child speaks most at home was tested in cognitive interviews.
The majority of the 50 respondents who responded to this item found it relatively easy to answer.
Four respondents were confused by the formatting of the first two response options: “Child is not
able to speak,” and “English” and thought it was one option “Child is not able to speak English”
because there is a bracket around these two options to indicate that the respondent should skip to
another question, and the response boxes were faint. Respondents were looking for an English
only response and most located it during the think aloud, although one respondent marked
“English and another language equally” because he or she could not find the correct response.
The response boxes on this item were filled with white to make it clearer that these were two
separate response options. Four respondents were still confused with the new formatting and
thought the first two response options were grouped together. One respondent suggested adding
an arrow after each response option rather than the bracket around the two options.

Other Items in the Child’s Background Section
A few other items in the Child’s Background section presented problems for respondents. A
couple of respondents suggested changing “sex” to “gender” for the item on the child’s sex. A
few respondents were confused and frustrated why the item on Hispanic, Latino, or Spanish
origin is separate from the item on the child’s race, and suggested that there should be a Hispanic
option in the race item. Some responded “White” or left the item blank when responding “Yes”
to Hispanic, Latino, or Spanish origin.
Some respondents had problems with skip patterns in two other items. Five respondents who
responded that the child was born in the U.S. did not skip out of the next item on how old the
child was when he/she first moved to the U.S. In two instances, respondents incorrectly
responded to the item on being enrolled in ESL, bilingual education, or an English immersion
program when they should have skipped out after the previous item on the language the child
speaks most at home.
2.2.4 Your Household

Two items of interest on how many people live in the respondent’s household were tested in
cognitive interviews. The following wording changes and order were made to the two items (see
Table 8 below) during cognitive testing.

Table 8. Wording changes for the items on how many people live in household
2012 Wording
Including yourself, how
many total people live
in this household?

[____] people

Other than the parents
or guardians already
reported, how many of
the following people
live in the household
with this child?

Example: Brother(s) 2

Write ‘0’ if none.

This child’s…

Brother(s)
Sister(s)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent’s
girlfriend/boyfriend/pa
rtner
Other relative(s)
Other non‐relative(s)

Original Wording (and
Order) on 2016 Survey
How many of the following
people live in this
household with this child?

Do not include this child in
your answer.

Example: Brother(s) 2
Write ‘0’ if none.

Brother(s)
Sister(s)
Parent(s)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent’s girlfriend/
boyfriend/partner
Other relative(s)
Other non‐relative(s)
Enter the total number of
people living in this
household with this child
(This number should be
equal to the sum of a
through k above).

Do not include this child in
your answer.
[____] people

Wording Change [and
Order Reversed)
How many people live in
this household?

Include adults and children
who are temporarily away
from home (for example,
living in college housing) if
they have no other
permanent home.

[____]

Final Wording (and
Order)
How many people live in
this household?

Include adults and
children who are
temporarily away from
home (for example, living
in college housing) if they
have no other permanent
home.

[____]

How many of the following
people live in this
household with this child?

Do not include this child in
your answer.

Example: Brother(s) 2

Write ‘0’ if none.

Brother(s)
Sister(s)
Mother (birth, adoptive,
step, or foster)
Father (birth, adoptive,
step, or foster)
Aunt(s)
Uncle(s)
Grandmother(s)
Grandfather(s)
Cousin(s)
Parent’s
girlfriend/boyfriend/part
ner
Other relative(s)
Other non‐relative(s)

How many of the
following people live in
this household with this
child?

Do not include this child
in your answer.

Example: Brother(s) 2

Write ‘0’ if none.

Of the 50 respondents who responded to the item, three respondents had problems with the
original item. For who is in the household, problems included searching for an option for
“mother” and not including themselves in the count; and being uncertain how to respond about a
boyfriend who is at the house the majority of the time but who does not legally live in the
household. The latter respondent decided not to include the boyfriend.
Five respondents had difficulty responding to the count. Problems included respondents’
difficulty with the “a through k” language in the parentheses and reading “a” as an article, and
including the child in the count. A respondent claimed that the item had “a lot of words,” and “a
lot to read to ask a pretty simple question,” and suggested that the item should ask how many
people live in this household excluding the child, and then to break it down further below.
In the next round of testing, the item order was reversed and the exact wording from the
household screener was implemented, based on feedback from the initial cognitive interviews.
All the remaining respondents comprehended the revised first item “How many people live in
this household?”
The revised item “How many of the following people live in this household with this child?”
that only had a change from “Parent(s)” to “Mother (birth, adoptive, step, or foster)” and “Father
(birth, adoptive, step, or foster)” to match another item in the “Your Household” section, still
presented problems similar to what was reported on the original item. Four respondents included
the child in the counts, three respondents were uncertain whether to include some household
members, i.e., two children away at college, a friend of the child’s who was living with the
family for a period of time, and a brother-in-law who is a long-term houseguest. All three
decided not to include these people. One of these respondents suggested that more information
should be included in this item similar to the other household item about whether or not to
include long-term houseguests. One respondent made a suggestion that “it might be worthwhile
to indicate on the questionnaire that if you want the total number or people in this item to be the
other household item minus 1.”
Other Items in the Your Household Section

The majority of the respondents did not have any problems with the other items in the Your
Household section. The item on the total income of all persons in the household was confusing
for a few respondents because they did not know who in the household to include in the response
and what income to include, for example, child support or disability. The item on which
language(s) are spoken at home by the adults caused some confusion for a couple of respondents
because they thought it was referring to the language spoken with the child, or they incorrectly
did not include English when they were fluent in English. Respondents who lived in an
apartment or a converted garage found the item on whether “this house” is owned, rented or
occupied by another arrangement problematic, and one respondent suggested that the item
should say “house or apartment.”

2.2.5 Child’s Family

Two items of interest on the respondent’s education level were tested in cognitive interviews.
The majority of the respondents did not have any problems when responding to these items on
the highest grade or level of school that the respondent completed and whether he or she is
currently attending or enrolled in school. The most common problem for a few respondents was
that they were not sure if they should be responding about themselves or their child on both
items.
Additionally, on the first item a couple of respondents did not understand what “vocational
diploma after high school” means, and one of these respondents actually had a “vocational
certificate” although responded “high school diploma or equivalent (GED)” on the questionnaire.
On the second item, a couple respondents did not initially see “job training” at the end of the
item, and one respondent was confused because the item is asking two questions (whether
respondent is in school or receiving vocational education or job training) and suggested that it
would be more clear to ask for the “student status.”
No changes were made to these items during cognitive testing.

2.3 ECPP Items
2.3.1 Finding and Choosing Care for Your Child

Respondents with children age five or younger completed additional items related to finding and
choosing care for a child. Three of these are items of interest and are discussed below.
Difficulty Finding Child Care

One item of interest on the primary reason for difficulty finding child care was tested in
cognitive interviews. This item was tested by the five respondents who had indicated in the
previous item that they faced some difficulty finding the type of child care or early childhood
program they wanted for their child. The following wording changes were made to the item
during testing (see Table 9 below).

Table 9. Wording changes for the item on primary reason for difficulty finding child care
Original Wording
What was the primary reason
for the difficulty finding care?

Cost
Location
Quality
Lack of open slots for new
children
Needed a program for
children with special needs
Other reason: Specify

______________________________

Wording Change
What was the primary reason for
the difficulty finding care?

Mark X ONE only.

Cost
Location
Quality
Lack of open slots for new
children
Needed a program for children
with special needs
Other reason: Specify

______________________________

Final Wording
What was the primary reason for
the difficulty finding child care
or early childhood programs?

Mark X ONE only.

Cost
Location
Quality
Lack of open slots for new
children
Needed a program for children
with special needs
Other reason: Specify

______________________________

Of the two respondents who responded to the initial item, one did not have any difficulty
responding. The other respondent had difficulty understanding whether or not it was possible to
mark more than one option and marked several options for this item including “Cost,” “Quality,”
and, “Lack of open slots for new children.”
After the instructions “Mark X ONE only” was added to the item, the remaining respondents
correctly responded to the item without any confusion. In the final version, the wording was
changed to mirror the language of another item on choices for child care “or early childhood
programs” where the respondent lives.
Importance of Reasons for Choosing Child Care

One item of interest asks how important it was in choosing the childcare arrangement or program
where their child spends the most time. Five new sub-items were tested in cognitive interviews
and they are described below.
Ratings from a Website
Of the nine respondents who answered this sub-item, all but one had no difficulty understanding
the item or finding a response that matched their experience. One respondent did not fill out this
item and was uncertain whether or not the question was referring to the child care provider’s
website or to an external website.

Recommendations from friends and family
Of the nine respondents who answered the sub-item on how important recommendations from
friends and family were in choosing a childcare arrangement or program, none of the
respondents had difficulty understanding the question or selecting a response.
Accreditation of care provider
Two versions of this sub-item were tested on how important accreditation of the care provider
was in choosing a childcare arrangement or program (see Table 10 below).
Table 10. Wording changes for sub-item on importance of accreditation of care provider
Original Wording
Accreditation of the care provider (e.g. from
the National Association for the Education
of Young Children or the National Early
Childhood Program Accreditation)?
Not at all important
A little important
Somewhat important
Very important

Wording Change/Final Wording
Accreditation of the care provider?
“Accreditation" is another way to judge the quality of a
child care program. Family child care homes and child
care centers can choose to get accredited by a child
care accrediting organization, such as the National
Association for the Education of Young Children or the
National Early Childhood Program Accreditation.
I do not know about the accreditation of my child’s
care provider.
Not at all important
A little important
Somewhat important
Very important

The original item was tested by four respondents; three of these respondents were unsure what
accreditation was. Based on these cognitive interviews, the item was changed to add a definition
of accreditation and the following response option “I do not know about the accreditation of my
child’s care provider.” The revised version of the item was tested by five respondents. Adding
the definition seemed to be helpful for respondents to better understand what the item was
asking; several were able to better articulate the meaning of accreditation.
Religious orientation of the program
Several of the nine respondents who answered this sub-item on how important the religious
orientation of the program was in choosing a childcare arrangement interpreted this item to mean
that the child would attend a program run by a church or religious group. Respondents did not
appear to have difficulty understanding the item or finding a response.

QRIS
Three versions of this sub-item on how important a local Quality Rating Improvement System
(QRIS) was in choosing a childcare arrangement or program were tested in cognitive interviews
(see Table 11below).
Table 11. Wording changes for sub-item on importance of QRIS
Original Wording
Did not use a QRIS

Wording Change
I have not heard of a QRIS

Final Wording
I have not heard of a QRIS

Not at all important

Did not use a QRIS

Not at all important

A little important

Not at all important

A little important

Somewhat important

A little important

Somewhat important

Very important

Somewhat important

Very important

The original response options were tested by four respondents. All four respondents were
unaware what a QRIS was and chose “did not use a QRIS.” Based on these cognitive interviews
and to make the item more parallel to the accreditation sub-item, a response option, “I have not
heard of a QRIS” was added. Two respondents tested the item with this additional response
option, which both respondents selected. To streamline the response options, the option of “did
not use a QRIS” was removed and the revised item was tested. Respondents did not have any
problems with selecting from the final response options.
Early Childhood Program Curriculum

Three versions of this item of interest that asks about the type of curriculum offered at the early
childhood program were tested in cognitive interviews (see Table 12 below).

Table 12. Wording changes for item on early childhood program’s curriculum
Original Wording
Which of the following best
describes this program’s
curriculum?

Montessori

Special program emphasis‐
such as science or math
emphasis, performing arts,
talented or gifted preschool,
foreign language immersion,
etc.

Special education‐ primarily
serves students with
disabilities

The Creative Curriculum®

None of the above

Wording Change
Which of the following best
describes this program’s
curriculum?

Montessori

Special program emphasis‐
such as science or math
emphasis, performing arts,
talented or gifted preschool,
foreign language immersion,
etc.

Special education‐ primarily
serves students with
disabilities

Play‐based curriculum, such as
The Creative Curriculum® or
Reggio® curriculum

None of the above

Final Wording
Which of the following best
describes this program’s
curriculum?

Montessori

Another play‐based curriculum,
such as The Creative
Curriculum®, High Scope or
Reggio Approach

Special program emphasis‐
such as science or math
emphasis, performing arts,
talented or gifted preschool,
foreign language immersion,
etc.

Special education‐ primarily
serves students with disabilities

None of the above

This item of interest was tested by nine respondents. They were generally not familiar with The
Creative Curriculum on the original response option, although one respondent said that the child
attends a program with a play-based curriculum similar to The Creative Curriculum. Only a
couple of respondents were familiar with Montessori programs although not familiar with the
specifics of the curriculum. Respondents were also not familiar with the Reggio curriculum.
Based on these cognitive interviews the term “play-based” and another example of a play-based
curriculum were added to this response option.
The final response options were further revised since Montessori is also a play-based curriculum,
another example of a play-based curriculum was added, and the order of the response options
was changed to make the item clearer. Respondents were typically unaware of the curriculum
used in their child’s early childhood program.
Other ECPP Items

Respondents did not have any problems or issues with the other four ECPP items that were tested
in cognitive interviews.

Appendix A – Interview Protocol

2016 Parent and Family Involvement in Education Survey (PFI) and Early Childhood Program Participation
(ECPP) Survey Cognitive Interview Protocol
Interview ID:
Parent First
Name:
Age of Child:

Date of
Interview:
Interviewer

Also, this interview will be audio-recorded so researchers can review the recording later. Any information that refers to you or your family
personally, like your name or name of your child’s school, will not be included in our report, and your responses will be used for research purposes
only.
Remember, you are not being tested; we want you to tell us what you think about the survey.
If at any time you want to stop, just let me know.
[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]
Before we continue, do you have any questions about what I just said? [Answer any questions the parent may ask.]

IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER WITH A COPY OF THE FORM. ASK THE
RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY
QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.
IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE
While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you figure out your answers
to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the
answers you give me. To help you get started on thinking aloud, we are going to do a practice activity.
First, I will show you an example of how to think out loud. Then I will give you a chance to practice it.
QUESTION: How many windows are there in the house or apartment where you live?
[Model the think aloud process for the parent, including reading the question out loud. Elaborate on counting all of the windows in your house or
apartment and as needed you can make up pieces of information to emphasize the think aloud process e.g., “there is a door at the back of the house
and it has windows on it; I think I’ll count those as windows. In the living room there are windows with panes in them and I think I’ll count each pain
as a separate window...” When you are modeling, write down the number of windows in your house/apartment as you are thinking aloud.]
[Once finished the model practice think aloud] I’m going to give you a piece of paper with the question on it. I’m going to ask you to read out loud
the question and think aloud as you decide on your answer.

Give respondent handout with think aloud practice.
How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell
me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL.

REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB
THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.

Screener
ASK THE RESPONDENT TO FILL OUT
THE ENTIRE SCREENER. REMIND THEM
TO THINK ALOUD IF THEY ARE NOT
DOING SO.
Q3. What is the person’s month and year of
birth?
[If the respondent is having difficulty with the
month and/or year] How did you come up with
your answer?
Q5. Is this person currently in…?

How did you come up with your answer?
[If respondent chooses “homeschool”] What
does “homeschool” mean to you?
DO NOT PROBE SPECIFICALLY ABOUT
HOMESCHOOLING HERE. FOR THE
SCREENER WE ONLY WANT TO
UNDERSTAND HOW THEY ARE
INTERPRETING “HOMESCHOOL.”

PFI/ECPP Survey
Child’s Schooling
ASK THE RESPONDENT TO FILL OUT
THE SURVEY. ASK QUESTIONS AND USE
PROBES ONLY AS NECESSARY; REMIND
RESPONDENT TO THINK ALOUD
Instructions: It is not necessary to probe on
the instructions. Note if the respondent has any
problems if they read them.

Q1-Q2: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q3. Is it his/her district-assigned school? A
district-assigned school is the school that your
local public school district told you that this
child can attend, based on the location of your
residence.

[Ask only if responded to this question] How
did you decide on your answer?
What does “district assigned” mean to you?
How did you know what school your child was
going to attend? How do other parents in your
child’s district know which schools their child
will attend?
Q4-Q13: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Q14. Is he/she currently enrolled in any high
school Advanced Placement (AP) classes?

What were you thinking of when you answered
this question?
What do “Advanced Placement (AP) classes”
mean to you?
What grade-level are the students who usually
take AP classes in your child’s school district?
36

Q15-Q21: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Q22. Is this child taking any school-related
courses online instead of in-person with the
teacher? Do not include courses that use the
Internet only for selected assignments.

How did you decide on your answer?

Q23. Is that instruction [internet] provided by
any of the following places?

How did you decide on your answer?
[If chooses a response option or “Someplace
else:”] Tell me more about that. How does
your child access the classes? Where is he/she
when accessing the classes? Who provides the
class?
Q24: It is not necessary to probe on this
question. Let the respondent continue the think
aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Q25. Some parents decide to educate their
children at home rather than send them to a
public or private school. Is this child being
schooled at home instead of at school for at
least some classes or subjects?

In your own words, tell me what this question
is asking about.
How did you determine whether to answer yes
or no?
[If respond “yes,”] Do not probe on the
meaning of homeschool until Q30.
[If respond “no” and if not evident from think
aloud to previous questions] What does
“homeschool” mean to you?
Make a note if the respondent did not skip
properly, but do not probe on skip pattern.]
Q26. Which of the following statements best
describes your homeschooling arrangement
for this child?

How did you come up with this answer?
[Do not probe on the meaning of homeschool
until Q30. Check to see if respondent skipped
properly, but do not probe on skip pattern.]

Q27. How many hours each week does this
child usually go to a public or private school
for instruction? Do not include time spent in
extracurricular activities.

How did you come up with your answer?

[Check to see if respondent filled in this
question although they indicated that the child
is homeschooled.]
Q28. There are many different reasons that
parents choose to homeschool their children.
Did your family choose to homeschool this
child because:

How did you determine how to answer yes or
no to each of these questions [a-j]?
Tell me more what you were thinking when
answering these questions.
[Do not probe on the meaning of homeschool
until Q30.]
Q29. Of the reasons your family chose to
homeschool this child, which one would you
say is the most important to you?

How did you come up with your answer?
[If not evident from think aloud to previous
questions] What does “homeschool” mean to
you?

Q30. Thinking about all years this child has
been homeschooled, which of the following
subject areas has this child been taught
during his or her home instruction?

How did you decide what to mark?
How do you cover these subjects during home
instruction?
Is there anything missing?
39

Q31. Which of the following subject areas are
being taught to this child now?

How did you come up with your answer? Can
you tell me more about what the child is being
taught now?

Child’s Health
REMIND RESPONDENT TO THINK
ALOUD

Q32-Q35: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Q36. Are these services provided by any of the
following sources?

How did you decide on your answer?

Q37. Are any of these services provided
through an Individualized Education Plan
(IEP) or services plan?

How did you come up with your answer?
What does a “services plan” mean to you?

How did you come up with your answer?
In your own words, what is this question
asking?
[If answered No] What does joint custody
arrangement mean to you?

Q45. If yes, does this child spend most of the
time at this address, at another address or
equal time at both addresses?
How did you come up with your answer?

[If needed] Can you describe the arrangement
your child has?

Q46. What language does this child speak
most at home?

It is not necessary to probe on this question.
Note the respondent’s response, especially for
those who are parents of ELL students. Record
in your notes whether the respondent reads the
response options below the top two responses.

Q47. It is not necessary to probe on this
question. Let the respondent continue the think
aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Your Household
Q48. How many people live in this
household?

How did you come up with your answer?
Who are you including in the counts?
Q49. How many of the following people live
in this household with this child?

Tell me more about what you were thinking
when you answered this question?
Who are you including in the counts?
In your own words, what is this question
asking?
Q50-Q58: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents

Child’s Family
REMIND RESPONDENT TO THINK
ALOUD

Q59. What is the highest grade or level of
school that this parent or guardian
completed?

[Only if respondent chooses “high school
diploma or equivalent”, “vocational diploma
after high school”, or “some college, but no
degree”:]
How did you come up with your answer?
What does “vocational diploma after high
school” mean to you?

Q60. Is he or she currently attending or
enrolled in a school, college, university, or
adult learning center, or receiving vocational
education or job training?

In your own words, what is this question
asking?
If parent wants to discuss another child in K-12 (not the first listed on the screener), there can be an informal follow-up, but it should be
parent-directed- they can provide additional insight regarding how they might have answered some questions for another child if there is
time.

THE REMAINING QUESTIONS ARE FOR PARENTS WHO ALSO HAVE A CHILD UNDER AGE-5, NOT YET IN KINDERGARTEN,
AND WHO IS IN A CARE ARRANGEMENT.
Tell the respondent the following:
The remaining questions are about your other child who is under age 5, not yet in kindergarten and in a care arrangement. How old is this child?
What type of care arrangement does this child have? [Record this in your notes.]
Please continue and answer the remaining questions. Please remember to think aloud.

Finding and Choosing Care for your Child
REMIND RESPONDENT TO THINK
ALOUD
Q61-64: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q65. What was the primary reason for the
difficulty finding child care or early childhood
programs?

Please tell me more about how you chose your
answer.
Can you tell me more about that?
Q66. How important was each of these
reasons when you chose the child care
arrangement or program where this child
spends the most time?

66h. Ratings on a website?
How did you decide what to mark?
In your own words, what is 66h asking?
Could you provide some examples?
66i. Recommendations from friends and
family?
How did you decide what to mark?

In your own words, what is 66i asking?

66j. Accreditation of the care provider?
"Accreditation" is another way to judge the
quality of a child care program. Family child
care homes and child care centers can choose
to get accredited by a child care accrediting
organization, such as the National Association
for the Education of Young Children or the
National Early Childhood Program
Accreditation.

How did you decide what to mark?
In your own words, what is 66j asking?
[If definition seems confusing:] In your own
words, what is the text [point to definition]
telling you?

66k. The religious orientation of the
program?
How did you decide what to mark?
In your own words, what is 66k asking?
66l. Your city or state’s Quality Rating and
Improvement System (QRIS)?
How did you decide what to mark?
In your own words, what is 66l asking?
What does QRIS mean to you?

[If applicable:] How did you learn about the
QRIS? How did you use it? Do you remember
what it is called for your state or area—that is,
which QRIS did you use?
Q67. Which of the following best describes
this program’s curriculum?
How did you come up with your answer?
[If respondent chose Montessori:] Tell me
what Montessori means to you.
[If respondent chose Play-based curriculum:]
Can you tell me more about your response?
What does “play-based” mean to you?
Are you familiar with The Creative
Curriculum?
How about HighScope?
How about Reggio Approach?
Are there other play-based curriculum that you
are aware of?
[If respondent chose None of the above:]
How would you describe the type of
curriculum used in your child’s program?
Do you know the name of the curriculum used
in your child’s program?

FINAL QUESTION

Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to the lobby. Please give the respondent incentive
envelope [$40 gift card], and thank them for their time and effort.

Cognitive Interview Findings
for the Adult Training and
Education Survey (ATES) 2016:
Draft Report
Prepared for the National Center for
Education Statistics (NCES) by
Stephanie Cronen & Kirsten Manville

June 2015

Cognitive Interview Findings for the
Adult Training and Education Survey
(ATES) 2016: Draft Report

June 2015

Stephanie Cronen & Kirsten Manville

1000 Thomas Jefferson Street NW
Washington, DC 20007-3835
202.403.5000 | TTY 877.334.3499
www.air.org

Contents
Page
Introduction ......................................................................................................................................3
Methodology ....................................................................................................................................3
Sample..............................................................................................................................................4
Key Findings ....................................................................................................................................4
Appendix A. Cognitive Interview Protocol .....................................................................................9

Introduction
Beginning in January 2016, the National Household Education Survey (NHES) is expected to
field the first full-scale administration of the Adult Training and Education Survey (ATES). The
American Institutes for Research (AIR) is working with the National Center for Education
Statistics (NCES) and staff from numerous other federal statistical agencies to develop a strategy
for measuring the prevalence of subbaccalaureate credentials—in particular, industry-recognized
certifications, licenses, and educational certificates—and work-related training and education in
the United States. This multiphase research effort began in 2009 and includes the development
and testing of questions that will be included in the ATES. In Phase I, AIR contracted with
Shugoll Research to conduct focus groups and cognitive interviews with respondents holding
less than a bachelor’s degree who work in fields that may offer subbaccalaureate opportunities.
In Phase II, the Center for Survey Methods at AIR designed and conducted cognitive interviews
of potential survey items that may be used to measure certifications and licenses. In Phase III, we
extended this work in two stages—for the originally planned 2015 NHES administration and for
the 2016 rescheduled administration—to include measures of participation in and characteristics
of work-related training and education. In addition, we conducted further testing of revised
measures of credential attainment.
This draft report documents the key findings from the English language cognitive interviews
conducted in Phase III for the 2016 administration. The revised report will include key findings
from the Spanish language cognitive interviews that are currently being conducted.

Methodology
Each cognitive interview lasted approximately one hour and was conducted using primarily a
“think-aloud” approach with concurrent probing as needed, based on a structured protocol. The
methodology was developed by a senior researcher at AIR in consultation with NCES and
drawing on best practices and methods from cognitive science (see Appendix A for the protocol).
These interviews were designed to identify problems of ambiguity or misunderstanding in item
wording. The interviews were conducted by interviewers who received training in the cognitive
interview technique and in the interview protocol by a senior researcher at AIR. Researchers
from AIR and NCES listened to a subset of interviews and provided guidance and feedback
throughout the study.
During each interview, the study participant was welcomed by the interviewer and led to an
interview room. The participant was then told that the purpose of the interview was to help
answer questions about how people respond to items on schooling. All participants were assured
that their participation was voluntary and that their answers would be used only for research
purposes and not disclosed, or used, in identifiable form for any other purpose except as required
by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers explained the
think-aloud process (see below) and modeled a think-aloud practice with a sample question.
Participants were then given a paper copy of the questionnaire to fill out.

American Institutes for Research

Cognitive Interview Report for ATES 2016—3

The interviews were conducted using “concurrent probing,” a technique in which interviewers
asked respondents to “think aloud” as they answer the survey items. The cognitive interviews
also included a verbal probing component, conducted after completion of the think-aloud portion
for a given section of the questionnaire. The verbal probes included a combination of preplanned, item-specific questions identified before the session as important and ad hoc questions
that the interviewer identified as important from observations during the interview, such as
clarifications or expansions on points raised by the participant.

Sample
Cognitive interview participants were recruited through the recruitment firm Elliott Benson in
Sacramento and San Mateo, CA and by AIR staff in Washington, DC and Grand Rapids, MI. All
interviews were conducted in person at these same locations. Respondent received $40 for their
participation.
The majority of respondents were required to either have a professional certification, an
educational certificate, or other work-related training. A small number of additional cases were
included to represent the general population. Based on screening data, the number of respondents
within each target group was:1


Certification or license: 28



Certificates: 18



Work-related education or training: 22



General population: 2

AIR conducted a total of 48 cognitive interviews between March 3, 2015, and April 17, 2015.
The interviews were conducted in four rounds to facilitate revision and re-testing.

Key Findings
Findings are presented by survey section, and changes made to the survey to address response
issues are described below.

Items on Education
The follow-up item that asks for the field of study for highest level of education was difficult for
some respondents to follow, either because they did not see their field listed, or because they did
not notice the instructions to mark one field only. To address these issues, we simplified the label
for the two most problematic fields of study (healthcare and social studies, political science,
economics, or history) and underlined the word “one” in the stem:

These target group numbers sum greater to the total number of interviews completed because respondents could
have combinations of target criteria.

American Institutes for Research

Cognitive Interview Report for ATES 2016—4

“Which one of the following best describes the field of study for the highest level
of school you have completed? (Mark ONE only.)
The item intended to measure participation in adult basic education was eliciting false positive
responses, due to respondents’ generalization of the term “literacy.” Classes such as computer
literacy and college composition courses were being reported under this item, and people who
taught rather than took classes pertaining to literacy or reading were reporting participation.
Therefore, we underlined the word “your” in the stem and instructed respondents not to include
college-level classes:
Since leaving high school, have you taken any literacy classes to help improve
your reading? Do not include college-level classes.
The item designed to measure college enrollment was confusing to some respondents for several
reasons. There were two different time referents in the stem (“currently” and “after high school”)
and the item appeared after the question about literacy classes, which seemed to be creating an
order effect. Therefore, some respondents interpreted the question as being about remedial
classes. To address these issues, we moved the item to appear after the first two educational
attainment items, and deleted “after high school”:
Are you currently enrolled at a college, university, technical or trade school, or
other school?

Items on Certifications and Licenses
In prior administrations, follow-up questions referred to the most recent certification or license;
however, we found that most recent credentials were not always being reported due to low
salience; i.e., not all of these credentials were considered currently relevant or as the most
important credential to all respondents, particularly in fields where respondents are required to
obtain multiple credentials. For 2016, we therefore tried a new approach by asking about
respondents’ most important certification or license. Based on interview findings, we determined
that this approach is valid, and that most respondents can easily identify what they consider to be
their most important certification or license.
A familiar issue in this set of follow-up questions persisted, however; some respondents
conflated multiple certifications or licenses when responding. Each item already reminds
respondents to report on their most important certification or license; therefore, no further
revisions were made.
The item that serves as the primary indicator of the credential being a license rather than a
certification was difficult to answer for several respondents who were not sure if a government
entity issued their credential. For some, it was the parenthetical examples that were causing their
confusion; therefore, we deleted those examples and simplified the item to:
Is your most important certification or license required by a federal, state, or
local government agency in order to do that kind of work?
We also added a second question from the 2010-11 ATES to help identify false positive reports:
American Institutes for Research

Cognitive Interview Report for ATES 2016—5

Can your most important certification or license be revoked or suspended for any
reason?

Items on Certificates
The main certificate item continues to be a challenge, due to the overuse of the term among
education and training providers. The result of this overuse is confusion between certificates and
certifications and between educational certificates (based on a postsecondary program of study)
and certificates of completion (based on a single course or training). In the current round of
interviews, we found these same issues, but to a lesser extent. It appears to be helping to (a) ask
about certifications and licenses before certificates; and (b) ask about other types of certificates
(e.g., from an employer for completing a training) before asking about educational certificates.
Therefore, only minor modifications were made to the certificates items for readability purposes.
Specifically, we deleted some unnecessary text in the stem and response options to make the
item less cognitively burdensome.

Items on Work Experience Programs
One of the primary purposes of the 2016 interviews was to further test the relatively new items
on participation in and characteristics of work experience programs. Overall, we found that these
items performed well, with the exception of a moderate amount of over-reporting by respondents
who misinterpreted the items to pertain to more general on-the-job experience. We addressed this
to the extent possible by revising the stem of the main item to refer explicitly to types of work
experience programs:
Have you ever completed an internship, co-op, practicum, clerkship, externship,
residency, clinical experience, apprenticeship, or similar program?
Some respondents also had difficulty finding the field of study for their program. Therefore, we
made the healthcare field easier to find and added nursing assistant to the list.
Another relatively common area of overreporting was for respondents to report that their work
experience program was part of various types of schools’ education or training program. To get
respondents to better attend to the response options, we revised the stem to be more general and
the response options to be more specific:
Which one of the following best describes your last work experience program? (Mark ONE
only.)
It was not part of a formal education program
It was part of a high school program
It was part of a school program after high school and below an Associate’s degree
program
It was part of an Associate’s degree program
It was part of a Bachelor’s degree program
It was part of an advanced degree program or other program beyond a Bachelor’s degree

American Institutes for Research

Cognitive Interview Report for ATES 2016—6

General Issues
We identified two areas of response error across multiple survey sections of the English version
of ATES. The first type of error was respondents’ tendency to mark a response to only one item
among a list of items that calls for a response to each item in the list. We tried several variations
on item formatting and instructions, and ultimately chose to include the instruction “(Mark one
box for EACH ITEM below.)"
The second type of error we encountered across sections was unemployed respondents who
missed the “Not applicable, not currently working” response option that applied to them. To
increase the salience of this option, we moved it to appear as the first option rather than the last.

Summary
AIR conducted four rounds of cognitive interviews to test questions for their potential to measure
the prevalence of subbaccalaureate credentials and participation in and characteristics of workrelated training among adults in the United States. During the cognitive testing, the AIR team
found that some respondents had problems (1) understanding the purpose of several items, and
(2) locating the correct response option. To address these problems, the questions and response
options were simplified and re-ordered in some cases, and instructions were added or modified.
In addition, one new item was added to the certifications and licenses section to help identify
false positive reports.

American Institutes for Research

Cognitive Interview Report for ATES 2016—7

American Institutes for Research

Cognitive Interview Report for ATES 2016—8

Appendix A. Cognitive Interview Protocol
Interview ID:
First Name:

Date of
Interview:
Interviewer

STEP 1: WELCOME AND INTRODUCTION SCRIPT
Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is
suggested content with which the interviewer should be thoroughly familiar in advance. Text in
brackets [ ] presents instructions for the interviewer.
Hello, my name is and I work for American Institutes for Research. It’s nice to meet
you, thanks for coming to help us out today.
[Create small talk to build rapport with the respondent]
You are here today to help us out with a survey that we are working on for the U.S. Department of
Education. This survey asks about adults’ education and training experiences. I will ask you to
complete the survey as if you were taking a real survey. There are no right or wrong answers, and
it’s ok if you do not know some of the answers. As you are answering the questions I will ask you
about your answers. This will help us make the survey better and clearer for other adults like you.
To make sure we learn everything we can from this session, I am going to be taking notes while you
are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and
my colleague will also be observing our session today behind the glass or by video].
Also, this interview will be audio recorded so researchers can review the recording later. Any
information that refers to you personally, like your name or employer, will not be included in our
report, and your responses will be used for research purposes only.
Remember, you are not being tested; all we want is to learn how well the survey works.
If at any time you want to stop, just let me know.
[If, for any reason, the participant is no longer interested in participating, thank the participant for
his/her time and end the interview.]
Before we continue, do you have any questions? [Answer any questions they may ask.]
[Provide respondent with a copy of the informed consent form. Ask the respondent to read the
form (or read it to him/her, depending on respondent preference), answer any questions, and
have the respondent sign the form. Leave a separate copy of the form with the respondent.
If the respondent has consented to recording, start the recorder.]

STEP 2: THINK-ALOUD PRACTICE
While you fill out the survey I want you to talk about how you are answering the questions. Hearing
you talk about how you decide on your answers will help me understand how to make the questions
better. I am more interested in how you arrive at your answers than in the answers you give. To help
you get started on thinking aloud, we are going to do a practice activity. [Show an example by
asking yourself the question and answering it using ‘think-aloud’]
I’m going to give you a piece of paper with a question in it. I’m going to ask you to read out loud the
question and think aloud as you decide on your answer.
[Give respondent handout with think aloud practice.]

American Institutes for Research

Cognitive Interview Report for ATES 2016—9

How many glasses of water did you drink yesterday?
Please tell me what you are thinking while you are figuring out your answer.
Probe as needed for detail: How did you arrive at that answer? What were you visualizing when
you were deciding on that answer? Can you tell me more about what you are including in your
count?
When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about
your answers to the questions in the survey.
[Hand respondent a copy of the questionnaire and a pen or pencil.]

STEP 3: COMPLETION OF THE QUESTIONNAIRE
Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on
your answer. Please read aloud anything you would read to yourself if I were not here.
Remember that I am interested in hearing you talk about what you are thinking as you figure out
your answers to the questions.
[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY
ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR
QUESTIONS ARE REALLY HELPFUL.]

Useful Probes—Use only as needed when response is unclear or R appears confused or
unsure
Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking.
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Tell me what you are
thinking.
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old
question before answering this question]: Tell me what just happened. Tell me about what you
were just doing.

Education
STOP RESPONDENT AFTER Q5 – Probe on Q1-5

NOTES

ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY
Instructions: It is not necessary to probe on the instructions. Note if the
respondent has any problems if they read them.
Q1-Q4: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that

American Institutes for Research

Cognitive Interview Report for ATES 2016—10

indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q5.
Since leaving high school, have you taken any literacy classes to help
improve your reading? Do not include college-level classes.

“Your” underlined.
Probe change – to ask
what classes are literacy
classes.

(Ask everyone) What types of classes would you include as literacy classes?

Certifications and Licenses
STOP RESPONDENT AFTER Q15 – Probe on Q6-15
Q6-7: It is not necessary to probe on this question. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.

Q8: [Only probe if response to Q7 > 1.]
The next few questions ask about the certification or license that you
consider to be your most important. What is the name of your most
important certification or license?
What do the words “most important” mean to you in this question?
How did you decide which certification or license was your most important?
How easy or hard was it to choose your most important certification or
license?

Q9: It is not necessary to probe on this question. Let the respondent continue
the think aloud and only probe if respondent says something that indicates a
potential problem, or if there is confusion or an inconsistent interpretation
compared to other respondents.
Q10:
Is your most important certification or license required by a federal, state,
or local government agency in order to do that kind of work?
Who issued your certification or license?
(If R seems confused) How easy or hard was it to answer this question?
Q11-14: It is not necessary to probe on this question. Let the respondent

American Institutes for Research

Cognitive Interview Report for ATES 2016—11

continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q15. How useful has your most important certification or license been for
each of the following? (a-d)

[If respond to Q7 is > 1]: How easy or hard was it to only answer about
your most important certification or license for each question?
Q8 through 15
Tell me a little bit more about your most important certification or license—
in your own words, what was the process you went through in order to get
it?
Note to interviewers: What we’d like to find out is whether R really has a
certification or license, and whether they have one but also have a certificate
from a course or a program (a series of courses). Information about courses
or training they took to prepare, where they took it, whether they had to
pass a certification or license exam, where they took that exam, and who
issued the certification or license all provide good clues.
Keep in mind that someone might go to a school (or other organization) to
take course(s) to prepare for this type of credentialing exam, then take the
exam and have both (1) a certificate of completion (30a) or post-secondary
certificate (30d) from going through this preparation process as well as (2)
a certification and/or license issued after passing the credentialing exam.
Therefore, some Rs report something under both certifications and
certificates and it’s legitimate for them to do so. Other Rs just confuse the
two and report certificates under certifications and vice versa.
To help you understand the difference, a certification or license is
recognized in the R’s field as proof of qualification to do a particular type of
job, and usually requires an “official” exam that isn’t specific to a
particular school or other training provider—everyone in the state or
country would take the same exam, regardless of whether they took any
preparation courses or not. A state usually issues licenses, and professional
associations (like the American Association of Medical Assistants) usually
issue certifications for their particular field.
In contrast, a certificate simply indicates that the R has participated in some
type of education or training to learn content in an area. It does not
“authorize” them to practice in that area. A certificate of completion (30a)
is usually based on a single course and offered by a variety of providers, and
a post-secondary certificate (30d) is awarded by a school after someone has
completed a specific series of courses (program of study), just like a college
degree. It is considered a college credential, but generally takes less time to
get than an associate’s degree.
Note also that many colleges or other organizations partner with

American Institutes for Research

Cognitive Interview Report for ATES 2016—12

credentialing bodies to host their exams, so a R might take courses or go
through a certificate program at a community college and then take the
certification or licensing exam there. The school doesn’t “issue” the
certification, however; they are just serving as proctors for the state or
association who does issue it. This makes it a challenge for us to
differentiate between certifications and certificates, so we try to learn about
the educational process Rs went through as well as the type of exam they
took, if any.
IF RUNNING BEHIND OR R IS REPORTING SOMETHING OTHER
THAN LICENSES AND CERTIFICATIONS, MOVE THEM ON TO
CERTIFICATES SECTION.
OTHERWISE, STOP RESPONDENT AFTER Q24 – Probe on Q16-24
Q16-24 (if answer “Yes” to Q16 – probe): Tell me a little bit more about
your second-most important certification or license—in your own words,
what was the process you went through in order to get it?
Q16-24. Do you have another currently active certification or license? (if
answer “Yes” to Q16 – probe)
How did you decide which certification or license was your second most
important?
How easy or hard was it to choose your second most important certification
or license?

Q17 through 24
For questions 17 through 24, was it easy or hard to answer for just your
second most important certification or license?
Did you think about other certifications or licenses when you answered these
questions?
Q19. Is your second most important certification or license required by a
federal, state, or local government agency in order to do that kind of work?
Who issued your certification or license?

Q20-23: It is not necessary to probe on this question. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q24. How useful has your second-most important certification or license
been for each of the following?(a-d)
How easy or hard was it to only answer about your second most important

American Institutes for Research

Cognitive Interview Report for ATES 2016—13

certification or license for each question?

STOP RESPONDENT AFTER Q26 – Probe on Q25-29

Q25-29: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.

Certificates
STOP RESPONDENT AFTER Q38 – Probe on Q30-38
Q30a.
People sometimes earn certificates from an education or training program.
These are different from certifications or licenses. Do not include
certifications or licenses here. Have you ever earned any of the following
types of certificates?
a. A certificate for completing a training program from an employer,
employment agency, union, software or equipment manufacturer, or other
training provider.
(If R choose “yes”) Tell me a little more about what you’re reporting for
30a. In your own words, what was the process you went through in

order to get it?
(If R appears to be reporting same credential as in certifications/licenses)
How is this related to the certification or license you reported earlier?

Note to interviewers: Similar to what we’re trying to learn in the
certification and license section, we need to know more about how R got the
certificate, from what type of organization (get the name if you can), and if it
is the same thing or related to the certification or license reported earlier (if
applicable).
Q30d. People sometimes earn certificates from an education or training
program. These are different from certifications or licenses. Do not
include certifications or licenses here. Have you ever earned any of the
following types of certificates? d. A certificate—not a degree—for
completing a program at a community or technical college, or other
school after high school. Do not include teaching certificates or
college degrees.
(If R choose “yes”) Tell me a little more about what you’re reporting for
30d. In your own words, what was the process you went through in order to

American Institutes for Research

Cognitive Interview Report for ATES 2016—14

get it?(If R appears to be reporting same credential as in
certifications/licenses) How is this related to the certification or license you
reported earlier?
Q32. Who gave you your last post-secondary certificate?
Who gave you your last post-secondary certificate (get name)?
How easy or difficult was it to answer this question?
(If R chooses “another college or university” or “Someplace else” and it’s
not clear based on the name): Can you tell me more about where you got
your last post-secondary certificate?
Q33. About how many hours of instruction did you complete in order to
earn your last post-secondary certificate?
How did you determine your answer?
How easy or difficult was it to answer this question?
Q34: It is not necessary to probe on this question. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q35. To earn your last post-secondary certificate did you have to
complete…
Were there any other instructional requirements needed to complete this
program other than what is listed?

Q36: It is not necessary to probe on this question. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q37 Is your current job related to your last post-secondary certificate?

Question order changed.

How easy or difficult was it to answer this question?

Q38: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.

American Institutes for Research

Cognitive Interview Report for ATES 2016—15

Work-Based Learning Program
STOP RESPONDENT AFTER Q49 – Probe on Q39-49

Section name change –
from Work Experience
Program

Q39: Have you ever completed an internship, co-op, practicum, clerkship,
externship, residency, clinical experience, apprenticeship, or similar
program?

Revised question
wording.
Probe on this item
higher priority now.

(IF YES) Tell me about your program:
 What job was it for?


Could you describe a typical day in the program?



What type of organizations were involved in it and what were their
roles?



(If applicable) Was it related to the [certification/license/certificate]
that you reported earlier? (If yes) How?

Note to interviewers: Similar to the previous sections, we’d like to find
out if Rs are reporting something new here or something that they have
reported previously
Q40. (If applicable) If yes, What type of work was your last work-based
learning program for? (Mark ONE only.)

Can you tell me how you chose this category? How easy or hard was
it to find your answer on that list?
Were there any categories that are missing?
[Pay attention to R and if they want to select the category headers that
don’t have check boxes; probe if they are uncertain of where to mark their
response]
Q41-42: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q43 a-c. (If applicable) As a part of your last work-based learning
program did you…

Question wording
changed

(If yes to any of a-c): Can you tell me a bit more about the connection
between your training program and the classes, training, or studying
you did?
American Institutes for Research

Cognitive Interview Report for ATES 2016—16

Q44: It is not necessary to probe on this question. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q45. (If applicable) Which one of the follow best describes your last workbased learning program?

New response option
(3rd & 6th option);
Changed response order

Was it easy or hard to answer this question?

Please tell me about how you chose your answer for this one?

Q46. (If applicable) Did (or will) your last work-based learning program
help you earn a professional certification or license?

Can you tell me in your own words what that question is asking?
Q47-Q49: It is not necessary to probe on these questions. Let the
respondent continue the think aloud and only probe if respondent says
something that indicates a potential problem, or if there is confusion or an
inconsistent interpretation compared to other respondents.

Employment [If short on time these sections are not priority]
STOP RESPONDENT AFTER Q70 – Probe on Q50-70
Q50-Q67: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q68. (If applicable) Did you have a license that was required by a
federal, state, or local government agency to do this job?
In your own words, what is this question asking?

[Check for consistency, if they answered inconsistently between either
Q10/19/28 and Q68 then probe] You reported in (Q10/Q19/Q28) “yes/no”

American Institutes for Research

Cognitive Interview Report for ATES 2016—17

but you reported “yes/no” in Q68. Could you tell me the difference between
these two questions?

Q69-Q70: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.

Background [If short of time have them skip to Q82-84 after the Work-Based
Learning section]
Q71-81: It is not necessary to probe on these questions. Let the respondent
continue the think aloud and only probe if respondent says something that
indicates a potential problem, or if there is confusion or an inconsistent
interpretation compared to other respondents.
Q82. Do you have Internet access on a cell phone?
Please tell me about how you chose [no, yes] for this one.

Revised question
wording (changed
smartphone to cell
phone)

Q83. Do you have Internet access at home on a computer or tablet?
Please tell me about how you chose [no, yes] for this one.
[If not obvious from response], Where are you thinking about? [Find out
where “at home” was interpreted by R to be.]
Q84. How often do you use the Internet?
Please talk to me about how you got to your answer here.
[Look for indications that the response options are not inclusive enough/
adequate.]

FINAL QUESTION
Is there anything else you would like to tell me about this questionnaire that you haven’t had a
chance to mention?
(IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS

STEP 4. THANK YOU FOR PARTICIPATING
We are finished. Thank you for helping us. The information you’ve provided will be very helpful for
making this survey better.
Interviewer Instructions: Collect materials and escort them out of the interviewing room and to the
lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.

American Institutes for Research

Cognitive Interview Report for ATES 2016—18

LOCATIONS
Domestic
Washington, D.C.
Atlanta, GA
Baltimore, MD
Chapel Hill, NC
Chicago, IL
Columbus, OH
Frederick, MD
Honolulu, HI
Indianapolis, IN
Naperville, IL
New York, NY
Rockville, MD
Sacramento, CA
San Mateo, CA
Waltham, MA

International
Egypt
Honduras
Ivory Coast
Kyrgyzstan
Liberia
Tajikistan
Zambia

1000 Thomas Jefferson Street NW
Washington, DC 20007-3835
202.403.5000 | TTY 877.334.3499
www.air.org

Memo
Date:

May 20, 2015

To:

Sarah Grady, NCES

From

Danielle Battle and Meghan McQuiggan, AIR

Re:

Key Findings from the 2015 PFI/ECPP Homeschool Cognitive Interviews

Introduction
In 2016, the National Household Education Survey (NHES) will field the Parent and Family
Involvement in Education Survey (PFI), which contains a homeschool component for households
with homeschooled children. This report summarizes the findings from the supplementary
homeschool cognitive interviews conducted to test and revise a subset of PFI items based on
potential issues initially observed in the 2012 collection and the 2014 Feasibility Study. This
report provides supplemental information to the “2016 Parent and Family Involvement and Early
Childhood Program Participation Surveys Cognitive Laboratory Final Report” (Cole, Gonzalez,
Fronberg, & Chavez, 2015).
In 2014, the National Center for Education Statistics (NCES) conducted a feasibility study using
the NHES. Sampled households were first sent a screener survey, which asked individuals to
enumerate the members of their household, as well as answer a few questions about each person.
The responses to the screener items then determined if someone in the household was eligible for
one or both topical surveys: the Afterschool Programs and Activities (ASPA) survey and the
Adult Training and Education (ATES) survey.
One screener item asked respondents to report the enrollment status of each household member.
The response options included “public or private school, or preschool”; “homeschool instead of
school for some or all classes”; “college, university, or vocational school”; and “not in school”.
Depending on the version of the screener the household received, the homeschool option would
have been presented either first or second.
Cognitive interviewing and analysis of the feasibility study data revealed that the concept of
“homeschooling” may be unclear to some respondents. All households who received the ASPA
survey had indicated on the screener that the sampled child was enrolled in public or private
school or preschool. However, in the ASPA topical questionnaire, about 2 percent of sampled
children were reported to be homeschooled. If respondents selected “homeschool” on the topical
survey, they were also asked how many hours per week the child attended a public or private
school. Of those who reported that the child was homeschooled, 34 percent indicated the child
spent 25 hours or less per week in public or private school, suggesting that these children are
1

likely “true homeschoolers.” The other 66 percent, who reported 25 hours or more per week in a
traditional school setting, likely misinterpreted the meaning of “homeschooling” (Jackson,
McQuiggan, & Megra, forthcoming).
Although these findings only apply to a small percentage of NHES respondents, they suggest
that there is ambiguity over the term “homeschooling,” causing some parents of homeschooled
children to receive a topical survey intended for parents of enrolled children. As a result, NCES
decided to conduct a supplemental round of cognitive interviews specifically with parents of
homeschooled children. In these interviews, parents were asked to complete the PFI survey
intended for parents of children enrolled in public or private school (PFI-Enrolled) but which
contained a section about homeschooling. The goal of the interviews was to examine how
parents of homeschooled children would respond to questions about traditional “brick and
mortar” school environments, since homeschooling households that misreported on the screener
might receive this instrument in future NHES administrations.
Cognitive interview respondents were recruited through personal contacts and Craigslist in
Washington, DC (4) and an interviewer contact in rural Michigan (1). The homeschooled
children reported in the topical instruments ranged in age from seven to 18. When asked the
children’s grade equivalencies, two parents reported second grade, one reported sixth grade, one
reported ninth grade, and another reported twelfth grade. Three respondents reported that their
child attended a traditional school before homeschooling and two respondents reported their
child had always been homeschooled. All respondents indicated that their children were
homeschooled full-time. Respondents were given a $40 American Express gift card for their
participation.
The purpose of this memo is to provide key results from the five supplemental homeschool
cognitive interviews conducted using the PFI-Enrolled questionnaire. This memo summarizes
the main issues found, as well as provides recommendations for making these items clearer for
both parents of homeschooled and enrolled children. All questionnaire changes based on
interview findings are bolded in the item discussions below.
This memo highlights six key findings:


Screener: In total, respondents were able to navigate the screener questionnaire without
any problems. When probed on the meaning of “homeschool,” all respondents articulated
a clear understanding of this concept. Some parents noted that the idea of gradeequivalency may be more difficult for homeschooled children than enrolled children.



Traditional school items: Homeschool parents often verbalized that these questions did
not apply to them. At times, parents left items blank, and at other times, they satisficed by
answering based on their homeschooling situation.



Online instruction items: Multiple homeschool respondents did indicate that their child
participated in online courses, but they were often unable to find an instruction provider
among the response options provided that matched their child’s educational experience.



Homeschool items: Parents were able to easily answer questions about their child’s
homeschooling. The items about subjects taught during all homeschooling and taught
2

currently were the same for most respondents, depending on the flexibility of the child’s
curriculum. It may be difficult for parents to report on subjects taught “now” based on
how the child’s curriculum is delivered across a school year and how the term “now” is
interpreted.


Health, Child’s Background, Household Members, Child’s Family sections:
Respondents were able to navigate these sections without many problems.



Your Household section: This new section worked well, and respondents were able to
easily answer the new Internet items. All respondents indicated frequent Internet use,
including for homeschooling purposes, and reported access on smartphones, tablets, and
computers.

Key Findings
Screener Instrument
Overall, respondents encountered few problems with the screener instrument. On question 5,
which asks about the enrollment status of household members, two respondents initially marked
“homeschool” when reporting information about themselves. After they realized their mistake
and corrected their responses to “not in school”, both parents verbalized that they were
mistakenly thinking about their children when answering. It is likely that this pattern was due to
a priming effect in which respondents knew they were recruited as homeschool parents, which
impacted their responses and should not be reflected in the larger data collection.
After respondents selected “homeschool” for their children’s enrollment status, interviewers
probed on the meaning of homeschooling, and it was evident that these homeschooling parents
had a clear understanding of this concept. One respondent said his child is homeschooled
because “we elect not to send her to a public school or private institution and to educate her
ourselves”. Another said that his child is homeschooled because he is “enrolled in an accredited
program that we administer at home…most people consider it homeschool because we are the
principal educators.” One respondent added that homeschooling is a “global term” encompassing
a spectrum of arrangements and philosophies. He described how his children are enrolled in an
accredited and structured homeschooling curriculum, while others, sometimes called
“unschoolers,” practice more experiential learning. This respondent did note that he is unsure if
“unschoolers” would categorize their children as “homeschooled,” and suggested adding
language to clarify that “homeschooling” includes all curriculum-based, distance education, and
experiential learning methods.
For all household members enrolled in a school program, respondents were supposed to answer
question 6, which asked for the person’s current grade equivalent. One respondent noted that this
question was difficult for him to answer since his child does not progress through traditional
grades, but said he would write 2nd grade since she was seven years old. This respondent
suggested using “age appropriate grade” to make this language applicable to homeschooled
children. Another respondent commented that this could be challenging since some homeschool
instruction is self-paced, and a child may be on-grade for some subjects, while ahead or behind
3

grade-level for others. Overall, these suggestions should result in further testing of this item
wording and alternative item wording before used in data collection.

Topical Instrument
Child’s Schooling
The first half of this section was challenging for homeschool parents. Multiple respondents
expressed that these items seemed to apply only to children in public and private schools. For
example, question 2 asked the type of school the child attends. Three respondents left this item
blank, saying that no response option applied to their situation. One parent marked “private, nonreligious,” but added that most homeschool parents wouldn’t categorize homeschooling as such.
Respondents noted that several items in this section did not apply to them, and one respondent
recommended rewording the items to say “school/ homeschool.” Another parent suggested
general instructions telling homeschool parents to think of their “home learning environment”
when reading the questions about “school.” Another respondent mentioned that they would have
thrown away the survey if they received it in the mail after trying to answer the first few items
that did not apply to their homeschool child.
Question 3 on the topical questionnaire asked respondents if the child attended his or her
“district-assigned” school. Three respondents marked “no” and one left it blank, noting that this
item did not apply to homeschoolers. One respondent marked “yes,” and when probed, said it
was because his child is homeschooled in the ward where they live.
Question 7 asked if parents considered other schools for the child. One parent interpreted
“schools” as “curriculum,” explaining that they did investigate other homeschooling curriculums
for the child.
Question 8 asked respondents if they considered the performance of schools, such as test scores
and dropout rates, when selecting a school for their child. One respondent explained that
monetary reasons might be another factor, as he prefers a non-traditional approach to education,
but found that non-traditional private schools were too expensive. Another respondent added that
learning outcomes were an important factor in selecting a program, as he wanted a
homeschooling curriculum that emphasized reading and writing.
Question 14 asked parents if the child was enrolled in any high-school level Advanced
Placement courses. All respondents reported “no” for this item, but one respondent noted that
parents unfamiliar with public schools may not know what AP courses are. Another respondent
added that homeschool parents may confuse this with community college courses, which many
upper-level homeschool children take.
Question 22 asked if the child participates in any online courses or instruction. Two respondents
reported “no,” saying that their children were too young. Another respondent reported “no,”
because they did not use the Internet for specific coursework but as a supplement to their
teaching. One respondent said “yes,” explaining that his family pays tuition for access to a
distance education school in which the children have access to online courses, transcripts, and an
4

advisor. Another marked “yes” and said that her child uses K12, which also links the child to
other online programs to practice the concepts.
If respondents indicate that the child is taking online courses, he or she is asked in question 23 to
report who provides the online instruction. The two respondents both wrote in the “other—
specify” box, and reported “other online homeschooling program” and “our homeschool.”
NHES: 2012 data showed that homeschoolers often wrote other online schools or virtual
academies into this box. This suggests that for this item, the response options may not be suitable
for homeschooled students, requiring additional testing for future administrations.
Question 25 asked if the sampled child is “schooled at home instead of at school for at least some
classes or subjects.” All respondents marked “yes” for this item, and two added that they could
have just skipped to this question as the previous items about traditional schooling did not apply
to them. It was discussed to move this item to the first page (where the item was located in
NHES:2012) but NCES decided that it could introduce additional error as those who do
homeschool would be skipped out of items that could provide additional information about some
types of homeschool households.
Question 26 asked parents to mark the statement that best described their child’s homeschooling
arrangement. All respondents marked “homeschooled for all classes or subject areas,” but one
parent mentioned that he initially wasn’t sure how to answer since his children participate in coops, which are outside of the home.
Question 28 lists a variety of reasons parents might homeschool their children, and asked
respondents to mark if each was a factor in the parent’s decision to homeschool the child.
Among the respondents, “concerned about the school environment, such as safety, drugs, or
negative peer pressure,” “dissatisfied with the academic instruction at other schools,” and
“interested in a nontraditional approach to children’s education” were the most common
responses. One respondent’s main reason was his child’s deafness, which he categorized as a
“physical or mental health problem,” and another parent explained that her child was being
bullied, which she considered “other special needs.” Another respondent reported in the “OtherSpecify” box that one reason not on the list was supervision. The respondent felt that if their
child was homeschooled, they would be under the respondent’s direct supervision.
Although no parents selected “yes” for question 28i, “you prefer the flexibility homeschooling
provides to this child during frequent moves,” two respondents initially just read the first half of
the item, interpreting it simply as the flexibility homeschooling provides. After one respondent
read the remainder of the option, he wrote “flexibility” in the “other—specify” box, explaining
that homeschooling allowed his child to explore other educational and extracurricular
opportunities. Further analysis was conducted using NHES: 2012 data and it was determined that
“frequent moves” was not a common write-in response. Therefore, it was recommended that
item 28i be deleted from the PFI-Enrolled and PFI-Homeschool instruments. The item was
removed from the NHES:2016 instruments.
For this set of items, one respondent noted that it was difficult to mark “yes” or “no” since some
were considerations, but not main reasons for homeschooling. He suggested a scale so that
5

parents could indicate how much each factor weighed in their decision. Two respondents
verbalized that some of the factors were benefits they discovered since starting homeschooling,
but not initial reasons, and they did not mark those options.
When asked to indicate the main reason for homeschooling in question 29, one respondent
marked “h—nontraditional approach to education” and explained that they wanted to educate
their child through a love of learning. Others marked “g—special needs” and “e—physical or
mental health problem,” explaining their children’s experiences with bullying and deafness,
respectively. Two other respondents marked “a—concerned about the school environment.” One
respondent described that after researching schools, he was worried about school security, and
the other said that the child was enrolled in a public school that the parent felt was unsafe and
they wanted to ensure the safety of their child and be able to protect them
Question 30 asked respondents to mark all of the subjects their child had been taught since
starting homeschooling. Most respondents selected several subjects, but in general, parents of
younger children did not select upper level mathematics and science courses. When probed on
how those courses were covered, respondents had a variety of responses. One respondent
explained that his child attends a curriculum-based co-op, as well as a co-op where parents teach
courses such as yoga, museum studies, performing arts, etc. Another parent explained that her
child’s academic subjects are taught online through K12, and the child meets up with other
homeschoolers weekly for subjects like art, music, and physical education. The third parent
described a similar situation, where his children complete academic courses through an online
curriculum, but go to an art studio for art, learn piano from local college students, and go to
physical education classes with other homeschoolers. Another respondent explained that a
certified instructor comes to his home daily to deliver the academic curriculum, and the child
does not participate in activities outside of the home.
Question 31 asked parents to report on the courses the child is taking “now”. Three respondents
marked all of the same subjects as in question 30, noting that these two items are the same for
homeschoolers. However, two respondents selected different subjects for this item based on the
courses covered this year in the child’s curriculum. One parent questioned if “now” meant in the
current season or in the academic school year, explaining that the child takes physical education
in the spring, but not in the winter. Another parent had difficulty with the term “now” and was
not certain if it meant today or the last few months, and chose to select every subject she taught
her child because they do not use a strict curriculum. The parent also recommended including an
item about curriculum instead of this item.
Respondents were also probed on if any subjects were missing from these lists. Some
suggestions that parents provided included: performing arts, dance, handwriting, logic,
religion/spirituality, gardening, and current affairs. One respondent mentioned including a writein box since there are so many topics covered through their homeschooling that do not fall in the
traditional school curriculum.

Child’s Health
Respondents encountered no problems with the “Child’s Health” items.

Child’s Background
Overall, respondents had no problems with the “Child’s Background” items. Question 44 asked
if the child lives “at this address and another address.” One parent initially marked “yes,” reading
only the first half of the item. After seeing “and another address,” she changed her response, but
verbalized that parents may answer this item incorrectly if they are rushing through the survey.
Therefore, it is recommended that this option be revised to “does this child live at this
address and another address (for example, because of a joint custody arrangement)?” This
recommended revision was implemented for the NHES:2016 PFI-Enrolled, PFIHomeschool, and ECPP instruments.

Household Members
Respondents had no trouble accurately reporting household totals and roster counts.

Child’s Family
Overall, respondents easily navigated the questions about the child’s parents. One respondent
noted that he would prefer if the Parent 1 questions said “are you…” instead of “is this…”. For
question 60, which asks about employment status, one respondent mentioned that the wording
“employed for pay or income” was odd, and another questioned why “for pay or income” was
not included with the “self-employed” option. One respondent had difficulty answering this item
for his wife, as she is both a stay-at-home parent and employed part-time.

Your Household
Due to inaccuracies in reporting household members in NHES: 2012, it was decided that the
“Your Household” section would be moved ahead of the “Child’s Family” section. However,
this led to concerns about the income item being too early in the instrument. Additionally, one
homeschool respondent noted that the internet access items in this section seemed out of place
since the previous questions were about education. Therefore, it was recommended that the
“Your Household” section be renamed “Household Members,” and a new “Your
Household” section follow “Child’s Family,” containing the items about benefits, income,
home ownership, and Internet access. This recommended revision was implemented for
the NHES:2016 PFI-Enrolled, PFI-Homeschool, and ECPP instruments. This revised format
was tested in these supplemental interviews, and respondents had no trouble accurately
responding to these items.
For Question 99, all respondents interviewed indicated that they had Internet access on a
smartphone, either through a mobile service provider or Wi-Fi. Four respondents indicated in
question 100 that they had Internet on a computer or tablet, and several mentioned that their
7

children use the Internet on Kindles and iPads. One respondent did not have Internet access
through a computer or tablet because they live in a rural setting and it is difficult to get Internet
except through their phones. In question 101, all respondents reported using the Internet “every
day,” noting that they check their own email daily and use it for their children’s education. One
respondent added that homeschool parents in particular are usually well-connected, as they use
the Internet to participate in online homeschooling groups and forums.

Summary
Respondents in these interviews provided a substantial amount of information regarding their
interpretations of “homeschooling,” their children’s homeschooling arrangements, methods of
instruction, and academic coursework. However, respondents did struggle with some schooling
items, revealing questions that may benefit from additional testing as they are particularly
challenging for parents of homeschooled children. Despite difficulty with select items,
respondents were able to answer the majority of questions included on the instrument, resulting
in few changes to this and other applicable NHES surveys.

1. Since “frequent moves” was not a common write-in in NHES: 2012, and respondents
may confuse this option as simply the flexibility of homeschooling, item 28i should be
deleted from the instrument. This change applies to the PFI-Enrolled and PFIHomeschool instruments.

2. “And” should be underlined in question 44 to ensure that this item captures children who
live at multiple addresses as opposed to just one. This change applies to the PFI-Enrolled,
PFI-Homeschool, and ECPP instruments.

3. The “Your Household” section should be renamed “Household Members,” and the
benefits, income, home ownership, and Internet access items should be moved to a new
“Your Household” section after “Child’s Family”. This change applies to the PFIEnrolled, PFI-Homeschool, and ECPP instruments.

References
Cole, S., Gonzalez, R., Fronberg, K., & Chavez, S. (2015). 2016 Parent Family Involvement in
Education and Early Childhood Program Participation Surveys Cognitive Laboratory Final
Report. Washington, DC: U.S. Department of Education, National Center for Education
Statistics.
Jackson, M., McQuiggan, M., and Megra, M. (forthcoming). NHES:2014 Feasibility Study
Report. Washington, DC: U.S. Department of Education, National Center for Education
Statistics.

Appendix: Questionnaire and Interview Protocol

2016 Parent and Family Involvement in Education Survey (PFI)
Cognitive Interview Protocol (for parents of homeschooled children)
Interview ID:
Parent First
Name:
Age of Child:

Date of
Interview:
Interviewer

Also, this interview will be audio-recorded so researchers can review the recording later. Any
information that refers to you or your family personally, like your name or the name of your child,
will not be included in our report, and your responses will be used for research purposes only.
Remember, you are not being tested; we want you to tell us what you think about the survey.
If at any time you want to stop, just let me know.
[If, for any reason, the participant is no longer interested in participating, thank the participant for
his/her time and end the interview.]
Before we continue, do you have any questions about what I just said? [Answer any questions the
parent may ask.]

IF THE RESPONDENT DID NOT BRING A SIGNED CONSENT FORM, PROVIDE HIM/HER
WITH A COPY OF THE FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ
IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY
QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE
COPY OF THE FORM WITH THE RESPONDENT.
IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE
While you fill out the survey I want you to talk about how you are answering the questions. Hearing
you talk about how you figure out your answers to the questions will help me understand how to
make the questions better. I am more interested in how you arrive at your answers than in the
answers you give me. To help you get started on thinking aloud, we are going to do a practice
activity.
First, I will show you an example of how to think out loud. Then I will give you a chance to practice
it.
QUESTION: How many glasses of water did you drink yesterday?
[Model the think aloud process for the parent, including reading the question out loud. Elaborate on
counting all of the glasses of water and as needed you can make up pieces of information to
emphasize the think aloud process.
[Once finished the model practice think aloud]
I’m going to give you a piece of paper with the question on it. I’m going to ask you to read out loud
the question and think aloud as you decide on your answer.
Give respondent handout with think aloud practice.
How many glasses of water did you drink yesterday?
Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when
you were deciding on that answer? Can you tell me more about what you are including in your
count?
When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about
your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN OR PENCIL.

STEP 3: COMPLETION OF THE QUESTIONNAIRE
Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on
your answer. Please read aloud anything you would have read to yourself if I were not here.
Remember that I am very interested in hearing you talk about what you are thinking as you figure out
your answers to the questions.
REMEMBER TO TELL THE PARENT THROUGHOUT THE QUESTIONNAIRE THAT
THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO
YOUR QUESTIONS ARE REALLY HELPFUL.

Useful Probes
Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are
thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old
question before answering this question]: “Tell me what just happened. Tell me about what you
were just doing.”
Screener
ASK THE RESPONDENT TO FILL OUT
THE ENTIRE SCREENER. REMIND THEM
TO THINK ALOUD IF THEY ARE NOT
DOING SO.
Q3. What is the person’s month and year of
birth?
[If the respondent is having difficulty with the
month and/or year] How did you come up with
your answer?
Q5. Is this person currently in…?
How did you come up with your answer?
[If respondent chooses “homeschool”] What
does “homeschool” mean to you?

PFI Survey
CHILD SELECTION: SELECT THE FIRST CHILD AGES 5-17 ON THE SCREENER
THE R REPORTED TO BE CURRENTLY HOMESCHOOLED.
Child’s Schooling
ASK THE RESPONDENT TO FILL OUT
THE SURVEY ABOUT THE CHILD
SELECTED. ASK QUESTIONS AND USE
PROBES ONLY AS NECESSARY; REMIND
RESPONDENT TO THINK ALOUD

Instructions: It is not necessary to probe on
the instructions. Note if the respondent has any
problems if they read them.

Q1-24: Pay attention to how the R reacts to
these items and take notes and probe
accordingly. Since the child is homeschooled
these items may not apply. If R is having
difficulties remind the R to fill out the
survey as they would if they received it at
home.
[Probe if necessary] How would you react if
you were to receive this survey in the mail at
home?
Q1-Q2: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q3. Is it his/her district-assigned school? A
district-assigned school is the school that your
local public school district told you that this
child can attend, based on the location of your
residence.
[Ask only if responded to this question] How
did you decide on your answer?
What does “district assigned” mean to you?
How did you know what school your child was
going to attend? How do other parents in your
child’s district know which schools their child
will attend?
Q4-Q13: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q14. Is he/she currently enrolled in any high
school Advanced Placement (AP) classes?
What were you thinking of when you answered
this question?
What do “Advanced Placement (AP) classes”
32

mean to you?
What grade-level are the students who usually
take AP classes in your child’s school district?
Q15-Q21: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q22. Is this child taking any school-related
courses online instead of in-person with the
teacher? Do not include courses that use the
Internet only for selected assignments.
How did you decide on your answer?
Q23. Is that instruction [internet] provided by
any of the following places?
How did you decide on your answer?
[If chooses a response option or “Someplace
else:”] Tell me more about that. How does
your child access the classes? Where is he/she
when accessing the classes? Who provides the
class?
Q24: It is not necessary to probe on this
question. Let the respondent continue the think
aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q25. Some parents decide to educate their
children at home rather than send them to a
public or private school. Is this child being
schooled at home instead of at school for at
least some classes or subjects?

In your own words, tell me what this question
is asking about.
How did you determine whether to answer yes
or no?
[If respond “yes,”] Do not probe on the
meaning of homeschool until Q29.
33

[If respond “no” and if not evident from think
aloud to previous questions] What does
“homeschool” mean to you?
Make a note if the respondent did not skip
properly, but do not probe on skip pattern.]
Q26. Which of the following statements best
describes your homeschooling arrangement
for this child?
How did you come up with this answer?
[Do not probe on the meaning of homeschool
until Q29. Check to see if respondent skipped
properly, but do not probe on skip pattern.]
Q27. How many hours each week does this
child usually go to a public or private school
for instruction? Do not include time spent in
extracurricular activities.
How did you come up with your answer?
[Check to see if respondent filled in this
question although they indicated that the child
is homeschooled.]
Q28. There are many different reasons that
parents choose to homeschool their children.
Did your family choose to homeschool this
child because:
How did you determine how to answer yes or
no to each of these questions [a-j]?
Tell me more what you were thinking when
answering these questions.
[Do not probe on the meaning of homeschool
until Q29.]
Q29. Of the reasons your family chose to
homeschool this child, which one would you
say is the most important to you?
How did you come up with your answer?
[If not evident from think aloud to previous
questions] What does “homeschool” mean to
you?
34

Q30. Thinking about all years this child has
been homeschooled, which of the following
subject areas has this child been taught
during his or her home instruction?
How did you decide what to mark?
How do you cover these subjects during home
instruction?
Is there anything missing?
Q31. Which of the following subject areas are
being taught to this child now?
How did you come up with your answer? Can
you tell me more about what the child is being
taught now?

Child’s Health
REMIND RESPONDENT TO THINK
ALOUD
Q32-Q35: It is not necessary to probe on
these questions. Let the respondent continue
the think aloud and only probe if respondent
says something that indicates a potential
problem, or if there is confusion or an
inconsistent interpretation compared to other
respondents.
Q36. Are these services provided by any of the
following sources?
How did you decide on your answer?
Q37. Are any of these services provided
through an Individualized Education Plan
(IEP) or services plan?
How did you come up with your answer?
What does a “services plan” mean to you?

Child’s Background
Q38-Q43: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q44. Does this child live at this address and
another address (for example, because of a
joint custody arrangement)?
How did you come up with your answer?
In your own words, what is this question
asking?
[If answered No] What does joint custody
arrangement mean to you?
Q45. If yes, does this child spend most of the
time at this address, at another address or
equal time at both addresses?
How did you come up with your answer?
[If needed] Can you describe the arrangement
your child has?
Q46. What language does this child speak
most at home?
It is not necessary to probe on this question.
Note the respondent’s response, especially for
those who are parents of ELL students. Record
in your notes whether the respondent reads the
response options below the top two responses.
Q47. It is not necessary to probe on this
question. Let the respondent continue the think
aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.

Your Household
Q48. How many people live in this
household?
How did you come up with your answer?

Who are you including in the counts?
Q49. How many of the following people live
in this household with this child?
Tell me more about what you were thinking
when you answered this question?
Who are you including in the counts?
In your own words, what is this question
asking?
Q50-Q57: It is not necessary to probe on these
questions. Let the respondent continue the
think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Pay particular attention Q53, noting if the
respondent seems uncomfortable answering the
income item.
Q58a. Do you have Internet access on a
smartphone?
Please tell me about how you chose [no, yes]
for this one.
Q58b. Do you have Internet access at home
on a computer or tablet?
Please tell me about how you chose [no, yes]
for this one.
[If not obvious from response], Where are you
thinking about? [Find out where “at home” was
interpreted by R to be.]
Q59. How often do you use the Internet?
Please talk to me about how you got to your
answer here.
[Look for indications that the response options
are not inclusive enough/ adequate.]
Child’s Family
REMIND RESPONDENT TO THINK
ALOUD
Q60-Q71: It is not necessary to probe on these
questions. Let the respondent continue the
37

think aloud and only probe if respondent says
something that indicates a potential problem, or
if there is confusion or an inconsistent
interpretation compared to other respondents.
Q72. What is the highest grade or level of
school that this parent or guardian
completed?
[Only if respondent chooses “high school
diploma or equivalent”, “vocational diploma
after high school”, or “some college, but no
degree”:]
How did you come up with your answer?
What does “vocational diploma after high
school” mean to you?
Q73. Is he or she currently attending or
enrolled in a school, college, university, or
adult learning center, or receiving vocational
education or job training?
In your own words, what is this question
asking?
Q74-Q100: It is not necessary to probe on
these questions. Let the respondent continue
the think aloud and only probe if respondent
says something that indicates a potential
problem, or if there is confusion or an
inconsistent interpretation compared to other
respondents.

FINAL QUESTION
Do you have anything else you would like to tell me about this questionnaire that you haven’t had a
chance to mention?

STEP 4. THANK YOU FOR PARTICIPATING
We are finished. Thank you for helping us. The information you’ve provided will be very helpful for
making this survey better.
Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to
the lobby. Please give the respondent incentive envelope [$40 gift card], have the respondent sign the
receipt, and thank them for their time and effort.

Memo
Date:

June 3, 2015

To:

Sharon Boivin and Sarah Grady, NCES

From

Anna Sandoval Giron and Meghan McQuiggan, AIR

Re:

Preliminary Findings from the 2015 Spanish Cognitive Interviews

Introduction
In 2016, the National Center for Education Statistics (NCES) will conduct the second full-scale,
mail-based administration of the National Household Education Survey (NHES). The NHES
uses a two-stage design in which sampled households are first sent a screener survey and asked
to list the members of their household, as well as provide some key information on each
individual, including their date of birth, sex, and school enrollment status. The responses from
the screener survey determine if any household members are eligible to receive one of the NHES
topical instruments, which ask additional questions about the sampled individual’s education.
In 2016, the NHES will contain three topical surveys: the Early Childhood Program Participation
(ECPP) survey, the Parent and Family Involvement in Education (PFI) survey (Enrolled and
Homeschool), and the Adult Training and Education Survey (ATES). All contact materials,
screener instruments, and topical instruments are administered in both English and Spanish.
Additionally, the NHES:2016 will contain a web experiment in which some respondents will be
asked to complete the screener and topical surveys online.
In order to reduce measurement error and ensure high-quality translations, all Spanish materials
are being tested using cognitive interview methods. Spanish interviews are being conducted in
three phases. Phase 1 tested the Spanish screener and topical letters, postcard reminder, screener
questionnaire, and new ATES items that had not previously been translated. Phase 2 tested the
topical questionnaires, including the ECPP, PFI- Enrolled, and ATES. These instruments
contained translations from the 2012 administration, as well as new translations due to item
revisions after the English cognitive interviews. Phase 3 will test the contact materials for the
web survey, however those findings are not yet available.
Phase 2 and 3 were ongoing but mostly complete, at the time this memo was written The memo
documents all findings to date from the Spanish interviews, and no further substantive changes
are anticipated.

Sample
Phase 1 interviews took place in Washington, DC, San Mateo, CA, and Houston, TX with 26
respondents who are Spanish dominant speakers (they either speak only Spanish or are most
comfortable speaking Spanish). Phase 1 interviews were conducted between March and April
2015.
Phase 2 interviews took place in Washington, DC, San Mateo, CA, Houston, TX, and Tucson,
AZ with 82 respondents (to date) who are Spanish dominant; – 20 on ECPP, 38 on PFI-Enrolled,
and 24 on ATES. Phase 2 interviews were conducted between April and May 2015.

Key Findings: Phase 1
Screener and Topical Letters
General Issues
In the first round, respondents had difficulty with the names of government agencies, as
translating these organizations often led to sentences that were quite lengthy. For example, the
opening sentence from the screener letter, including the Spanish version, is shown below:
I am pleased to inform you that the U.S. Census Bureau has selected your household to
participate in the 2015 National Household Education Survey, which we are conducting on
behalf of the U.S. Department of Education.
Me place informarle que la Oficina del Censo de los Estados Unidos ha escogido su hogar para
que participe en la 2015 Encuesta Nacional en Hogares sobre Educación, la cual estamos
realizando para el Departamento de Educación de los Estados Unidos.
Respondents often skipped these sentences or did not understand the information. When asked
who sponsored the study, six of the 20 respondents understood that it was the U.S. Department
of Education. Four respondents said “Census,” and others reported “Department of Commerce,”
“the government,” “teachers,” and “the schools.” Three respondents said they did not know who
was sponsoring the study.
Based on these results, it was recommended that the opening paragraph of all letters be
simplified so that the language is easier for readers to understand. For round 2, this wording was
changed to the following:
I am pleased to inform you that your household has been selected to participate in the 2015
National Household Education Survey. This is a U.S. Department of Education survey that is
being conducted by the U.S. Census Bureau.

Me complace informarle que su hogar ha sido escogido para participar en la Encuesta Nacional
en Hogares sobre Educación del 2015. Esta es una encuesta del departamento de Educación
realizada por la Oficina del Censo de los Estados Unidos.
After the wording was changed, six of the seven respondents in Round 2 interviews reported that
the study’s sponsor was the “Department of Education,” “Census Bureau,” or “the government.”
Therefore, it is believed that simplifying this language made it clearer to respondents that the
study was sponsored by a government entity.
At the request of NCES, the term “conducted” was changed to “administered” in the English
version, but the Spanish translation remains the same.
Translation Issues
Both the screener and topical letters include the phrase “token of appreciation,” which is
translated as “dinero en efectivo.” In round 1, 19 of 20 respondents believed that this meant
“cash.” For round 2, the phrase was re-translated as “muestra de aprecio.” No respondents in
round 2 had any difficulty with this language, and it is believed that this translation better
captures the concept of “token of appreciation.”
The letters also contained the phrase propósitos estadísticos, “statistical purposes.” Several
respondents did not understand this wording, suggesting the need for an improved translation.
For round 2, the wording was changed to fines estadísticos in an attempt to make the text shorter
and more readable.
The ECPP topical letter specifies that the study provides information about educación de niños a
temprana edad, “children’s early learning,” however, in the questionnaires, this was translated
as educación a niños pequeños, “education for small children.” In order to be consistent, it was
recommended after round 2 that all references to children’s early learning match the language
used in the questionnaires.
The ATES topical letter specifies that the survey is about adiestramientos y credenciales que los
adultos reciben para trabajar, “training and credentials that adults get for work.” Several
respondents reported that the word adiestramientos, “training,” seems to refer to animal training,
and instead suggested using the word entrenamiento. Once this revision was made for round 2,
respondents had no issues with this language.
At the request of NCES, some references to “work” were dropped from both the English and
Spanish letters to place less emphasis on employment because respondents are not required to be
working to be eligible for the survey.

Screener Questionnaire
General Issues
Question 5 on the screener asks respondents for the individual’s school enrollment status. For
people not in school, the response option provided is a word-for-word translation, no está en la
escuela, “not in school.” In round 1, respondents had trouble with this item, often looking for a
response option that said ninguna, “none,” as is used in many other Spanish surveys. For round
2, ninguna was used in place of no está en la escuela, and respondents who were not in any of
the school categories had less trouble finding an applicable response.
Respondents also had trouble with the homeschooling response option in question 5. This option,
in English and Spanish, is below:
Homeschool instead of attending a public or private school for some or all classes.
Escuela en el hogar (homeschool) en lugar de asistir a una escuela pública o privada para
algunas o todas las clases.
The concept of homeschooling confused multiple respondents. Generally, they either thought
this was referring to a parent supporting a child’s education at home, or an adult using
educational materials for their own self-improvement purposes. Due to the perceived low
prevalence of homeschooling among Spanish-speaking families, this is believed to be a
conceptual issue because Spanish respondents simply are unfamiliar with homeschooling.
Therefore, it was recommended that any translation revisions should only be made to address
substantial changes in the English homeschooling questions, which were tested separately.
Respondents also had difficulty understanding question 6, which asks for each individual’s
current grade equivalent, or “grado actual de esta persona, o su equivalente.” Some respondents
in cognitive testing interpreted this as meaning the highest level of education the person had
completed, particularly for individuals not enrolled in school. However, no changes were made.

Key Findings: Phase 2
ECPP, PFI, and ATES
General Issues
Overall, respondents across the three topical surveys had difficulty following skip patterns. In
round 1, eight of 10 respondents did not skip to other sections or questions when they should
have. When further probed, two respondents commented that in other “official” questionnaires
they had been told to fill in all possible information, so they chose to complete as many questions
as possible, disregarding skip instructions.

Across all surveys, respondents also found questions related to race challenging. In the first
round, four of 10 respondents commented that they could not find an adequate response option
(Hispanic or Latino) among those provided. Some eventually chose White, commenting that that
is what they are “supposed” to choose.
No changes were made to skip instructions in the questionnaire since respondents were making
the more desirable reporting error of commission instead of error of omission, however, NCES
intends to implement post-survey edits for skip patterns which will take into account this
cognitive finding.

PFI and ECPP
General Issues
These two surveys contain a question asking if “this child is currently enrolled in English as a
second language, bilingual education or an English immersion program?” Three parents
expressed that they thought this was asking if the teacher could communicate with the child in
Spanish. It is believed that this is both a translation issue, and a conceptual one in that parents are
unfamiliar with “immersion programs.” In Round 2 we discovered that the issue was the
translation of immersion programs. The original translation of the question used programa
especial de inglés to refer to immersion programs. In addition the original translation used the
term lengua, “tongue” to refer to language. We determined these two terms/phrases were causing
confusion to respondents and suggested a new translation of the question that reflects the
language commonly used by schools and parents. NCES implemented this recommendation.
New suggested translation: Actualmente, ¿asiste este niño(a) a clases de inglés como segundo
idioma, educación bilingüe o a un programa de inmersión al inglés?
Translation Issues
These topical instruments also ask parents to report any of the sampled child’s health conditions.
Although two respondents were confused by this section and chose to leave it blank, the others
had no problems. However, two terms in this section proved challenging for respondents. The
phrase impedimiento del habla, “language impediment,” was marked with a yes by some parents.
During probing these parents explained that they chose yes for this option because their children
are learning to talk in two languages and have not mastered one of the languages. NCES has
decided not to make any changes to the survey based on this finding.
Another respondent had difficulty with the term, Trastorno generalizado del desarrollo (PPD),
“Pervasive Developmental Disorder (PDD).” AIR researchers have followed-up on this
translation and reviewed the wording in the U.S. Department of Health and Human Services. The
phrase Trastorno generalizado del desarrollo (PPD) is the commonly used translation to refer to
these disorders. Currently we believe that parents that have a child diagnosed with Pervasive
Developmental Disorder will be able to identify themselves and correctly answer the question.
No change is recommended.

ECPP
General Issues
In round 2, parents with children in home-based daycare had difficulty categorizing this type of
care. When asked if the child receives home-based care from a non-relative, parents of children
in home-based daycares often responded “yes.” However, when respondents got to the series of
questions about daycares, preschool, and prekindergarten programs not in private homes, they
wanted to report the programs in this section as well, and it seemed that respondents were not
reading the “not in a private home” text. Overall, four of seven respondents in this round had
response error in this set of questions. No change was recommended.

ATES
General Issues
Similar to the English ATES interviews, respondents had difficulty distinguishing between
certifications, certificates, and licenses.
In the second round, it was discovered that respondents with education and licensing from
foreign organizations and governments often had difficulty categorizing their credentials. One
respondent explained that in her country, her high school education is considered two separate
certificates- she can earn both a high school degree and technical career degree. It is believed
that this issue may not be limited to Spanish speakers, but to many immigrant groups.
Translation Issues
The ATES questionnaire asks respondents if their most important certification or license can be
revoked or suspended for any reason. “Revoke” is currently translated as “revocada,” but the
term revocada produced confusion for some respondents. Due to the confusion around this term,
it was recommended that this language be revised to quitar, which, is a synonym of revocada
meaning means “remove” or “take away.” .The current and recommended translations are below.
English version: Can your most important certification or license be revoked or suspended for
any reason?
Current translation: Su certificación o licencia más importante, ¿podría ser revocada o
suspendida por alguna razón?
Suggested translation: ¿Le podrían quitar o suspender su certificación o licencia más
importante por alguna razón?

Summary
Overall, these two phases of cognitive interviews provided a lot of insight into Spanish speakers’
interpretation and understanding of NHES contact materials and questionnaires. Respondents
were able to highlight areas of confusion and problematic language, which led to improved
wording and translations. For example, in the screener and topical letters, introductory text
containing government agency names was simplified to make it more readable. In the screener
questionnaire, the response option for individuals not in school was revised to reflect a response
option that Spanish speakers are more familiar with. Testing of the screener questionnaire also
reinforced that homeschooling continues to be a challenging concept for Spanish speakers.
Cognitive testing also revealed concepts and language that needed to be revised on the topical
questionnaires. While some translation changes were made during testing, others require
additional testing and investigation before revisions can be made. Across all of the instruments,
some of the items that were difficult included Spanish immersion programs, child language
impediments, school detentions and suspensions, home-based daycare programs, and the
revocation of adult credentials. The final analysis of the interview data, as well as future
cognitive interviews, will continue to inform the development and refinement of Spanish NHES
surveys.

Appendix A: Phase 1 and Phase 2 Protocols

Phase 1 Protocol

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview.
Also, keep track of the time so that your total interview time with the respondent lasts 1 hour or
less.
Before starting the interview, use this table to confirm the participant info that we received from
the recruitment firm. If the respondent indicates that some of the listed information is incorrect,
please cross it out and enter the correct information in the relevant cell.*

Background Information

Confirmed


Date & Time
Name of Interviewer
Parent/Respondent First Name
Respondent’s Gender

Male

Female

Respondent’s Age
If parent fill following questions:
Child Enrolled in School

Yes

Name of School

(only if R is willing to share)

Type of School

Public

Charter

Private

Homeschool

Student’s Grade
Age of Student
Assigned ID

*This cover sheet should be kept separate from the rest of the data and
be turned in with the rest of the paperwork. This cover sheet should not
be copied, shared, or left in an unlocked office.

Documento Adjunto 3
Protocolo en Español para las Entrevistas Cognitivas
MATERIALS NEEDED FOR INTERVIEW
□ INTERVIEWER PROTOCOL
□ RESPONDENT LETTERS AND QUESTIONNAIRE
□ DO YOU HAVE THE CORRECT VERSION OF THE LETTERS, PROTOCOL,
AND QUESTIONNAIRE?
□ CONSENT FORM (TWO COPIES)
□ INCENTIVE
□ PAYMENT RECEIPT
□ DIGITAL RECORDER AND EXTRA BATTERIES
□ PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT
Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is
suggested content with which the interviewer should be thoroughly familiar in advance. Text in
brackets [ ] are instructions for the interviewer.
Hola, mi nombre es y trabajo para [COMPAÑIA]. Mucho gusto de conocerle.
Gracias por venir a ayudarnos el día de hoy.
[Platique de algún tema casual para establecer una buena relación con el participante.]:
Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos
realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta
acerca de [la educación de sus hijos, la búsqueda y selección de cuidado infantil para los padres
con hijos de cinco años de edad o menores, o acerca de sus certificados de estudios].
Este estudio es sobre [la educación de su hijo(a), cómo buscaron y escogieron el para su hijo(a)
para aquellos padres con niños de cinco años o menores, y sobre sus certificados de educación.
Como parte de este estudio, le voy a pedir que lea algunas cartas y que me diga lo que piensa
que estas cartas le están pidiendo que haga. Además, voy a pedirle que tome una breve encuesta
como que si estuviera tomando una encuesta real. No existen respuestas correctas o incorrectas
y está bien si usted no sabe algunas de las respuestas. A medida que usted vaya leyendo las
cartas y respondiendo a las preguntas, yo le voy a hacer algunas preguntas acerca de sus
respuestas. Esto nos ayudará a que las cartas y la encuesta sean mejores y más claras.
Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas
mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va
a observar nuestra sesión de hoy].

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan
revisar la grabación más tarde. Cualquier información que se refiera a usted o su familia
personalmente, tal como su nombre o la escuela de su hijo(a), no será incluida en nuestro
informe y sus respuestas serán utilizadas para fines de investigación únicamente.
Recuerde que usted no está tomando un examen. Queremos que nos diga lo que usted piensa
acerca de las cartas y la encuesta.
Si en algún momento usted desea detenerse, solamente déjemelo saber.
[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y
termine la entrevista.]
Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a
cualquier pregunta que pueda hacer el participante.
PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE
RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON
RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT
SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.
IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: SCREENER LETTER
Ahora me gustaría que complete esta actividad como que si estuviera sucediendo en su casa.
Voy a darle este sobre y quiero que usted haga lo que usted haría normalmente si usted
recibiera y abriera estos materiales en su casa.
Voy a salir del cuarto por aproximadamente cinco minutos para dejar que usted lea los
materiales y luego regresaré para que podamos hablar al respecto.
[ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL
CUARTO/AREA.]

[Cuando el entrevistado termine:]
Gracias por tomarse el tiempo para leer estos materiales.
Screener letter
Ahora tengo unas cuantas preguntas para usted, pero me gustaría que las
responda sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias.
Me gustaría preguntar acerca de su reacción a la carta. Por favor, recuerde
que no hay respuestas correctas o incorrectas, sólo estamos tratando de saber
esta carta puede ser interpretada.
PREGUNTE:
 En general, ¿Quién manda esta carta?


¿Qué se espera que haga usted después de leer la carta?
11



¿Para qué se va a usar la información recolectada por este estudio?

Gracias.
Ahora quiero preguntarle acerca de los detalles de la carta. Voy a devolverle la carta y podemos
hablar más acerca de ella.
[Anote cualquier lugar en el que el entrevistado dude al leer, secciones que se salte el
entrevistado, palabras que el entrevistado pronuncie incorrectamente o cualquier cosa acerca de
la que haga comentarios el entrevistado.]
Quién
Por favor, lea en voz alta el primer párrafo de la carta.
[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 ¿Después de leer este párrafo, puede decirme quien le está enviando
esta carta?
 ¿Puede decirme quién cree usted que es el patrocinador de esta
encuesta?
 ¿Acerca de que cree usted que se trata esta encuesta? En términos
generales, usted no tiene que darme detalles específicos, solo lo que
usted piensa acerca de lo que leyó en este párrafo.
Qué
Continuemos. Por favor, ¿puede leer el segundo párrafo?
[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 Hablemos acerca de la primera oración acerca de que su hogar fue
elegido al azar. En sus propias palabras, ¿qué quiere decir esto?
 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles?
Cómo
Continuemos. Por favor, ¿puede leer la siguiente oración y los puntos?
[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 ¿Qué dice esto que se le enviará en la siguiente semana?
 Usamos el término “muestra de aprecio.” ¿Qué significa este término
para usted?
Privacidad
Continuemos y hablemos acerca de este texto en el cuadro. Por favor, ¿Puede
leerlo en voz alta?
12

[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 ¿Hay algunas palabras aquí que no le sean familiares?
 ¿Qué cree que está tratando de decir este párrafo?

STEP 3: THINK-ALOUD PRACTICE
Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las
preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así
entender cómo mejorar estos materiales. Me interesa más saber cómo llega a sus respuestas que
en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer
un ejercicio de práctica.
Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una
oportunidad para que practique.
[Model the think aloud process for the respondent, including reading the question out loud.
Elaborate on counting all of the glasses of water as needed you can make up pieces of
information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee
I drank in the morning, I drink three cups of coffee, I am going to count those since that is the
only liquids I have in the morning...” When you are modeling, write down the number of glasses
of water as you are thinking aloud.]
[Once finished the model practice think aloud]
Voy a darle un papel con una pregunta. Le voy a pedir que lea en voz alta la pregunta y que
piense en voz alta mientras decide su respuesta.
[Give respondent handout with think aloud practice.]
¿Cuántos vasos de agua tomó usted ayer?
Por favor dígame lo que está pensando mientras me da su respuesta.
Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta?
¿Qué estaba visualizando mientras estaba decidiendo que responder? ¿Puede decirme más
acerca de lo que usted está incluyendo en su conteo?
Cuando haya completado el ejercicio: Gracias. Ese es el tipo de detalle que busco cuando usted
me hable acerca de sus respuestas a las preguntas de la encuesta.
Preguntas de seguimiento (probes) En sus propias palabras, ¿puede decirme lo que dice esa
pregunta?
¿Qué significa la palabra [término] en esta pregunta para usted?
¿Qué tan fácil o difícil es recordar [tema]?
13

¿Qué tan fácil o difícil es escoger una respuesta?
Dígame lo que está pensando.
¿Cómo llegó a esa respuesta?
Usted dijo [respuesta]. ¿Puede decirme más acerca de esto?
[En respuesta a las acciones o comentarios del entrevistado]: Dígame que pasó. Dígame que
estaba haciendo.
[Si nota alguna señal visual, como cejas fruncidas o una expresión de confusión]: “Dígame que
está pensando.”
[Si nota alguna señal visual que indique confusión como voltear las páginas para regresar a
secciones anteriores o volver a leer preguntas anteriores antes de responder a esta pregunta:
Dígame qué es lo que acaba de pasar. Dígame acerca de lo que acaba de hacer.”

STEP 4: SCREENER QUESTIONNAIRE
Ahora voy a darle una breve encuesta. Me gustaría que trate de completar las preguntas como si
lo estuviera haciendo en casa, excepto que quiero que piense en voz alta mientras decide su
respuesta, así como lo hicimos durante el ejercicio. Por favor lea en voz alta cualquier cosa que
usted hubiera leído para sí misma, como si yo no estuviera aquí.
Recuerde que estoy muy interesada/o en escucharle hablar acerca de lo que está pensando
mientras decide las respuestas para las preguntas.
RECUERDE DECIRLE AL ENTREVISTA DURANTE LA ENTREVISTA QUE ESTA
HACIENDO UN BUEN TRABAJO AL HABLAR EN VOZ ALTA, Y QUE SUS
RESPUESTAS A LAS PREGUNTAS SON MU UTILES.
Screener
POR FAVOR LLENE TODA LA ENCUESTA Y PIENSE EN VOZ ALTA
COMO HICIMOS EN EL EJERCICIO.
Q3. ¿Cuál es el mes y año de nacimiento de la persona?
[Si el participante tiene dificultad con el mes yo año] ¿Cómo llegó a esta
respuesta?
Q5. ¿Asiste esta persona actualmente a. . .?
¿Cómo llegó a esta respuesta?
[Si no es evidente por lo que dijo al pensar en voz alta al responder preguntas
previas] ¿Qué quiere decir para usted “Escuela en el hogar (homeschool) en
lugar de asistir a una escuela pública o privada para algunas o todas las clases?

STEP 5: TOPICAL LETTERS
[PARA PADRES DE FAMILIA O TUTORES: LLENE EL NOMBRE DEL PRIMER NIÑO/A
LISTADO EN EL CUESTIONARIO.
PARA ADULTOS: LLENE EL NOMBRE DEL PRIMER ADULTO LISTADO EN LA
ENCUESTA.]
Ahora tengo otra carta para usted. Por favor tome unos minutos para leerla.
[ENTREGUELE EL SOBRE Y UNA PLUMA O LÁPIZ AL ENTREVISTADO Y SALGA DEL
CUARTO/AREA.]

[Cuando el entrevistado termine:]
Gracias por tomarse el tiempo para leer estos materiales.
Topical Letters
Al igual que antes, tengo unas cuantas preguntas para usted, pero me gustaría que las responda
sin ver la carta. Por favor, ¿Puede devolverme la carta? Gracias.
Ahora me gustaría preguntarle acerca de su reacción a la carta. Por favor, recuerde que no hay
respuestas correctas o incorrectas. Sólo estamos tratando de entender cómo usted interpreta la
carta.
PREGUNTAR:






En general, ¿quién manda esta carta?
¿Qué se espera que haga usted después de leer la carta?
¿Para qué se va a usar la información recolectada por este estudio?
¿Acerca de qué se trata este estudio?

[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 ¿Acerca de quién debe responder la encuesta?
 ¿Acerca de qué cree usted que se trata esta encuesta? En términos generales, usted no
tiene que darme detalles específicos, sólo lo que usted piensa acerca de lo que leyó.
 ¿Hay algunas palabras aquí que no le sean familiares? ¿Cuáles?
Cómo
Continuemos. Por favor, ¿puede leer los siguientes dos párrafos?
[Espere a que el entrevistado termine de leer el párrafo.]
PREGUNTE:
 ¿Dónde puede encontrar más información acerca del estudio?

STEP 6: SCREENER POSTCARD
El patrocinador de la encuesta también envía esta postal. Por favor, tómese un minuto para
leerla.
SCREENER POSTCARD
PREGUNTE:
 ¿Cuál es el propósito de la postal?
 ¿Quién manda la postal?
 ¿Hay algunas palabras aquí que no le sean familiares?
otros comentarios
Gracias.

Continúe con la entrevista

STEP 7: ATES ITEMS
La última actividad que le voy a pedir que hagamos hoy es responder a algunas preguntas acerca de su
adiestramiento laboral. Estas son preguntas nuevas que se están considerando para la encuesta y sus
comentarios acerca de ellas serán útiles. Al igual que hicimos antes, por favor, responda a las preguntas
y " piense en voz alta " mientras contesta las preguntas.

ATES ITEMS
[haga preguntas de seguimiento (probes) si es necesario y aplican al
encuestado]
Q5. Ahora veamos lo que hizo para preparase [Q5. ¿Hizo usted alguna de
las siguientes cosas al prepararse para obtener su certificación o licencia
más importante? (Marque UNA casilla en cada una de las siguientes
líneas.)]



Por favor lea la letra D y dígame, en sus palabras, ¿que significa
para usted esto? [opción d: Obtuve un título universitario].
¿Ahora hablemos de la opción E? [opción e: Participé en
formación o capacitación laboral, pasantía o práctica profesional, o
un programa de aprendiz.]

Q6. Veamos esta pregunta:
¿Qué tan útil ha sido su certificación o licencia más importante para cada
una de las siguientes situaciones?


Quiero que lea la opción C otra vez y me diga, en sus propias
palabras, ¿qué significa esto para usted? [opción c. Mantener el
interés de los empleadores o clientes en usted].



Ahora veamos la opción D ¿Qué significa esto para usted? [opción
d: Mejorar sus habilidades de trabajo].

Q7. Ahora veamos las preguntas sobre su experiencia de trabajo. Por favor
lea la pregunta 7 otra vez. ¿Me puede decir que significa el termino
“educación o formación/capacitación formal” para usted?


Hablemos de “pasantía o práctica profesional”, ¿que significa para
usted esto?



Y finalmente programa “de aprendiz”, ¿que significa para usted
esto?

FINAL QUESTION
¿Tiene algo más que le gustaría hablar sobre estos materiales que no hemos tenido la
oportunidad de hablar?

THANK YOU FOR PARTICIPATING
Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para
mejorar las cartas y encuesta.
Interviewer Instructions: Collect materials and escort respondent out of the interviewing room
and to the lobby. Please give the respondent incentive envelope [$40 cash] and thank them for
their time and effort.

Phase 2 Protocols

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview.
Also, keep track of the time so that your total interview time with the respondent lasts 1 hour or
less.
Before starting the interview, use this table to confirm the participant information we received
from the recruitment firm. If the respondent indicates that some of the listed information is
incorrect, please cross it out and enter the correct information in the relevant cell.*

Background Information

Confirmed


Date & Time
Location
Name of Interviewer
Parent/Respondent First Name
Respondent’s Gender

Male

Female

Respondent’s Age
If parent fill following questions:
Child Enrolled in School

Yes

Name of School

(only if R is willing to share)

Type of School

Public

Charter

Private

Homeschool

Student’s Grade
Age of Student
Assigned ID

*This cover sheet should be kept separate from the rest of the data and
be turned in with the rest of the paperwork. This cover sheet should not
be copied, shared, or left in an unlocked office.

1000 Thomas Jefferson Street NW, Washington, DC 20007-3835 | 202.403.5000 | TTY 877.334.3499 | www.air.org

Esta página se ha dejado en blanco intencionalmente

Documento Adjunto 3 NHES 2016 PFI, ECPP, ATES
Protocolo en Español para las Entrevistas Cognitivas

PASO 1: BIENVENIDA E INTRODUCCION DE LA ENTREVISTA
Instrucciones para el entrevistador: El siguiente protocolo NO debe ser leído textualmente.
El texto en letra cursiva solo es contenido con el cual el entrevistador debe familiarizarse
antes de la entrevista. El texto entre paréntesis [] contiene instrucciones para el
entrevistador.
Hola, mi nombre es y trabajo para [COMPAÑIA]. Mucho gusto de conocerle.
Gracias por venir a ayudarnos el día de hoy.
[Platique de algún tema casual para establecer una buena relación con el participante.]:
Usted está aquí hoy para ayudarnos con un nuevo estudio de investigación que estamos
realizando para el Departamento de Educación de los Estados Unidos. Este estudio pregunta
acerca de [la educación de sus hijos, la búsqueda y selección de cuidado infantil para los padres
con hijos de cinco años de edad o menores, o acerca de sus certificados o licencias].
Como parte de este estudio, le voy a pedir que complete una encuesta como que si estuviera
completando una encuesta real en su casa. No existen respuestas correctas o incorrectas y está
bien si usted no sabe algunas de las respuestas. A medida que usted vaya respondiendo a las
preguntas, yo le voy a hacer algunas preguntas acerca de sus respuestas. Esto nos ayudará a
que las preguntas en la encuesta sean mejores y más claras.
Debido a que la información que usted nos dará es muy importante, voy a estar tomando notas
mientras estamos hablando [INSERTE SI HAY UN OBSERVADOR: y mi colega también va
a observar nuestra sesión de hoy].

Además, esta entrevista será grabada en audio para que los investigadores del estudio puedan
revisar la grabación más tarde. Cualquier información que se refiera a usted o a su familia
personalmente, tal como su nombre [o la escuela de su hijo(a)], no será incluida en nuestro
informe y sus respuestas serán utilizadas únicamente para fines de investigación.
Recuerde que esto no es un examen. Queremos que nos diga lo que usted piensa acerca de las
preguntas de la encuesta.
Si en algún momento usted desea parar, solo déjemelo saber.
[Si por algún motivo el participante ya no tiene interés en participar, agradézcale por su tiempo y
termine la entrevista.]
Antes de que continuemos, ¿tiene alguna pregunta sobre lo que acabo de decir? [Responda a
cualquier pregunta que pueda hacer el participante.]
21

DELE AL ENCUESTADO UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO. PEDIRLE
QUE LEA LA FORMA (O LEASELA, DEPENDINEDO DE LA PREFERENCIA DEL
ENCUESTADO) RESPONDA CUALQUIER PREGUNTA Y PEDIRLE QUE FIRME LA FORMA.
DELE UNA COPIA DEL FORMULARIO DE CONSENTIMIENTO AL ENCUENTADO
SI EL ENCUESTADO HA DADO SU CONSENTIMIENTO EMPIECE LA GRABACION

PASO 2: PRACTICA DE PENSAR VOZ ALTA
Mientras esté completando la encuesta, quiero que hable acerca de cómo está contestando las
preguntas. Quiero escucharla(o) hablar acerca de cómo usted llega a sus respuestas para así
entender cómo mejorar estos materiales. Me interesa más saber cómo llega a sus respuestas que
en las respuestas que usted me dé. Para ayudarle a empezar a pensar en voz alta vamos a hacer
un ejercicio de práctica.
Primero, le voy a dar una demostración de cómo pensar en voz alta. Después le voy a dar una
oportunidad para que practique.
[Model the think aloud process for the respondent, including reading the question out loud.
Elaborate on counting all of the glasses of water as needed you can make up pieces of
information to emphasize the think aloud process e.g., “I don’t know if I should count the coffee
I drank in the morning, I drink three cups of coffee, I am going to count those since that is the
only liquids I have in the morning...” When you are modeling, write down the number of glasses
of water as you are thinking aloud.]
[Once finished the model practice think aloud]
Voy a darle un papel con una pregunta. Le voy a pedir que lea en voz alta la pregunta y que
piense en voz alta mientras decide su respuesta.
[Give respondent handout with think aloud practice.]
¿Cuántos vasos de agua tomó usted ayer?
Por favor dígame lo que está pensando mientras me da su respuesta.

Pregunte según sea necesario para obtener información detallada: ¿Cómo llegó a esa respuesta?
¿Qué estaba visualizando mientras estaba decidiendo qué responder? ¿Puede decirme más
acerca de lo que usted está incluyendo en su cuenta?
Cuando haya completado el ejercicio: Gracias. Ése es el tipo de detalle que busco cuando usted
me hable acerca de sus respuestas a las preguntas de la encuesta.
ENTREGUE AL ENCUESTADO UNA COPIA DEL CUESTIONARIO Y UNA PLUMA
O UN LAPIZ
22

PASO 3: RESPUESTAS A LA ENCUESTA
Ahora voy a darle la encuesta. Me gustaría que piense en voz alta mientras decide su respuesta.
Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma(o), como si yo no
estuviera aquí.
Recuerde que estoy muy interesada(o) en escucharle hablar acerca de lo que está pensando
mientras decide sus respuestas a las preguntas.
RECUERDE DECIRLE AL ENCUESTADO QUE ESTÁ DANDO INFORMACIÓN MUY ÚTIL
CUANDO PIENSA EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON DE
GRAN UTILIDAD.

Preguntas Útiles
En sus propias palabras, ¿qué dice esta pregunta?
¿Qué significa para usted la palabra [término] en esta pregunta?
¿Qué tan fácil o difícil es recordar [tema]?
¿Qué tan fácil o difícil fue escoger una respuesta?
Dígame lo que está pensando.
¿Cómo llegó a esa respuesta?
Usted dijo [respuesta]. ¿Puede decirme un poco más acerca de cómo llego a esa respuesta?
[En respuesta a las acciones o comentarios del entrevistado]: Por favor, dígame que sucedió.
Dígame que estaba usted haciendo.
[Si nota alguna señal visual, como cejas fruncidas o una expresión de confusión]: Dígame lo
que está pensando

PFI: Encuesta sobre la experiencia de los estudiantes y las familias con sus escuelas
(NHES41-BC)

Educación del niño
PÍDALE AL ENCUESTADO QUE POR FAVOR LLENE EL CUESTIONARIO. RECUERDELE
QUE TIENE QUE PENSAR EN VOZ ALTA, SI NO LO ESTÁ HACIENDO
HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO
Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el
encuestado tiene problemas con ellas.
Q1-Q2 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q3 ¿Es esta la escuela asignada del distrito?
[Preguntar solamente si el encuestado respondió esta pregunta] ¿Cómo llegó a esta respuesta?
¿Qué significa para usted “escuela asignada del distrito”?
Q4-Q10 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q11 ¿En qué mes de este año escolar comenzó este niño(a) a asistir a su escuela actual?
¿Cómo decidió la respuesta a esta pregunta? ¿Le resulta fácil o difícil contestar esta pregunta?
¿Por qué?
Q14. ¿Está este niño inscrito en algún curso avanzado llamado Advanced Placement AP en
la escuela?
Si respondió si, ¿me puede explicar sobre estos cursos en los que su niño(a) está inscrito? ¿En
qué pensaba cuando respondió a esta pregunta?
¿En qué grado académico se encuentran los estudiantes que generalmente están inscritos en un
curso avanzado en el distrito escolar de su hijo(a)?
Q15-21 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q22. ¿Está este niño(a) tomando algún curso relacionado con la escuela por el internet en lugar
24

de hacerlo de manera presencial con un maestro(a)?
Si responde si - por favor dígame un poco más acerca de esto. ¿Me puede decir porque usted
eligió esta respuesta?
¿Me puede describir a que se refiere?

Q23. ¿Cuál de los siguientes lugares proporciona esas clases?
¿Cómo decidió su respuesta?
[Si seleccionó una respuesta o la opción de “Algún otro lugar”] Por favor dígame un poco más
acerca de esto. ¿Cómo tiene acceso su hijo(a) a estas clases? ¿En qué lugar se encuentra cuando
está tomando dicha clase? ¿Quién provee esa clase?
Q24. No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de
pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un
problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados
Q25. Algunos padres deciden educar a sus hijos en el hogar en vez de mandarlos a una
escuela pública o privada. ¿Recibe este niño sus clases en el hogar en lugar de en la escuela
para algunas clases o asignaturas?
¿Por favor, me puede decir con sus propias palabras qué es lo que se le está preguntando aquí?
¿Cómo determinó su respuesta?
[Si respondió que “si”] ¿Me puede decir porque respondió si?
[Si respondió que “no”] ¿Me puede decir porque respondió no?

[Si se aplica] Q26. ¿Cuál de las siguientes frases describe mejor el arreglo de educación en
el hogar para este niño?
[Si el encuestado elige la respuesta a o b]
¿Porque eligió esta respuesta? ¿Me puede explicar cómo funciona este arreglo?
[Si el encuestado elige una de las siguientes respuestas: “Este niño no recibe la educación en el
hogar” o “Este niño asiste a la escuela pública o privada para todas las clases o asignaturas]
¿Porque eligió esta respuesta?
25

[Si no es evidente indague el significado de la escuela del hogar]

[Si se aplica] Q27. Pensando en todos los años en los que este niño(a) ha recibido educación
en el hogar, ¿en cuál de las áreas a continuación ha recibido este niño(a) enseñanza durante
su instrucción en el hogar?
[Se han añadido 16 opciones de respuesta a esta pregunta. Por favor anote si el entrevistado tiene
problemas con el número y el tipo de asignaturas enumeradas]
[Si no es evidente indague el significado de la escuela del hogar]
¿Cómo determinó cómo responder en cada una de las preguntas?
¿Cómo enseña las asignaturas durante el periodo de instrucción en el hogar?
¿Cree que falta algo?

[Si se aplica] Q30. Hay diferentes razones por las que los padres eligen la educación en el
hogar para sus hijos. ¿Eligió su familia la educación en el hogar para su hijo porque:
¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]?
Platíqueme un poco más sobre lo que estaba pensando cuando respondió las preguntas.
[Si no es evidente indague el significado de la escuela del hogar]
[Si se aplica] Q31. Entre las razones por las que su familia decidió educar a este niño(a) en
el hogar, ¿cuál diría usted que es la más importante para usted? Escriba la letra de la
pregunta 30 para la razón más importante por la que decidió educar a su niño(a) en el
hogar.
¿Qué tuvo en cuenta al elegir esta razón?
Platíqueme un poco más sobre lo que estaba pensando cuando respondió a esta pregunta.
Familias y Escuela
Q32-36. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Tareas
Q37-43. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Al final de la sección pregunte:
Quiero hacerle preguntas sobre algunas palabras que usamos en esta sección de tareas.
¿Me puede decir cuando en esta parte dice tarea o tareas a que se refiere?

Actividades en familia
Q44-46. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Salud del niño(a)
Q47 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de
pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un
problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q48. ¿Le ha dicho un profesional de la salud o de la educación que este niño(a) tiene alguno
de los siguientes problemas de salud?
[Note como responde el encuestado cada una de las preguntas de la a-m.]
¿Hay alguna palabra o frase que no le es familiar?
¿Hay algún problema de salud que su niño(a) tiene que no está listado?
Q49- 50. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que
indique un problema potencial, o si hay confusión o alguna inconsistencia en la
interpretación comparado con otros encuestados.
Q51. ¿Ofrecen estos servicios algunas de las siguientes fuentes?
[note como responde cada una de las preguntas de la a-d el encuestado]
27

¿Cómo determinó su respuesta?
¿Hay alguna palabra o frase que no le es familiar?
Q52. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de
servicios para la familia (IFSP por sus siglas en inglés), un programa individualizado de
educación (IEP por sus siglas en inglés) o plan de servicios?
[en esta pregunta estamos tratando de determinar que vocabulario utilizan los padres de familia
para referirse a el IFSP y al IEP]
¿Cómo determinó su respuesta?
¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus
siglas en inglés)?
¿Qué significa para usted un “programa individualizado de educación” o IEP?
Q53-56. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Historia del niño(a)
Q57-62. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q63. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado del
acuerdo de custodia)?
¿Cómo determinó su respuesta?
En sus propias palabras, ¿qué dice esta pregunta?
[Si la respuesta fue NO] ¿Qué significa “arreglos de custodia compartida”?

Q64. Si contestó afirmativamente, este niño(a) ¿pasa la mayor parte del tiempo en este
domicilio, otros domicilios, o el mismo tiempo en ambos?
¿Cómo determinó su respuesta?
28

Q66. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación
bilingüe o a un programa especial de inglés?
¿Cómo determinó su respuesta?
¿Hay alguna palabra o frase que no le es familiar?

Los miembros de su familia
Q68. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las
siguientes personas viven en el hogar con este niño(a)?
¿Cómo determinó su respuesta?
¿A quién está incluyendo en su cuenta?

La familia del niño(a)
Q71-115. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Su hogar
Q116-Q124. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

PFI: Encuesta sobre la experiencia de los estudiantes y las familias con sus escuelas: Escuela del
Hogar

ESTAS PREGUNTAS SON PARA LOS PAPAS QUE SUS HIJOS ESTAN EN LA
ESCUELA DEL HOGAR
Dígale al encuestado lo siguiente:

Estas preguntas son sobre su hijo que está en la escuela del hogar. ¿Qué edad tiene el
niño?

Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta
PIDALE AL ENCUESTADO QUE PORFAVOR RELLENE EL CUESTIONARIO.
RECUERDELE QUE TIENE QUE PENSAR EN VOZ ALTA, SI NO LO ESTA HACIENDO
HAGA PREGUNTAS CONFORME SIENTA QUE SEA NECESARIO
Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el
encuestado tiene problemas con ellas.
Escuela del Hogar
Q1-Q5 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q6. ¿Cuántas horas a la semana va normalmente este niño(a) a una escuela para recibir
instrucción? No incluya el tiempo que dedica a actividades extracurriculares.
¿Cómo determinó su respuesta?
[Verifique si la persona respondió a esta pregunta a pesar de haber indicado que el niño(a) está
en escuela domiciliaria (o escolarización en el hogar).
Q10. ¿Cuál de las siguientes afirmaciones describe mejor el estilo de enseñanza que se usa
para la educación en el hogar de este niño(a)?
¿Cómo determinó su respuesta?

Q17. Hay diferentes razones por las que los padres eligen la educación en el hogar para sus
hijos. ¿Eligió su familia la educación en el hogar para su hijo porque:
30

¿Cómo determinó cómo responder sí o no en cada una de las preguntas [a-j]?
Dígame un poco más sobre lo que estaba pensando cuando respondió estas preguntas.
Q18. Entre las razones por las que su familia decidió educar a este niño(a) en el hogar,
¿cuál diría usted que es la más importante para usted?
¿Cómo determinó qué opción escoger [a-j]?
Platíqueme un poco más sobre lo que estaba pensando cuando escogió esta opción.
[Si no es evidente, haga preguntas sobre el significado de la escuela del hogar]

Q20. Pensando en todos los años que este niño(a) ha recibido la educación en el hogar, ¿en
cuál de las áreas a continuación ha recibido este niño(a) enseñanza durante su instrucción
en el hogar?
¿Cómo determinó qué responder s en cada una de las preguntas?
¿Cómo cubre usted las asignaturas durante el periodo de instrucción en el hogar?
¿Hace falta algo?

La salud del niño(a)
Q27-30. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o s i hay confusión o alguna inconsistencia en la interpretación
comparado con otros encuestados.
Q31. ¿Ofrecen estos servicios algunas de las siguientes fuentes?
¿Cómo determinó su respuesta?
[Solo si el encuestado tiene a su niño(a) en una escuela privada]¿Qué significa para usted que
estos servicios se ofrezcan en la escuela privada del niño(a)?
Q33. ¿Se ofrecen alguno de estos servicios por medio de un plan individualizado de
servicios para la familia (IFSP por sus siglas en inglés), un programa individualizado de
educación (IEP en inglés) o plan de servicios?
¿Cómo determinó su respuesta?
¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus
siglas en inglés)?
31

¿Qué significa para usted un “programa individualizado de educación” o IEP?

Historia del niño(a)
Q38-42. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q43. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado
del acuerdo de custodia)?
¿Cómo determinó su respuesta?
En sus propias palabras, ¿qué dice esta pregunta?
[Si la respuesta fue NO] ¿Qué significa para usted arreglos de custodia compartida?

Q44. Si contestó afirmativamente, este niño(a) pasa la mayor parte del tiempo en este
domicilio, otros domicilios, o el mismo tiempo en ambos?
¿Cómo determinó su respuesta?
Q45. ¿Qué idioma habla este niño(a) principalmente en el hogar?
¿Cómo decidió su respuesta?

Los miembros de su familia
Q47. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las
siguientes personas viven en el hogar con este niño(a)?
¿Cómo determinó su respuesta?
¿A quién está incluyendo en su cuenta de las personas que viven en su hogar?
La familia del niño(a)
Q62. ¿Cuál es el grado o nivel escolar más alto que esta persona completó?
[Solamente si el respondiente escoge “diploma de estudios secundarios o equivalente”,
“diploma vocacional obtenido después de la escuela secundaria”, o “algo de
32

colegio/universidad, pero sin título”:]
¿Cómo determinó su respuesta?
¿Qué quiere decir para usted “diploma vocacional obtenido después de la escuela secundaria”?
Q63. Actualmente, ¿asiste esta persona a una escuela, universidad o centro de educación
para adultos o recibe educación vocacional o capacitación laboral?
En sus propias palabras, ¿qué dice esta pregunta?

ECPP: El Futuro de nuestros niños: Una Encuesta sobre el cuidado y la educación de niños
pequeños. (NHES-21BC(SP))

Estas preguntas son para los padres que tienen algún niño(a) menor de cinco años., que todavía
no está inscrito en preescolar (kindergarten).
Dígale al encuestado lo siguiente:
Estas preguntas son sobre su hijo que es menor de cinco años de edad, que todavía no está
inscrito en preescolar (kindergarten). ¿Qué edad tiene este niño(a)? ¿Qué tipo de arreglos u
opciones ha establecido para el cuidado de su hijo(s)? [Anote la respuesta]
Por favor responda las siguientes preguntas y acuérdese de pensar en voz alta
Cuidado Infantil y Programas para la Niñez
Q1-11 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q12. ¿Le ayuda alguna de las siguientes personas, programas u organizaciones a pagarle a
este pariente por el cuidado del niño(a)?
¿Me podría decir que información cree que esta pregunta le está pidiendo?
Si la persona contesto la pregunta Q12b. b. Asistencia temporal a familias necesitadas
(TANF)
Si el encuestado respondió sí. ¿Me puede describir este programa?
Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta?

Q13-Q16 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q17 Actualmente cuida a este niño(a), con regularidad, una persona que no es su pariente
ya sea en su casa o en otra casa?
Si el encuestado respondió sí. ¿Me puede describir el lugar?
Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta?
¿Qué significa para usted una persona que no es su pariente?
Q18-Q37 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Q38. (Si se aplica) ¿Es este programa de Head Start o de Early Head Start?
¿Me podría decir un poco más sobre qué le están preguntando aquí?
Si el encuestado respondió sí. ¿Me puede describir este programa?
Si el encuestado respondió no. ¿Me puede decir por qué respondió que no a esta pregunta?
Q39-48 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
[pregunte solo si no ha hecho estas preguntas en la pregunta Q12]
Q49 ¿Le ayuda alguna de las siguientes personas, programas u organizaciones a pagarle a
este pariente por el cuidado del niño(a)?
¿Me podría decir que información cree que esta pregunta le está pidiendo?
Si la persona contesto la pregunta Q49b. b. Asistencia temporal a familias necesitadas
(TANF)
Si el encuestado respondió sí. ¿Me puede describir este programa?
34

Si el encuestado respondió no. ¿Me puede decir porque respondió no a esta pregunta?
Q49-53 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Buscar y Seleccionar Cuidado Infantil para su Niño(a)
Q54. (Solo si no se le ha preguntado ya sobre el programa de Head Start) ¿Ha asistido este
niño(a) alguna vez a un programa de Head Start o de Early Head Start?
¿Me podría decir que información cree que esta pregunta le está pidiendo?
Si el encuestado respondió sí. ¿Me puede describir este programa?
Si el encuestado respondió no. ¿Me puede decir por qué respondió no a esta pregunta?
Q55-Q56 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q57. ¿Cuánta dificultad tuvo para encontrar el tipo de cuidado infantil o programa de
educación temprana que usted quería para este niño(a)?
¿Cómo determinó su respuesta?
¿Me podría decir un poco más sobre su respuesta?
¿Qué significa para usted el término “educación temprana”?
Q58. ¿Qué tan importante fue cada una de estas razones cuando usted seleccionó el
cuidado infantil o el programa en el que este niño(a) pasa la mayor parte del tiempo?
¿Hay alguna otra razón que es importante para usted y que no ha encontrado en la lista de
opciones?
Q59h. ¿El Sistema de Calificación y Mejoramiento de la Calidad (Quality Rating and
Improvement System, QRIS) de su ciudad o estado?
¿Me podría decir un poco más de su respuesta?

Q59k. ¿Está acreditado el proveedor del servicio de cuidado infantil?
¿Me podría decir un poco más de su respuesta?
Q60 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de
pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique un
problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Actividades en familia
Q61-67. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Cosas Que su Niño(a) Podría estar Aprendiendo
Q68-75. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Salud del niño(a)
Q76-80. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q81. ¿Ofrecen estos servicios algunas de las siguientes fuentes?
¿Cómo determinó su respuesta?
¿Qué significa para usted que estos servicios se ofrezcan en la escuela privada del niño(a)?
Q82. ¿Se ofrecen algunos de estos servicios por medio de un Plan individualizado de
servicios para la familia (IFSP, por sus siglas en inglés) o un Programa individualizado de
educación (IEP, por sus siglas en inglés)?
¿Cómo determinó su respuesta?
Si el encuestado responde si:
¿Qué significa para usted un plan individualizado de servicios para la familia (IFSP por sus
siglas en inglés)?
36

¿Qué significa para usted un “plan individualizado de educación”?
Q83-86 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Historia del niño(a)
Q87 – 91. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.
Q92. ¿El niño(a) vive en este domicilio y en otro domicilio (por ejemplo, como resultado
del acuerdo de custodia)? No incluya propiedades vacacionales.
¿Cómo determinó su respuesta?
En sus propias palabras, ¿qué dice esta pregunta?
[Si la respuesta fue NO] ¿Qué significa “acuerdo de custodia compartida”?

Q94. ¿Qué idioma habla este niño(a) principalmente en el hogar?
¿Cómo determinó su respuesta?
¿Hay alguna palabra o frase que no le es familiar?

Q95. Actualmente, ¿asiste este niño(a) a clases de inglés como segunda lengua, educación
bilingüe o a un programa especial de inglés?
¿Qué cree le está preguntando la pregunta?
¿Hay alguna frase o palabra que no le sea familiar en esta pregunta?

Los miembros de su familia
Q97. Además de los padres o tutores legales que ya se incluyeron, ¿cuántas de las
siguientes personas viven en el hogar con este niño(a)?

¿Cómo determinó su respuesta?
¿A quién está incluyendo en su cuenta de las personas que viven en el hogar?

La familia del niño(a)
Q100-138. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Su hogar
Q139-Q145. No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su
proceso de pensar en voz alta y solamente haga preguntas si el encuestado dice algo que indique
un problema potencial, o si hay confusión o alguna inconsistencia en la interpretación comparado
con otros encuestados.

Encuesta ATES

PASO 3: LLENADO de la encuesta
Ahora voy a darle una encuesta. Me gustaría que piense en voz alta mientras decide su
respuesta. Por favor lea en voz alta cualquier cosa que usted hubiera leído para sí misma, como
si yo no estuviera aquí.
Recuerde que estoy muy interesada/o en escucharle hablar acerca de lo que está pensando
mientras decide las respuestas para las preguntas.
RECUERDA DE DECIRLE AL ENCUESTADO QUE ESTA HACIENDO UN MUY BUEN
TRABAJO PENSANDO EN VOZ ALTA Y QUE SUS RESPUESTAS A LAS PREGUNTAS SON
DE GRAN UTILIDAD

Escolaridad
HAGA PREGUNTAS SEGÚN SEA NECESARIO
Instrucciones: No es necesario preguntar sobre las instrucciones, nada más anote si el encuestado tiene problemas
con ellas.
Q1-Q2 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz
alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay confusión o
alguna inconsistencia en la interpretación comparado con otros encuestados.
Q3. ¿Está actualmente inscrito en una universidad, escuela técnica o comercial, u otra escuela?
38

En sus propias palabras, ¿qué dice esta pregunta?
[Si respondió Si] ¿Puede decirme algo más sobre las clases que toma? ¿Cuál es el campo de estudio?
¿Dónde toma las clases?
¿Qué significa para usted medio tiempo y tiempo completo?
Q5. Desde que salió de la escuela secundaria, ¿ha tomado usted clases de lectura para mejorar cómo lee? No
incluya clases a nivel universitario.
¿A qué se refiere esta pregunta?
Certificación y Licencias
Q6. ¿Tiene usted una certificación profesional o una licencia estatal o industrial que este actualmente activa?
No incluya licencias comerciales, tales como de licor o de máquinas expendedoras (vending machines).
[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.]
¿Me puede decir que cree que esta pregunta le esta preguntando?
Q7. (si APLICA) ¿Si contestó que sí, cuantas certificaciones y licencias tiene usted que estén actualmente
activas?
¿Cómo determinó su respuesta?
¿Qué significa para usted el término “actualmente activas” en esta pregunta?
En sus propias palabras, ¿qué dice esta pregunta?
Q8. ¿Cuál es el nombre de su certificación o licencia más importante?
¿Cómo determinó su respuesta?
¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la más importante?
Q9. ¿Para qué tipo de trabajo es su certificación o licencia más importante?
¿Qué tan fácil o difícil fue para usted recordar el tipo de trabajo para el que la certificación o licencia le sirve?
Q10. ¿Es su certificación o licencia más importante requerida por un gobierno federal, estatal o local para
realizar este trabajo?
En sus propias palabras, ¿qué dice esta pregunta?
¿A qué se refiere esta pregunta cuando dice “requerida”?

Q11. ¿Le podrían quitar o suspender su certificación o licencia más importante por alguna razón?
En sus propias palabras, ¿qué dice esta pregunta?
¿A qué se refiere esta pregunta cuando dice quitar o suspender?

Q12. ¿En qué año obtuvo su certificación o licencia más importante?
¿Cómo determinó su respuesta?
¿Qué tan fácil o difícil fue para usted recordar en qué año obtuvo la certificación o licencia más importante?
Q13a. ¿Se preparó usted para obtener su certificación o licencia más importante realizando alguna de las
siguientes actividades? Tomando clases en una universidad , escuela técnica o escuela vocacional (escuela para
aprender oficios)
¿Cómo determinó que contestar en esta pregunta?
Q14. ¿Es su certificación o licencia más importante para su empleo actual?
¿Cómo determinó qué contestar en esta pregunta?
En sus propias palabras, ¿qué dice esta pregunta?
[SI contesto afirmativamente] ¿Cómo puede usted determinar si la certificación o licencia es requerida para su
trabajo actual?
Q15. ¿Qué tan útil ha sido su certificación o licencia más importante para cada una de las siguientes
situaciones? (a-d)
¿Qué tan fácil o difícil fue para usted contestar solamente sobre la certificación o licencia más importante para cada
una de las opciones?
Q16. ¿Tiene otra certificación o licencia actualmente activa?
¿Cómo determinó qué certificación o licencia poner aquí?
Q17-Q22 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en
voz alta y solamente haga preguntas si el encuestado dice algo que indique un problema potencial, o si hay
confusión o alguna inconsistencia en la interpretación comparado con otros encuestados.
Q22a (Si aplica). ¿Se preparó usted para obtener su segunda certificación o licencia más importante
realizando alguna de las siguientes actividades?
22a ¿Tomando clases en una universidad, escuela técnica o escuela vocacional (escuela para aprender
oficios)?
¿Cómo determinó qué contestar en esta pregunta?
Q23 (Si se aplica). ¿Es su segunda certificación o licencia más importante para su empleo actual
¿Cómo determinó qué contestar en esta pregunta?
¿Con sus propias palabras, describa qué es lo que le están preguntando?
Después de la pregunta 23
¿Qué tan fácil o difícil fue para usted contestar solamente pensando en la certificación o licencia que usted decidió
fue la segunda más importante?
¿Mientras contestaba, pensó en sus otras certificaciones o licencias?

Q24. ¿Qué tan útil ha sido su segunda certificación o licencia más importante para cada uno de las siguientes
situaciones? (a-d)
¿Qué tan fácil o difícil fue para usted contestar solamente sobre su segunda certificación o licencia más importante
para cada una de las preguntas?
¿Me podría decir cuál piensa usted que es la diferencia entre las respuestas de estas categorías? ¿Fue fácil o difícil
seleccionar la respuesta?
Q26. (Si aplica). ¿Cuál es el nombre de su tercera certificación o licencia más importante?
¿Cómo determinó qué contestar en esta pregunta?
¿Qué tan fácil o difícil fue para usted determinar qué certificación o licencia es la tercera más importante?
Q27. (Si se aplica). ¿Para qué tipo de trabajo su tercera certificación o licencia más importante?
¿Cómo determinó qué contestar en esta pregunta?
Q29. ¿Es su tercera certificación o licencia más importante requerida por un gobierno federal, estatal o local?
¿Con sus propias palabras, describa qué es lo que le están preguntando?
Q25-29 (Si aplica)
¿Qué tan fácil o difícil fue para usted contestar solamente pensando en su tercera certificación o licencia más
importante?
¿Mientras contestaba, pensó en sus otras certificaciones o licencias?

Certificados
Q30. Las personas a veces reciben certificados por completar un programa educativo o de capacitación. Éstos
son diferentes de una certificación o licencia. No incluya aquí certificaciones/ licencias. ¿Alguna vez ha
obtenido usted algunos de los siguientes tipos de certificados?
[Observe como reacciona el encuestado y note cualquier confusión o preguntas que tenga.]
¿Me puede decir que cree que esta pregunta le está preguntando?

Q30d. Un certificado—no una titulación—por completar un programa en una universidad técnica o de la
comunidad, o alguna otra escuela después de la escuela secundaria. No incluya certificados de maestro o
títulos universitarios?
¿Porque eligió esta respuesta (si/no)?
Q31. (Si contestó que sí.) Nos referiremos a los certificados en la pregunta 30d como certificados ‘postsecundaria’. ¿Cuál fue el campo de estudio de su último certificado de post-secundaria? (Marque UNA casilla
solamente.)
¿Pudo encontrar alguna opción que encaje con su situación o cree que falta alguna opción?
¿Me podría hablar un poco más sobre su respuesta?

SI contesto “otro” ¿podría decirme más sobre su respuesta?
Q34. ¿Cuál de las siguientes opciones fue un requisito para matricularse en el último programa de certificado
post-secundaria?
¿Con sus propias palabras, describa qué es lo que le están preguntando?
¿Me podría hablar un poco más sobre su respuesta?
Q35. Para obtener su último certificado de post-secundaria, ¿tuvo que completar…? [a-b]
¿Cómo determinó las respuestas a las preguntas (a-b)?
Q37. ¿Está su empleo actual relacionado con su último certificado de post-secundaria?
¿Cómo determinó qué contestar a esta pregunta?
Q38 No es necesario indagar sobre esta pregunta. Deje que el encuestado continúe su proceso de pensar en voz alta
y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna
inconsistencia en la interpretación comparado con otros encuestados.

Programas de Experiencia laboral
Q39 ¿Ha completado alguna vez una pasantía o práctica profesional, programa de cooperación educativa,
practicum, asistente jurídico (clerkship), residencia médica, experiencia clínica, programa de aprendiz o
similar?
[Si aplica] Me podría decir sobre su programa:
¿Para qué trabajo fue?
¿Puede describir un día típico en el programa?
¿Qué tipo de organizaciones estaban involucradas y cuáles eran sus roles?
Q40. ¿Si su respuesta es sí, para qué tipo de trabajo fue su último programa de experiencia laboral?
¿Me podría decir un poco más acerca de cómo seleccionó su respuesta?
¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difícil fue seleccionar su respuesta?
Q41. ¿Cuánto tiempo duró su programa de experiencia laboral más reciente?
¿Cómo llegó a su respuesta?
Q42. (Si aplica) ¿Qué sueldo ganaba como parte de su último programa de experiencia laboral?
[Si no contesto rápidamente] ¿Pudo encontrar una respuesta que estuviera acorde a su experiencia? De no serlo así,
¿cuál hubiese sido una mejor opción para usted?
[Si ninguna opción fue seleccionada] ¿Me podría hablar un poco más sobre el salario que recibió?
Q43 a-c. Como parte de su último programa de experiencia laboral, ¿realizó usted alguna de las siguientes
actividades?
42

¿Cómo llegó a su respuesta?
(Si contestó Sí a alguna de las preguntas) ¿Me podría hablar un poco más sobre la relación entre su programa de
experiencia laboral las clases, su entrenamiento, o el estudio que llevó a cabo?
Q44 a-d. ¿Describe alguna de las siguientes actividades su programa de experiencia laboral más reciente?
¿Cómo llegó a su respuesta?
Q44a (Si contestó Sí) ¿Me podría hablar sobre las evaluaciones?
Q44c (Preguntar solo si contestó Sí) ¿Qué significa para usted tener “clasificación de obrero especializado”?
Q45. ¿Cuál de las siguientes opciones describe mejor su último programa de experiencia laboral?
¿Me podría decir cómo un poco más de cómo seleccionó su respuesta?
¿Encontró una categoría que refleja su experiencia? ¿Qué tan fácil o difícil fue seleccionar su respuesta?
Q46. (Si aplica) ¿Considera que su último programa de experiencia laboral le ayudó (o le ayudará) a lograr
una certificación o licencia profesional?
¿Me podría decir con sus propias palabras, que es lo que le están preguntando?
Q47a-c. (Si aplica) ¿Qué tan útil fue su programa de experiencia laboral más reciente para cada uno de las
siguientes opciones?
¿Cómo seleccionó su respuesta para cada una de las preguntas?
¿Me podría decir cuál cree usted que es la diferencia entre las categorías de las diferentes respuestas?
¿Fue fácil o difícil seleccionar su respuesta?
Q48. (Si aplica) En su trabajo actual, ¿qué tan a menudo usa las habilidades o el conocimiento que aprendió
durante su último programa de experiencia laboral?
¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando?
¿Me podría hablar más sobre lo que usted estaba pensando cuando respondió esta pregunta?
Q49. (Si aplica) ¿Está su trabajo actual relacionado con su último programa de experiencia laboral?
¿Me podría decir, con sus propias palabras, qué es lo que le están preguntando?

Empleo
Q50-Q67 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en
voz alta y solamente indague si el participante dice algo que señale un problema potencial, o si hay confusión o
alguna inconsistencia en la interpretación comparado con otros encuestados.
Q68. (Si aplica) ¿Tenía usted una licencia requerida por una agencia del gobierno federal, estatal o local para
realizar este trabajo?
¿Con sus propias palabras, describa qué es lo que le están preguntando?

[Verificar si es congruente con las respuestas anteriores]
Q69-70 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz
alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna
inconsistencia en la interpretación comparado con otros encuestados.

Historial
Q71-84 No es necesario indagar sobre estas preguntas. Deje que el encuestado continúe su proceso de pensar en voz
alta y solamente indague si el encuestado dice algo que señale un problema potencial, o si hay confusión o alguna
inconsistencia en la interpretación comparado con otros encuestados.

PREGUNTA FINAL
¿Hay algo más sobre lo que le gustaría hablar acerca de estos materiales? ¿Hay algo acerca de lo que
no ha tenido oportunidad de hablar?

PASO 4. GRACIAS POR PARTICIPAR
Hemos terminado. Gracias por ayudarnos. La información que nos ha dado será muy útil para mejorar
las cartas y la encuesta.

Instrucciones para el entrevistador: Recoja los materiales y acompañe a los encuestados al
vestíbulo. Por favor dele a los encuestados el sobre con el incentivo, y agradézcales por su
tiempo y su ayuda.

File Type	application/pdf
File Modified	2015-06-03
File Created	2015-06-03