Developmental Studies Results
Appendix 5 NHES 2019 Developmental Studies Results.docx
National Household Education Survey 2019 (NHES:2019)
Developmental Studies Results
OMB: 1850-0768
National Household Education Survey 2019 (NHES:2019)
Full-scale Data Collection
OMB# 1850-0768 v.14
Appendix 5 – NHES 2019 Developmental Studies Results Summaries
June 2018
revised July 2018
Table of Contents
Research and Design Work in Preparation for NHES 2019 1
2017 National Household Education Surveys Program (NHES) Usability Testing (OMB# 1850-0803 v.157) 2
2017 National Household Education Survey (NHES) Web Data Collection Test (OMB# 1850-0803 v.163) 15
NHES 2019 Spanish Materials Focus Groups Round 2 - Recruitment (OMB# 1850-0803 v.229) 108
NHES 2019 Spanish Materials Focus Groups Round 2 (OMB# 1850-0803 v.232) 108
NHES 2019 Screener Cognitive Interviews - Recruitment (OMB# 1850-0803 v.230) 112
NHES 2019 Screener Cognitive Interviews (OMB# 1850-0803 v.231) 112
Research and Design Work in Preparation for NHES 2019
In preparation for NHES 2019, several research projects were undertaken. Some projects are in progress during the time of preparation of this submission. These projects were designed to improve various aspects of the methodology, implementation, and content of the NHES 2019 recruitment and data collection effort. Presented in this document are summaries from each of these research projects, which are excerpted from their respective full reports. Text that includes direct quotations and participant direct identifiers has been deleted from this document.
2017 National Household Education Surveys Program (NHES) Usability Testing (OMB# 1850-0803 v.157)
Memo
To: Christine Cox, and Carolyn Pickering, ADDP
From: Erica Olmsted-Hawala, Christopher Antoun, Brian Falcone, Rebecca Keegan, & Lin Wang, CSM
Date: November 4, 2016
Re: NHES Usability Quick Report
Introduction
This document provides a summary of usability issues observed during the evaluation of the NHES web instrument. First, we describe some usability successes. This is followed by a description of several general usability issues as well as some issues that were specific to the mobile Web version of the survey. We conclude by describing several usability issues involving the way that respondents understood/interpreted the survey questions.
Usability Successes
--Look and feel. Several participants said they liked the look of the screens and that they looked professional. One participant specifically mentioned she liked the lilac color.
--Mapping of response options to real-life circumstances. A participant who was self-employed indicated that the response options nicely fit her situation. Another participant who homeschooled her children also said that the people creating the questions had “done their homework.” Implying that the answer options fit homeschooling situations.
--Links. The link “Survey Length” worked for participants that were attempting to find out about how long the survey is. One slight caveat is that the name of the Survey Length page is “Burden Statement” and that has little meaning to participants’. Another caveat is that some first clicked on the FAQ link at the top or suggested they would call the help number.
-- Security question re-entry. One participant who had the screener with the security questions failed to write down her pin. The security question re-entry worked well for her.
-- School look-up feature.: We asked several PFI participants to look up specific schools and they were able to do it without any problems.
--Save and continue later. When the moderator stopped the participant and asked what he or she would do if they needed to take a break and continue working on the survey at a later time, most participants using laptops were able to find the “save and logout” link at the top of the page. Many participants using smartphones did not immediately click on the three bars at the top of the page because they didn’t know it was a menu, but eventually did try tapping them.
--Survey length and FAQs: When the moderator stopped the participant and asked what he or she would do with questions on the length of the survey or general questions about the survey, participants responded by doing a one or more of the following: clicking on the survey length link at the bottom of the page, clicking on the FAQ link at the top of the page, suggesting they would call the number listed on the main screen, or doing a google search on NCES.
--Security question re-entry: The one participant (who happened to be assigned the screener version that allows creation of a security question) who had forgotten to write down her PIN was able to log back in using her security question without any problems.
--Login Screen. Several participants expressed that they liked that the ID numeric fields auto-tab from one field to the next.
--General. One participant commented that they really like that some of the instructions were italicized. This allowed them to immediately recognize text as instructional and they could skim or skip them when they did not need the instruction to answer the question.
General Usability Issues (Mobile and Desktop)
--Order of topicals. The topicals were in an unexpected order. We expected the participant would be asked the topical ATES first and then asked the PFI topical second. However, the opposite occurred. The topicals ATES / PFIHS were in the expected order. We are not sure if this is intentional or not. Recommendation: Verify the topicals are appearing in the order you intend.
--Several questions are asked multiple times. Participants who have the dual topicals of ATES and one of the PFI/HS topicals or have ECPP and one of the PFI/HS are experiencing many screens that are repetitions. Several participants complained about the questions being repetitive. This is likely because they answered the background questions twice (e.g., marital status, ethnicity, race, education, relationship to the child, age (via the DOB question), etc.) In addition, for ECPP, they have answered other questions twice (e.g., about number of times they have eaten dinner together in the past week, their annual household income, whether they rent or own question and how long they have lived in their current house). For the ECPP/PFI/HS repetition, it caused confusion about what was happening with the survey – where participants verbalized whether some error was happening with the survey itself. For some participants this repetition of questions has been quite frustrating with two participants so far spontaneously verbalizing that they would not continue the survey if it was repeating the questions. In addition one participant skipped a set of questions that she said she had already answered. Recommendation: Un-duplicate the questions. If the respondent has already answered the question in the earlier topical or screener, do not repeat them later in the next topical.
-- Other-specify boxes. Many participants attempted to tap on the other-specify boxes before first tapping on the “yes” radio button. The box didn’t respond. One participant said “I am clicking on other and it’s not working.” Some participants then gave up on the box while others realized that they needed to tap the “yes” answer. This design of first needing to tap the “yes” prior to activation is problematic in the desktop version but is worse in the mobile version because the “other specify” boxes are sometimes above both the “yes” and the “no” response options. In these mobile cases, it appears like the box is broken when a participant taps on it and nothing happens, and they don’t see the “yes” and “no” choices because they are out of the line of sight. When participants using laptops clicked on the box, a “no symbol” (a circle with a diagonal line through it) appeared. Most participants (but not all) did eventually realize that they had to click on the “yes” before gaining access to the box, but it caused frustration to them. Finally, the other-specify box layout for mobile is inconsistent. In PFIHS, both the “yes” and “no” options are below the box and are out of the line of sight of the participant. But in the ATES design, the “yes” is on top the box and then the “no” is below it.
Recommendation: At minimum, for the mobile display, move the other-specify box to in-between the “yes” and “no” options as it is on some of the topical screens. Additionally, if a participant taps or clicks into the “other specify” box prior to tapping or clicking on the “yes” radio button, allow the respondent to write in the box and automatically mark the “yes” radio button. (Note: this is the strategy that ACS went to after we noticed the same participant behavior of first tapping on the box rather than on the radio buttons.) This is the optimal solution for both desktop and mobile displays.
--Estimated survey length. During the session when the moderator interrupted the participant to ask the participant to find out how long the survey is, some of the participants commented that the specified length of 8 minutes is not accurate (all participants in the usability test are in the dual topical assignment, so their time in the survey is quite a bit longer). Recommendation: Provide an accurate estimate of the average survey length.
--FAQ page. When participants navigated to the FAQ page, it was not immediately obvious to them how to return to the survey. On desktop, an error message appeared after the participant clicked on the “close” link. See Figure 2. After one participant clicked “close” and saw the “error” message, said she “didn’t want to close the window.” Instead she redirected herself to the tabs at the top of the page, leaving the FAQ open in another tab. She was able to redirect herself to the survey using the tabs at the top of the page, but this may not be a strategy that all respondents know.
On mobile phones, the close link is at the bottom and not all participants immediately saw it. The standard for mobile is to include a small “X” in the top right corner to close the window.
Figure 2. Screenshot of the desktop view after participant has clicked “Close” in the FAQs.
Recommendation: For desktop remove the message that reads like an error so that the window simply closes after the respondent clicks on “Close.” For Mobile, follow what is standard and use the “X” in addition to the “Close” link at the bottom of the page. See an example from the ACS mobile design in Figure 3.
Figure 3. Screenshot from ACS with small “x” in top right corner. In the middle, top of NHES FAQ page without “close” option. On the right, bottom of NHES FAQ page without “close” option.
--Numeric entry. A number of participants attempted to enter decimals when answering time questions, such as the number of hours of homework their child does per week. We have seen participants enter in “125” when meaning to enter “12.5” because the decimal didn’t appear. Participants corrected their error and rounded up, but it is possible that some respondents will not realize that the decimal doesn’t appear and report a much larger number than they intend. Recommendation: Consider allowing decimals.
--Color contrast. The color contrast for the open menu is not strong enough. The black links on the blue background are difficult to read. Recommendation: Make it a brighter color contrast so links are visible. See Figure 4
Figure 4. Screenshot of Mobile display of open menu. Black on dark blue is not enough of a color contrast.
--Work experience programs. On ATES, at least two participants attempted to select more than one option for work experience programs. Recommendation: Consider if allowing this is an option.
--Software bugs to fix:
On the screen pfi_fhhome (“how often does [fill childs name] do homework at home?”), the last response option is repeated with a code script of “/>” showing up.
On the screen pfihs_dualparent, the question (“Is the parent or guardian of [fill childs name]?”) is missing a word. It is particularly confusing when the referenced person is in PFIHS and is referred by an initial only.
In dual topicals, when the participant answered that they were born in the U.S., the question about when the participant moved to the U.S. appeared when it should have been skipped. This occurs in the Child’s Family section: where the participant is asked “How old were you when you first moved to the 50 United States” after answering in the previous question that she was born in the US. (Screen name pfihs_p1 agemv).
We noticed a fill that is not working. When answering the ECPP survey questions about people’s relationship to the child, it asks whether “person 2” is the parent or guardian of [child’s name], instead of filling Person 2’s name.
If the participant has made responses on a page and then uses “save and logout” the information they entered on the page they were working on is not saved. This could be frustrating for screens with many questions or response options to have to re-do. Recommendation: Save the information that has been entered on the page when “save and logout” is used.
Mobile Usability Issues:
--Problem with Windows Phone DOB screens: On the Windows phone (likely also on the Windows tablet which we didn’t test), the DOB dropdowns did not work as expected. Typically the question stem appears first and then once the participant taps into the field the pick list (Android phones) or wheel (IOS phones) appears. In the instance of the Windows phone however, the pick list showing a list of dates starting with “2017” appears and the participant does not know what the question is asking. There is no context for the pick list as the question stem has not been viewed yet. The one participant who had a windows phone said she thought that the pick list starting with a list of current years was asking her for the current year. Only after she selected “2016” did the pick list go away and then she saw the question and had to go in and make the change. Please note – this occurred for all the DOB dropdown questions on the Windows phone but it did NOT occur for the one other dropdown in the topicals we saw (number of times you eat together as a family). Recommendation: Fix the code so that for Windows mobile platforms, the pick list shows up only after the respondent taps into the DOB dropdown field.
--Missing response option label. There are a number of instances in the different topicals (we have seen it in PFIHS and PFI and ECPP) where the “No” is missing from a series of Yes/No questions. Screens where we have noticed it to occur include: Food stamps; Medicaid; CHIP; Section B; Food stamps ; Child’s health; and Making friends.” Recommendation: Find all instances of the missing “No’s” and fix them.
--Radio buttons on iPhones. On iPhones the selected radio button is small and hard to see. This makes it unclear whether a participant has selected a “yes” or “no” option. ASD has programed other radio buttons for mobile phones that work for both android and iPhones, see Figure 7, far left for a current ASD design that works on iOS devices. The images on right are two iterations of iOS designs that do not work optimally. Recommendation: Make it more obvious that a radio button has been selected with visual cues like in the survey pictured below on the far left.
Figure 7. On left, screenshot of a selected radio button for another survey that ASD created for CSM; in the middle, the tiny radio button on mobile iPhone screens, from expert review rounds;
--Security warning. A foreboding warning was displayed after participants loaded the test URL on a Chrome mobile browser (or after they clicked the links to “Survey Length” and “Security”). This happened with other mobile browsers but the warnings used weaker (less frightening) language. Recommendation: Ensure the actual production URL (and the links to “Survey Length” and “Security”) work in mobile Chrome and other mobile browsers without flagging a security warning.
-- Menu icon. The meaning of the menu icon -- three lines (also known as the hamburger menu) -- is not universally obvious. Some know the icon and click on it, others do not know that it means “menu.” Not labeling the three bars on smartphone is a problem. One participant said “I’m not sure this is a menu but I am just going to guess.” Recommendation: Replace the three lines with the word “Menu” or keep the icon but label it by adding the word “Menu” underneath it.
Another issue with the menu is that once it’s opened it is difficult to close. Tapping someplace else on the screen (aside from the menu) does not close it, which is how most menus operate. When one participant attempted to close it she accidently hit “save and log out,” which led her down a sub-optimal path. Recommendation: Have the mobile menu follow typical “close” behavior so that when you tap anywhere else on screen it closes the menu. See Figure 8.
Figure 8. Screenshot of some examples with the word “Menu.”
--Spacing on long scrolling pages. The spacing on the mobile grids was not consistent. (On mobile, the grids now appear as many questions on the same screen, which is fine and works well for mobile devices. However we noticed that for at least two of the “mobile grids" on the PFI topical (satisfaction with school and health issues with the child) the questions appeared to be too closely located together. One participant missed answering one of the questions but she did notice after she had scrolled up and checked the page. So, she was able to correct the missing data, but others may not notice. Other grids in the instrument (including the mobile grids in the ATES) appeared better spread out. For example, the “Yes” and “No” answers are too close together, and needs additional spacing. At minimum, additional spacing should aid in this issue. Some of the ATES spacing is appropriate. Note: it appeared that when there was a lot of content it was more crammed together – versus when there were only a few response options, it was better spaced. We have noticed that participants don’t mind the scrolling so adding in a little space for visual aid is an optimal solution. Recommendation: Make all mobile yes/no questions have a consistent amount of spacing, specifically increase the spacing in the two "mobile grids" in the PFI topical mentioned above. In addition, spacing between the last response option for a question and the next question-stem should be larger than the space between other response options within that question.
--Formatting of response option text. The spacing was off on longer response options that spanned two lines. The second line appeared in between radio buttons. This makes the screen look cluttered. Recommendation: For responses that span two lines, indent the second line to line up with start of first line above.
--Visibility of menu header. When participants scrolled down on the mobile page, and the moderator asked them to find out more about the survey, they sometimes failed to see the menu header (because it was hidden from view). On some mobile applications, a user can touch on the screen (but not on a response option) to make the header appear. Once the user taps on the screen again, the menu header disappears. Recommendation: Consider adding in this extra functionality.
--Wrong virtual keypad. The QWERTY keyboard (with letters) appeared after participants tapped into some of the numeric only entry fields. Recommendation: For all fields where only numbers will be entered, have the number pad come up. This leads to easier data entry and less chance of incorrect buttons being hit. See Figure 10.
Figure 10. Screenshot example of a keypad keyboard for numbers only fields.
--Alignment of “Don’t know” option. For the questions about the “Parent 2,” the “Don’t Know” label was in an odd alignment; is seemed to be misaligned with the checkbox icon and the text field. Recommendation: Review the location of the “Don’t Know” and reposition for the small screen.
--Volume of text on the login screens. A few participants commented on the excessive amount of text on the initial Login and PIN screens. See Figure 12. In addition, when participants logged back into the survey, the PIN screen repeated the first paragraph of text and User ID information, unnecessarily pushing the PIN entry box down the screen. Recommendation: Consider reducing the amount of text on the initial screens.
Figure 12. Screenshot of the initial login page on Mobile. Participants commented on the excessive amount of text on this screen.
--Dropdowns. The dropdowns were rendered differently on different devices. Sometimes they were rendered as pick lists that took up the full screen (more recent Android platforms), other times they were displayed as pick lists on the bottom third of the screen (older Android devices), and still other times they were displayed as a wheel. For the DOB screens (the majority of the NHES dropdowns), we think the best design is the more recent Android platforms that display a pick list. There has been anecdotal evidence from other usability studies1 where participants struggled with the iOS wheel (either failing to see it entirely or, as occurred in the NHES dry run, failing to click on the “Done” button to save their answer). Recommendation: Consider using a code to override the default iOS dropdown display and opt to follow the more recent Android look where the dropdown becomes a pick list that takes up the full screen.
--Size of numeric entry fields. Some numeric entry fields are slightly too small. When a participant entered two digits, half of the first number became hidden from view (see Figure 14).. This was the case until participants tapped out of the field. One participant commented about this issue on the question, “How many years have you lived at that address.” She said that she wasn’t sure if the first digit was a 1 or a 4. Recommendation: Increase the width of the boxes slightly so that both numbers can appear once they are entered.
Figure 14. Participant types “40” hours. The “4” is cut off.
--Inconsistent entry-methods for Hours each Week questions between PFI and ECPP: As visible in Figure 15 below, the entry methods are different by topical. One participant that noticed this much preferred the keyed entry rather than having to scroll through a long list of numbers. Recommendation: Make entry methods consistent within and between topicals for similar questions. Preferably, keyed-entry for higher values to avoid excessive scrolling.
Figure 15. Hours per/each Week Questions. PFI used keyed-entry.
Usability issues involving understanding/interpreting the question or response options:
--For ECPP topical, we noticed that some participants expressed confusion when answering the child care segments. For example:
One participant struggled after answering that the “program” her child attended was pre-kindergarten because the survey seemed to be referring to the school as pre-kindergarten but when she reads “pre-kindergarten,” she is thinking of the actual class that her child is in, not the school. It was not immediately obvious from the survey question and the use of the “pre-kindergarten” fill whether, when answering the questions, she should be thinking of her child’s class only or the whole school. This became particularly confusing when the question asks for the number in years and months when she started going to “this particular prekindergarten.”
For at least two participants in the ECPP topical, they were not sure what exactly was meant by “REGULAR BASIS”. See Figure 16
On the same screen, one participant expressed confusion about the phase “receiving care”. For this participant, it was unclear how he should have answered the question. Recommendation: It could help to define “receiving care” and be clear if watching over the child is what is really meant.
--A number of participants commented that they did not like the way one of the “50 United States or the District of Columbia” was worded. One participant said it made it sound like DC was not a part of the U.S. Recommendation: Simply refer to it as the United States and put the extra information in parentheses – for example: “United States of America (one of the 50 states or the District of Columbia)”.
--For the ECPP topical, (ecpp_cphrs) one participant, when answering the question about the numbers of hours each week that the child attends the school, mistakenly reported the number of hours per day that her child is in care.
--On the question about whether the parent had told the child a story (not including reading to child) we heard participants spontaneously verbalize different answers to the same behavior. For instance, some verbalized that they had told their child about their day so they were answering “yes” because that is a story about their day. Other parents said that they didn’t consider telling their child about their day to be a story and so answered “no”. Recommendation: Consider clarifying what is meant by telling a story to the child.
--One participant who only used a first initial for her children said that the clause that included their birth year was at times long in the questions and did make a few of the questions more confusing than they needed to be. During debriefing she mentioned that she would not have used the initials in a government survey if it didn’t explicitly say that she could. As well, there is at least one question that repeats the clause twice in the same question which became very difficult to interpret. The participant suggested for that question to use the initial clause (e.g., “E” born in 2011), only once and the second time, simply use the initial and leave off the rest of the clause. In addition, another participant that used a single initial and was homeschooled said that she did not need to have the part of the clause include “being homeschooled,” the month and year of birth was more than sufficient. This additional part to the clause makes the question seem redundant. Recommendation: Consider removing the instruction to use initials. If you include the initials, do not repeat the clause twice in one question. Also do not include the information about being homeschooled in the clause.
--One participant said of the log in page sentence that reads “The questions are needed so that we can ask question that are right for your household” that she was “super confused by that sentence.” She said the repetition of the word “questions” did not make sense. See Figure 21. Recommendation: Remove the sentence or clarify it.
Figure 21. Screenshot from mobile with confusing sentence circled in blue.
--One participant in the PFIHS topical asked whether the sports activities (such as gymnastics or swimming) count as private tutors.
--One participant was confused by the certificate question about GED because she wanted to make sure that it was known that she had a high school diploma.
--One participant initially interpreted “work experience program” to mean “work experience” and then went back and changed her answer..
--A few participants have commented that the answer option “Now married” is odd. Participants thought it should just say “married” without the “Now.” In addition, the question stem uses the word “current” so to repeat “Now” in the response option is redundant. See Figure 22. Recommendation: Remove the word “Now.”
Figure 22. Screenshot of laptop view of marital status question with “Now” in front of married response option.
--For the employment question, one participant wanted to know whether to count all hours or just paid hours.
--One participant treated the question about different government assistance programs like it was a check-all-that-apply question, clicking ‘yes’ on some items and skipping the rest.
--In ATES situations where the person works both full and part time - remove the follow-up question.
-- The ECPP question (ecpp_cpunit) about the easiest way to “report” childcare expenses received mixed comments. Recommendation: consider replacing the word “report” with something more colloquial such as “tell us” and perhaps even mention something like this: “We are about to ask you about your child care expenses and want to know if it’s easiest for you to tell us about these costs in daily, monthly, or yearly increments.”
--It appears that for participants in the ATES topical, many seem to pause on the question that states what the difference is between a certificate and a certification. It is not exactly a clear distinction for participants. Participants are rereading this question multiple times and are still confused.
--One participant in the ATES topical had the same certification in multiple states. The question asks to put the “most important” certification first. They were not sure how to respond to this. Eventually, they decided to use the one that they use the most even though they considered each one equally important.
--One of the answers listed in the FAQ page that discusses why someone was not asked any questions about their child’s education may not make sense for people in a dual topical survey. Consider rewording that FAQ.
--In an ATES topical, one participant said that the questions about whether she has a license to do her job feels like it is redundant. She said that she was already asked this earlier in a different manner. The question reads, “do you have a license that is required by a federal state or local government agency to do the job”
--One participant commented on the letter that if he had received this in the mail he may not have completed it because its justification was too vague. At least one participant scrolled to read the warning message, exploring if it was a secure site before logging in. There is a more updated warning message that the decennial census is using, we recommend using that wording as it has been cognitively tested and performs better. See Figure 23.
Figure 23. Screenshot of updated wording in the Terms and Conditions that was tested in the Decennial 2016 test and is being used for future decennial test.
--For one participant, the instructions to enter “first name, initial, or nickname” on the first “About You” question are located just above the field and was seen easily. On the next screen asking about household members, the screen had these same instructions but for this participant it was buried in a large amount of text. The participant missed this part and commented that it was odd that they were allowed to enter only a nickname or initial for themselves but not for other household members. Consider placing these instructions in the same location as the first screen (just above field) and separate from the rest of the text.
--At least two participants commented that the sex question should consider adding in an “other” option.
--The wording of calendar year vs 12 month year was confusing to at least one participant.
--The question in ATES asking the respondent to report the highest degree received caused some confusion. At least two participants that had some graduate school but with no degree did not prefer to choose that response option because they verbalized that they had a BA degree they just did not finish graduate school. The wording “but no degree” made them think it would not count their BA degree, so they underreported the amount of schooling they had received.
--In the section on “Families and school” the screen “fcschool” the question is a Likert like scale that only offers four options. One participant indicated she wanted another “in-between” option (so not an even number of choices).
--In one of the questions on the PFI topical, the parent is asked about how the child feels about how much homework she has. The participant mentioned that her child doesn’t really talk about it – and there was no option for “I don’t know” or “No opinion.” In this instance, the participant skipped the question
--On the screen “Hdintdis” the first option (a) mentions “mental retardation.” No participant commented on this but it is possible this term may offend some people as it is not politically correct. It may be more appropriate to use the term “intellectual disability.” When we googled this term we found an article that suggests in 2010 President Obama signed legislation that requires the term “mental retardation” to be now termed “intellectual disability” for most governmental purposes. See https://www.disabilityscoop.com/2010/10/05/obama-signs-rosas-law/10547/ (American Psychiatric Association. (2013). DSM 5. American Psychiatric Association.)
--One participant during the second vignette, when attempting to learn more about the survey went to google and found the NHES main website. She proceeded to read about their overview and mission statement on http://nces.ed.gov/nhes/. It is possible other respondents will do this. Recommendation: Ensure the website is updated for the current survey.
----A number of participants in the PFIHS topical initially were not sure how to respond to the question about the tutor because they all mentioned aloud that they were in a co-op and they were not sure if the co-op should be counted as a tutor. The very next question is about whether the child is in a homeschooling co-op. While all participants were able to figure it out after receiving the next question, it caused extra frustration and time when they were deciding how to answer the first question (not knowing that their answer would be in the next question). Recommendation: As it appears that almost all the participants are in some type of co-op (more so than private tutorials) it may make sense to switch the order of the questions so that the question about co-ops come before the question about private tutors.
--The question about how many total hours each week the child was homeschooled took a while to count for some of the participants. In addition, based on what the participants spontaneously verbalized as they counted, it appeared that participants counted the hours differently. For example, some participants counted strictly only the hours they worked with their children and did not count the co-op hours or time that other people worked with their children. Other participants counted all the time that their child was in instruction including co-op settings, extracurricular type activities such as organized sports, etc. Thus the range of hours varied wildly by participants. Recommendation: Consider adding in some instruction on what the respondent should consider when making their answer.
--For two participants, one of whom had used both public/private schooling for some of their child’s education as well as homeschooling, paused on the question asking how many years had the child been homeschooled. The question starts with the phrase, “Thinking about typical grade levels for which grade levels was [fill of child’s name] home schooled?” This question did not seem to bother respondents who had homeschooled their children for every school year. It was not clear exactly why this question was confusing, but it is something to keep watching.
In ATES, a few participants hesitated on the question about maintaining or renewing their certificate/license. One participant said that if you have a current license it is implied that you are maintaining/ renewing it.
--One participant in the ATES wanted to know if volunteering should count as work experience.
--There is no question specific help (like ACS) but at least one participant, during debriefing, said she would prefer to have some specific help for some of the questions.
--A few participants verbalized that they would like a progress bar. This typically occurred after the moderator interrupted the participant to ask them to complete a specific task (e.g., save and logout, length of survey, etc.) Note: Research is mixed on whether having progress bar is beneficial for longer surveys.
--On one of the early PFIHS screens there is an incorrect grammatical possessive. Recommendation: Fix this error.
--For the PFI public school topical, some participants verbalized that being asked the name and address of their child’s school was too personal.
--At least one participant said that they wanted a text field that they could specify another language when “other” is selected.
--For PFIHS, many participants were unsure about what “another educational publisher” means for the question about educational materials. It was perceived as too broad or vague.
--For ECPP, many participants struggled with approximating the number of books that belonged to their child. Several participants said that this number was in the hundreds and was very difficult to try to estimate. One participant suggested that a set of different ranges (e.g. 0-10, 10-25, etc.) to choose from would make this question easier to answer.
--For PFI health response options, the “Another impairment lasting 6 months or more” option confused at least 2 participants. The word “another” implied to them that the other response options needed to last 6 months or more to qualify.
--A general issue is that participants were at times unsure about what question to expect next. Two participants asked why the survey didn’t ask about their other children. They were surprised when the survey ended before they had answered questions about all their children. If respondents outside of the lab also think that they will have to answer questions about all of their children, they may breakoff. We think it might help to provide some context for respondents about what to expect. For example, it would be good to mention at the very start of the survey that only one of their children is sampled (“We will ask you several questions about [fill for child’s name] and their education. We ask each family about only one of their children, and so will not ask detailed questions about your other children.”). This will also help respondents focus on that particular child while answering questions. If respondents are participating in two topicals, then this should also be mentioned at the start of the survey so that the transition from one topical to the next is less abrupt (If in dual topical of ATES and PFI/HS or ECPP: “First we will ask about your work and then we will ask about your child [fill]. OR if in ECPP and PFI/HS: “First we will ask about your child [ fill ECPP child’s name] and then we will ask about [fill PFI/HS child’s name]. We will only ask you questions about these two children.) In other words, any “signposting” or guidance to give respondents a sense of where they are in the survey and what is coming next would help.
2017 National Household Education Survey (NHES) Web Data Collection Test (OMB# 1850-0803 v.163)
Findings presented in Appendix 6: Web Test Results.
2017 National Household Education Survey (NHES) Web Data Collection Test Updated (OMB# 1850-0803 v.182)
Findings presented in Appendix 6: Web Test Results.
National Household Education Surveys Program 2019 (NHES:2019) Focus Groups with Parents of Students using Virtual Education (OMB# 1850-0803 v.186)
TO: Sarah Grady, National Center for Education Statistics (NCES)
FROM: Dr. Kathleen Mulvaney Hoyer and Dr. Dinah Sparks, Activate Research, Inc.
SUBJECT: Homeschool and Virtual School Focus Groups – Overarching Themes and Key Findings
DATE: June 1, 2017
Background
To inform item development for the Parent and Family Involvement in Education (PFI) questionnaire and to facilitate understanding of the education experiences of families who engage in homeschooling (HS) and virtual schooling (VS), Team Sanametrix conducted two focus groups facilitated by Dr. Luis Huerta of Teachers College, Columbia University.
One focus group consisted of HS parents who include VS in their homeschool curriculum. The second focus group consisted of parents whose children’s primary or secondary education mode was VS. Some of these parents had children who were enrolled in brick-and-mortar schools and supplemented with virtual courses; others had children who were enrolled in virtual school full-time.
Both focus groups covered a variety of topics: types of schooling and delivery of instruction; financial costs, expenses, or fees; information informing schooling decisions; reasons for engaging in their schooling arrangements; and feedback to questionnaire item language. Additionally, the VS focus group explored the notion of any overlap in identification of VS parents as HS parents.
Selected Themes and Findings
Themes
Several themes emerged from the focus groups, including the following:
All focus group participants use VS in some manner.
Experience and engagement with VS varies widely. For some participants, students engage with a small amount of virtual content or courses. For others, students’ engagement with virtual and non-virtual content and courses is relatively even. For still others, students are enrolled in a full-time virtual course of study.
Participant self-identification is not uniformly based on the mode of instruction. Parent preferences and perceptions determine the label participants use to describe their education experiences.
Participants use many terms to refer to VS (e.g., “online,” “virtual,” “cyber”).
Participants report a variety of reasons for using VS.
Findings
The focus groups produced sets of key findings (see Appendices A and B). These key findings directly influenced proposed changes to item language for the combined PFI questionnaire, as follows:2
Use the terms “online, virtual, or cyber” throughout the questionnaire to refer to the phenomenon of virtual schooling.
In items that attempt to determine families’ schooling arrangements (i.e., item 2 of the combined PFI), distinguish online, virtual, or cyber postsecondary institutions from postsecondary institutions located in physical buildings.
Modify item 12 to include language about “schooling arrangements” rather than “school” to acknowledge the range of schooling modes in which families engage.
Include new items (items 18 and 19) to investigate families’ reasons for engaging in virtual schooling.
Include a new item (item 22) to investigate the number of virtual courses in which the sampled child is enrolled.
Modify items that identify families as homeschoolers (items 38 and 39) to investigate families’ use of homeschooling and virtual schooling.
Modify item related to credits for homeschooling families who enroll in virtual education (item 42).
Modify items related to sources of homeschooling curriculum into two separate items (one about virtual resources and one about physical resources; items 48 and 49).
To continue the work of item development, Team Sanametrix will continue to investigate these proposed changes in cognitive interviews.
Appendix A: Key Findings from Homeschool Focus Group
Type of School and Instruction Delivery
Key themes:
Reliance on full- or part-time virtual. Some parents did not rely heavily on online schooling, whereas others did.
Most parents in this group rely heavily on parent-led instruction and use the online/virtual part of education for extra support or practice items. Sometimes the online/virtual education was a full course; in these cases, parents use the online/virtual resources for content that they are not comfortable teaching (e.g., more advanced than they prefer to teach). The subjects for which parents used online/virtual instruction varied and included science, literature, mathematics, and test preparation. Many parents indicated they used online resources for mathematics. These parents’ schedules included a mix of parent-led instruction, instruction by others in the homeschooling community and/or tutors, and extracurricular or nonacademic activities.
For those parents who rely heavily on online instruction, in this group, even when the children were taking up to 75 percent of their instruction online, they still considered themselves to be homeschoolers. They had many extracurriculars and some other academic instruction that was parent-led or done through a homeschool co-op/group.
Format of online instruction. Among parents whose children take a full course (whether it is just one course or several courses), the format included both synchronous and asynchronous instruction, with asynchronous being the common practice. Parents said that students had access to teachers.
Sources of curriculum, information, or support. The parents stressed the importance of the homeschooling community and indicated that homeschooling parents collaborate with and support one another. They generally do not think that the public schools serve as a support for them. In fact, several parents were unaware of the services that public schools are able/required to provide, such as PE courses. In one instance, the parent shared an anecdote about the public school not being aware of its obligation to provide testing for the homeschool student.
Products/services mentioned:
Teaching Textbooks (http://www.teachingtextbooks.com)
Easy Peasy (https://allinonehomeschool.com)
Aquinas Learning (https://aquinaslearning.org)
IXL (https://www.ixl.com)
Right Start
Khan Academy
Classical Conversations (https://www.classicalconversations.com)
Write from the Heart (http://writefromtheheartclasses.com)
Write from the Home (http://www.writeathome.com)
ABCya (http://www.abcya.com)
Homeschooling Buyers’ Co-op (https://www.homeschoolbuyersco-op.org
Art of Problem Solving (https://artofproblemsolving.com)
Center for Talented Youth (http://cty.jhu.edu)
MIT OCW (https://ocw.mit.edu/index.htm)
Coursera (https://www.coursera.org)
A Beka Academy (https://www.abekaacademy.org)
Kolbe Academy (http://www.kolbe.org)
Apologia (http://www.apologia.com)
Sonlight (https://www.sonlight.com)
Financial Costs or Expenses
Key themes:
These parents report that they pay out-of-pocket for materials related to their children’s schooling. They do not rely on subsidies. Several of them indicated that the cost is substantial but did not share dollar amounts. One participant observed that more experienced homeschool families are able to capture resources for free, while newer families are fine to “throw money at it.”
Information Informing Schooling Decisions
Key themes:
Source of information. Parents rely on their own research (online research and/or research into the local public schools) and on communications with others in the homeschooling community for information about homeschooling and online schooling (both informally as well as formally – for instance, at a convention).
Criteria for decision. In making decisions about the appropriate schooling for their children, these parents reported that they valued both academics and ethics/morals. Some parents actively avoid the larger publishers and school materials related to Common Core.
Reasons for School Choice
Key themes:
Reasons for homeschooling. These included:
Desire for family closeness and/or parental influence over children
Ability to engage in family’s chosen ethical/moral instruction
For academic reasons
Because of dissatisfaction with public/private schools (dissatisfaction of both academics as well as behavior)
Parent knows children the best (i.e., is best situated to educate them)
Wanted education to be “joyful” for students
Didn’t want children to be away from home for 40 hours a week
Reasons for online schooling. These included:
Ability to offer instruction in subjects that are beyond parents’ ability to teach
To see how children compare to other students
To save time (because parent works outside the home)
For additional research or practice items to supplement parent-led instruction
Satisfaction with quality of online curriculum
Desire for children to be competent on computer
Desire for children to foster independence
Generally, religion, morals, family values were underlying themes for the homeschool decision.
Item Language
1) Is this person primarily currently:
Homeschooled instead of enrolled in a public or private school for some or all classes;
Enrolled in a public or private school, or preschool;
Enrolled in a college, university or vocational school, or
Not in school?
Key issues with this item:
Confusion due to having a child who is simultaneously homeschooled and enrolled in college courses
Confusion due to how to answer due to virtual schooling
2) There are many different reasons that parents choose to homeschool their children. Did your family choose to homeschool this child because:
-
Yes
No
(a) You are concerned about the school environment, such as safety, drugs, or negative peer pressure?
(b) You are dissatisfied with the academic instruction at other schools?
(c) You prefer to teach this child at home so that you can provide religious instruction?
(d) You prefer to teach this child at home so that you can provide moral instruction?
(e) This child has a physical or mental health problem that has lasted six months or more?
(f) This child has a temporary illness that prevents him/her from going to school?
(g) This child has other special needs that you feel the school can’t or won’t meet?
(h) You are interested in a nontraditional approach to children’s education?
(i) You have another reason for homeschooling your child? — Specify _______________
OR
2) There are many different reasons that parents choose to homeschool their children. How important were the following reasons in your decision?
-
Not at all important
A little important
Somewhat important
Very important
(a) You are concerned about the school environment, such as safety, drugs, or negative peer pressure?
(b) You are dissatisfied with the academic instruction at other schools?
(c) You prefer to teach this child at home so that you can provide religious instruction?
(d) You prefer to teach this child at home so that you can provide moral instruction?
(e) This child has a physical or mental health problem that has lasted six months or more?
(f) This child has a temporary illness that prevents him/her from going to school?
(g) This child has other special needs that you feel the school can’t or won’t meet?
(h) You are interested in a nontraditional approach to children’s education?
(i) You have another reason for homeschooling your child? — Specify ________
Key issues with this item:
Parents have mixed reactions to this item. While more stated they liked the Likert scale, some parents like the yes/no format better. One says that s/he likes the gradient format but with an added “N/A” option for each row. One parent said that s/he thought the gradient would give you better data, but another said that s/he thought the gradient would be burdensome for parents to answer and would deter them from answering.
3)
a) Does this supplemental instruction occur in:
Brick-and-mortar public school (K-12)
Brick-and-mortar private school (K-12)
Online school
College, community college, or university
b) How many hours each week does this child go to the following places for instruction?
Do not include time spent in extracurricular activities.
-
Hours
Brick-and-mortar public school (K-12)……………………………………………………….
_____
Brick-and-mortar private school (K-12)………………………………………………………
_____
Online school……………………………………………………………………………………………..
_____
College, community college, or university………………………………………………….
_____
OR
a) Does this instruction occur in:
Brick-and-mortar public school (K-12)
Brick-and-mortar private school (K-12)
Online school
College, community college, or university
b) How many hours each week does this child go to the following places for instruction?
Do not include time spent in extracurricular activities.
-
Hours
Brick-and-mortar public school (K-12)………………………………………………………..
_____
Brick-and-mortar private school (K-12)………………………………………………………
_____
Online school……………………………………………………………………………………………..
_____
College, community college, or university………………………………………………….
_____
Homeschool………………………………………………………………………………………………
_____
Key issues with this item:
Generally, the parents think the word “supplemental” is very problematic. Homeschooling is thought of as a holistic approach to education, so the notion of any part of the kids’ activities being a “supplement” to their education is not appropriate for these respondents.
Regarding the use of “brick and mortar,” parents think this is clear.
4) Is this child enrolled in any online courses? Note: Please include only courses that are for credit or are part of your homeschool curriculum.
No
Yes, for some courses
Yes, for all courses
Key issues with this item
The parents struggled with this one, because they didn’t know what would count. The takeaway for NCES may be that a question to disentangle online activities such as Kahn vs. an actual course may be meaningful for researchers but not practical to parents. The effort to capture online activity/virtual schooling may be too nuanced at this stage for the NHES surveys.
5) Some questionnaires are endorsed by organizations. For instance, the 2011-12 Schools and Staffing Survey school questionnaire was endorsed by several groups (please see example on following page).
What do you think about asking homeschool groups to endorse a questionnaire?
Possible groups to ask for endorsements could include:
Homeschool Legal Defense Association
National Black Home Educators
Coalition for Responsible Home Education
Homeschool Alumni Reaching Out
Key issues with this item
Parents generally said they would be more likely to fill out the questionnaire if it were endorsed by the HSLDA and state homeschooling organizations. They referred to the SASS example, which includes teacher organizations, and said they would be much less likely to fill out a questionnaire with these types of groups.
6) What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
What type of school does this child attend?
Note: Charter schools are public schools
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
What type of school does this child attend?
Note: Charter schools, both those that meet in brick-and-mortar facilities and those that are virtual, are public schools
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
We did not discuss this item at the focus group.
Appendix B: Key Findings from Virtual Schooling Focus Group
Type of School and Instruction Delivery
Key Findings
Intensity of online experience. The group contained a variety of experiences, which included the following:
Full-time virtual school
Brick-and-mortar schooling with supplemental virtual courses
Location of course access. Students accessed courses in a variety of settings, including at home and at a brick-and-mortar school.
Generally, students who were in brick-and-mortar schools who supplemented with online courses accessed those online courses at school.
Students who were fully online accessed those courses at home (or at another non-school location).
One student had a blended experience, where the instruction/assignments were all online and pacing was individualized, but the child went to a school building for face-to-face interaction with teachers two days a week.
Time spent on virtual instruction. Time spent on virtual activity ranged from 1.5-2 hours (for non-course supplemental instruction), to about 10-15 hours per week for part-time, to full-time.
Public/private schooling. Most parents in the group engaged in public online schooling. One parent had a child who attended a private school and had supplemental online courses in addition to brick-and-mortar schooling.
Types of providers. Providers varied and included the following:
State-wide virtual schools
Public schools/districts
Private school
Types of online courses. Types of online courses varied, both with respect to content and purpose.
Content
Core classes
Elective classes
Purpose
Main educational program (particularly for full-time students, but for others in some cases)
Credit recovery
Accelerated coursework
Reasons for choosing online schooling. The reasons parents chose virtual (also explored in more detail in later section, below) varied.
Child was sick and needed to catch up.
Online courses are required.
Child’s schedule requires flexible schooling arrangement.
Child has special need.
Child required more advanced courses than school offered.
Fees
Key Findings
Fees for academic content. Most parents reported that they did not pay fees for their children’s virtual learning/academic courses. Two exceptions existed:
One parent whose child takes college-level courses through Frontier Academy pays a fee per course.
The parent whose child attends private school pays tuition for the private school. There is no additional fee for the virtual courses, over and above the tuition.
Fees for supplies. Parents generally indicated that supplies were included/provided by the schools. For instance, one parent shared that there were no books to purchase because all of the content was available electronically. Another parent shared that his/her child was enrolled in a science course at one point and that the school sent materials including a microscope.
Fees for extracurriculars. Responses varied regarding fees for school-related extracurriculars. One parent reported that s/he did not pay fees for school-related extracurriculars, whereas another reported that s/he did pay fees for school-related extracurriculars.
Subsidies/support. The parents discussed a variety of available financial supports (though not all parents in this group qualified for and/or participated in these supports). The supports that parents discussed included the following:
Some schools will send families a computer if they cannot afford one
Some internet providers may provide discounted internet access if families qualify
A private school parent indicated that s/he gets a tax break (but this parent does not consider the tax break a tax credit and does not think about this break when s/he is paying tuition)
Identification with homeschooling
Key Finding
The majority of parents in this group do not identify as homeschoolers. Only one parent identifies as a homeschooling parent.
Information Informing Schooling Decisions
Key Findings
Source of information. Parents indicated that they learned of virtual schooling from the following sources:
Several parents reported that they learned about their virtual schooling options through counselors at their students’ previous (or current, in the case of part-time virtual schoolers) brick-and-mortar schools. By far, this was the most frequently-cited source of information among the parents in this group.
One parent reported that they learned about virtual schooling options through the experiences of a family member.
One parent of a child in a private school said that s/he did not necessarily need to learn about the virtual schooling option; the virtual schooling is just a part of the private school’s curriculum and the parent did not choose to opt his/her child into virtual schooling.
One parent had a child who attended full-time virtual school at about the same time that the state was starting its virtual school and was offering introductory meetings for parents. This parent attended these meetings.
Adequacy of information. Parents generally reported that they perceived the information they received to be adequate for them to make a decision.
Reasons for Virtual Schooling
Key Findings
Reasons for full-time virtual schooling included the following:
Child was not doing well in traditional public and private schools.
Child had special need and parent perceived that online setting would better address special need. (In this instance, child had ADD and parent perceived online schooling would remove distractions that occur in classrooms.)
Child has schedule conflict and is not able to be in traditional brick-and-mortar school. (In this instance, child is training for a sport.)
Reasons for part-time virtual schooling included the following:
Parent perceives that virtual schooling is best for child’s learning style.
Parent perceives that virtual schooling is best for parent’s parenting style.
Child is taking course to raise GPA.
Child is taking course to catch up in school after falling behind.
Child prefers online course to face-to-face course.
Taking some online courses is required for child’s degree completion.
Child is taking online courses to access advanced material not offered at brick-and-mortar school.
Parent does not have choice; online courses are part of school’s curriculum.
Item Language
1) Is this person primarily currently:
Homeschooled instead of enrolled in a public or private school for some or all classes;
Enrolled in a public or private school, or preschool;
Enrolled in a college, university or vocational school, or
Not in school?
Key finding for this item
Parents generally find this question to be appropriate, though one parent says that an option “Enrolled in virtual schools” or “enrolled in full-time virtual school” would be helpful to capture the experiences of some of the parents in the group.
2) What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
2) What type of school does this child attend?
Note: Charter schools are public schools
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
2) What type of school does this child attend?
Note: Charter schools, both those that meet in brick-and-mortar facilities and those that are virtual, are public schools
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
2) What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
OR
2) What type of school does this child attend?
Private, Catholic;
Private, religious but not Catholic;
Private, not religious;
Public school
Key finding for this item
Parents generally agree that the second option is the best.
3)
a) Does this instruction occur in:
Brick-and-mortar public school (K-12)
Brick-and-mortar private school (K-12)
Online school
College, community college, or university
Homeschool
b) How many hours each week does this child go to the following places for instruction? Do not include time spent in extracurricular activities.
-
Hours
Brick-and-mortar public school (K-12)……………………………………………………………….
_____
Brick-and-mortar private school (K-12)……………………………………………………………..
_____
Online school…………………………………………………………………………………………………….
_____
College, community college, or university…………………………………………………………
_____
Homeschool………………………………………………………………………………………………………
_____
Key finding for this item
Parents found this question to be appropriate.
4) Is this child enrolled in any online courses? Note: Please include only courses that are for credit.
No
Yes, for some courses
Yes, for all courses
Key finding for this item
Parents said that this makes question makes sense. One parent said that she does not know what the intent of the question is, but if s/he were asking the question, s/he would want to know if the course was for credit recovery or if it was the first time the student was exposed to the content.
Another parent noted word choice usage throughout the questions. Parents noted that many of these questions use the word “online,” but the words “virtual” and “cyber” are also ways to describe this form of schooling.
Wrap up
Key findings
One parent shared that s/he is very purposeful/intentional in his/her approach to child’s virtual schooling and spends a lot of time and effort on it.
One parent who is also a teacher shared that s/he has concerns over the extent to which virtual courses teach students course content and the extent to which students are held accountable for online learning.
One parent indicated that whether or not the students are accountable in online courses depends on the parent.
One parent indicated that online learning allows his/her child to follow passion (sport).
One parent wondered if all schooling will be virtual in the future.
National Household Education Surveys Program (NHES) 2017 Web Test Debriefing Interviews for Parents of Homeschoolers (OMB# 1850-0803 v.187)
Homeschool Debriefing Interviews Meeting
ESSIN 33: Early Childhood Coordination and NHES Survey Development
Meeting Information |
|||
Date: |
August 22, 2017 |
Time: |
1:30 – 3:30 PM |
Location: |
Virtual |
||
Dial-in # |
888-833-8895 |
Access Code: |
6293735 |
Chair: |
Kathleen Hoyer |
Recorder: |
Stephen Wenck |
Scheduled Time |
Actual Time |
||||
Start |
Stop |
Total Time |
Start |
Stop |
Total Time |
1:30 pm |
3:30 pm |
120 mins |
1:31 pm |
3:25 pm |
114 mins |
Invitees |
|||||
☒ |
Colleen Connelly (SANA) |
☒ |
Sarah Grady (NCES) |
☒ |
Ranetta Hardin (Activate) |
☒ |
Kathleen Hoyer (Activate) |
☒ |
Dinah Sparks (Activate) |
☒ |
Stephen Wenck (SANA) |
☒ |
David Martin (NCES) |
|
|
|
|
☐Attendees **Called in |
|||||
Agenda |
||
# |
Item |
Time |
1. |
Welcome and overview of meeting |
1:30 pm – 1:35 pm |
2. |
Discussion of contact information quality and profiles of respondents |
1:35 pm – 1:45 pm |
3. |
Discussion of individual interviews |
|
|
Interview 1 (Kathleen) |
1:45 pm – 1:55 pm |
|
Interview 2 (Kathleen) |
1:55 pm – 2:05 pm |
|
Interview 3 (Kathleen) |
2:05 pm – 2:15 pm |
|
Interview 4 (Colleen) |
2:15 pm – 2:25 pm |
|
Interview 5 (Ranetta) |
2:25 pm – 2:35 pm |
|
Interview 6 (Ranetta) |
2:35 pm – 2:45 pm |
4. |
Discussion of Common Themes |
2:45 pm – 3:00 pm |
5. |
Recommendations |
3:00 pm – 3:15 pm |
6. |
Final Questions |
3:15 pm – 3:30 pm |
Minutes |
|||||||||||||||||||||||||||||||||||||||||||
1. |
Welcome and overview of meeting
|
||||||||||||||||||||||||||||||||||||||||||
2. |
Discussion of contact information quality and profiles of respondents
|
||||||||||||||||||||||||||||||||||||||||||
4. |
Discussion of common themes
|
||||||||||||||||||||||||||||||||||||||||||
5. |
Recommendations
|
||||||||||||||||||||||||||||||||||||||||||
6. |
Final questions
|
||||||||||||||||||||||||||||||||||||||||||
Action Items |
|||||
# |
Description |
Assigned To |
Comments |
Status |
Due Date |
1 |
Provide IDs to Sarah so she can look at responses in combination with meeting minutes. |
Steve Wenck |
|
Complete |
8/24/2017 |
Action Items Status Color Code |
|
Green |
On schedule |
Yellow |
Possible delay or action required |
Red |
Immediate action required |
National Household Education Surveys Program 2019 (NHES:2019) Low Response-Propensity Parents Focus Groups (OMB# 1850-0803 v.188)
National Center for Education Statistics
National Household Education Surveys
Report of Focus Groups Among Low Response-Propensity Parents
May 2017
Prepared by
Hager Sharp
Table of Contents
Summary of Highlights and Conclusions 6
Detailed Findings by Location 12
Richmond, VA, April 24, 2017 25
Philadelphia, PA, May 2, 2017 40
Appendix A: Moderator’s Guide 55
Overview and Methodology
Overview
The National Household Education Surveys Program (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields two to three topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.
Modeling response propensity suggested that addresses from which NHES is unlikely to get a response share similar demographic characteristics, suggesting that response bias is an issue that may affect data quality. Analyses of the demographics of low-response propensity addresses indicate that occupants of these addresses are more likely to fall into at least one of the following groups:
Be age 35 or younger
Rent rather than own
Have less than a high school education
Have an income below $15,000
Identify as Hispanic
Anyone matching any one of these characteristics is less likely to respond. The response propensity model is a regression model that showed that, holding all other variables constant, each of the above characteristics is significantly related to responding at lower rates to previous NHES administrations. Therefore, NCES determined the need to focus research on parents that fall into any one of these categories. Not all conditions need to be fulfilled by individual focus group participants.
NCES hopes to improve NHES response rates and reduce response bias for the 2019 NHES by better understanding factors that motivate parents to complete (or decline to complete) the NHES questionnaire. NCES has focused specifically on understanding motivations to complete the survey among occupants in sampled addresses that are modeled to be unlikely to respond to the survey. The findings highlighted in this report will be used to refine the materials and strategies used for NHES recruitment.
Methodology
Hager Sharp conducted a total of eight in-person focus groups with parents from different states and school districts to understand their perceptions of the NHES study and how they would respond if selected for NHES. We worked with recruiters at local focus group facilities in Richmond, VA (Alan Newman Research), Philadelphia, PA (Schlesinger Associates), and Denver, CO (Fieldwork Denver) to recruit for and host the in-person focus groups. Focus group facility staff contacted people from their databases to recruit respondents who met the following eligibility criteria:
Parent of at least one child under age 18 in household
Age of parent is under 40
Household income is below $40,000
Hispanic, Spanish-speaking parents recruited for two focus groups
Eligibility criteria for focus group participants were based on the key regression model variables that were most salient for recruitment efforts, with Spanish-speakers being grouped together for language continuity. To meet realistic recruitment goals, some criteria were adapted from the regression model variables. For example, we determined parents who are 35 or younger would be too difficult to recruit under cost and schedule constraints, so we relaxed the criteria to parents under the age of 40. Similarly, we determined households with incomes below $15,000 would be too difficult to recruit under cost and schedule constraints, so we relaxed the criteria to households with incomes below $40,000. We also gathered information on rent/home ownership status and education levels, and most of the parents we recruited indicated they rent their home. To the extent possible, we recruited parents with a high school education or less. Details of participant characteristics are included in this report in the section on “Detailed Findings by Location.”
The configuration of the eight focus groups is as follows:
Market |
Language |
HH Income |
Age of Parent |
Race/Ethnicity |
Number of Groups |
We recruited 10 participants for each focus group, and eight or nine showed in each group. Topics of the discussions focused on identifying the benefits parents associate with NHES participation, the barriers they perceive to participating, which NHES advance materials they believe are most and least useful, and their suggestions for factors that may increase response rates for NHES. The focus group sessions lasted approximately 90 minutes, and all sessions were audio and video recorded. Hager Sharp will destroy the recordings upon final approval of this report. This report includes no personally identifiable information regarding participants. To facilitate recruitment for the focus groups and thank participants for their time, we gave each parent a $75 incentive for participation.
During the focus groups, using the moderator’s guide (Appendix A), a moderator led the participating parents through a discussion. Key research questions for this project are outlined as follows:
What value do parents perceive NHES has in education?
Does the format of NHES influence parents’ motivations to participate?
What information do parents use when deciding whether or not to participate in a survey like NHES? Who do they consult?
What barriers prevent or discourage participation in NHES?
What would minimize these barriers for parents?
What factors might motivate parents to participate in a voluntary survey such as NHES?
Does the U.S. Census Bureau “brand” help in motivating parents to complete the survey?
Does the $5 bill included in the survey motivate parents to complete it?
What are the perceived benefits of participation for schools?
What current or potential NHES messages or materials do parents find relevant, useful, and informative?
What would be the most effective channels, formats, and materials to use to communicate with parents?
What other groups or people (e.g., teachers, community organizations) would be effective in influencing parents on this topic, and what would be the best way to reach those influencers?
The moderator showed participants several documents that are proposed for use in NHES recruitment, and participants discussed each. The materials included:
Advance letter for recruitment – English
Advance letter for recruitment – Spanish
Sample screener survey – English
Sample screener survey – Spanish
To prepare this report of findings, Hager Sharp worked from transcripts of all groups to analyze trends and key themes. We have included verbatim comments from participants to illustrate and provide examples of themes and trends. We have also noted key areas of consensus and areas where participants differ in their views.
Summary of Highlights and Conclusions
Highlights and conclusions from the focus groups are organized by the key research questions we identified when planning the research:
What value do parents perceive NHES has in education?
Parents have essentially no awareness of the NHES, but they indicated a desire to learn more about the survey program. The topic of education is of critical importance to parents, and they are motivated to participate in any effort that they perceive might benefit their child’s education.
Parents generally assume policymakers will use Census and NHES data to allocate resources for education, youth services, and communities in general. They feel these data collections represent an important channel through which “their voices will be heard.”
Some also believe they should do their part to contribute to the accuracy of the data.
Does the format of NHES influence parents’ motivations to participate?
Many parents expressed a preference for an online survey, but some preferred to take a paper survey. All favored having an option to complete the survey either in paper or online. Some parents expressed an interest in having access to the survey through a mobile app, as they said they could take the survey while waiting at various points in their day.
What information do parents use when deciding whether or not to participate in a survey like NHES? Who do they consult?
Credibility
Parents rely heavily on the credibility of the source—i.e., the Department of Commerce, the Census, and the Department of Education—to decide if they will take the survey. Many or most had a favorable view of Census. Nearly all had a favorable view of the Department of Education. Some expressed concern regarding the Department of Commerce, associating it with the Internal Revenue Service (IRS). Several also commented that they take the Department of Commerce seriously, as they perceive it has financial implications.
Most parents said they would be more likely to complete the survey if they saw immediately that it is for the Department of Education. Some also suggested that if they heard about the survey from their child’s teacher or school, they would be more likely to participate.
Relevance
Another key factor in parents’ decision is relevance. Nearly all parents were inclined to participate in surveys regarding education, as they perceived their participation could benefit their child’s education.
Incentive
Most parents responded favorably to receiving an incentive for taking surveys, as they see it as a “fair” compensation for their time. They believe that the size of the incentive should be commensurate with the time required to complete the survey.
What barriers prevent or discourage participation in NHES?
The biggest barrier to completing the survey seems to be a lack of understanding its purpose and how it will be used. Parents consistently requested more information about the survey’s purpose and use in all focus groups.
The readability and relevance of the advance letter also represent key barriers to participation. The letter should be revised to include a larger font size as well as information from the Commonly Asked Questions section that currently appears in the back of the survey. Key information should be emphasized in bold and summarized in bullets. Some of the current language in the letter is confusing for parents. For example, language that says, “you cannot be replaced” seems to contradict language about the random sample approach referenced in the survey questionnaire. In addition to the advance letter, relevance throughout the package is a key barrier, as other components should also be revised to emphasize that the survey is about education.
Privacy and security concerns also represent barriers to survey completion. Parents, especially Hispanic and African American parents, had concerns about being “targeted” for their personal information. Some of the concerns were based on a wariness of scams, while other concerns were related to a fear of being targeted by the government. A few parents in urban areas had concerns about mail theft, in which neighbors might take their completed surveys and discover the names of their children, making them vulnerable to predators.
Another barrier may be related to the $5 incentive, as some participants expressed concern that many people would pocket the $5 without completing and returning the survey. While participants attributed this action to “other people” during the focus group session, it could be that they would also take this action in the privacy of their homes.
What would minimize these barriers for parents?
Making it clear—throughout the package—that the survey pertains to education would help minimize the barriers. Providing a brief description of the survey’s purpose and how it will be used would also provide a strong motivation for participation. Providing more information about the two-step process of data collection would let parents know what to expect. Participants suggested that they would be motivated to take the second-stage survey if they knew they would receive an incentive to take it. Therefore, providing information about the incentive within the information about the two-step process would be beneficial.
Incentives help in two ways: All participants indicated a willingness to look at the package more closely after finding the $5 bill, and some felt obligated to complete and return the survey in exchange for keeping the money.
Credibility is the key to minimizing privacy and security concerns. Parents seem to trust the Department of Education more than the Department of Commerce. Therefore, highlighting the involvement of the Department of Education throughout the package would be beneficial. Most parents recognize and have a positive view of the Census; therefore, the package should retain Census branding. It may be beneficial to suggest the Department of Education and the Census are working together in the effort, as this may allay some confusion about the roles of various government agencies and offices.
What factors might motivate parents to participate in a voluntary survey such as NHES?
For participating in the screening survey, parents strongly expect information about the purpose of the survey and how it will be used, especially if it will benefit their community or their child in some way. Many would also be interested to learn about the results of the survey, and they would be willing to go online to read a brief report.
A key motivational factor for parents is the desire to improve their child’s education and educational environment. Therefore, the package should emphasize the survey is about education. The topic of education is more motivating for parents than the general topic of the Census. Parents perceive the Census as an important tool that policymakers use to allocate resources to communities. Parents are motivated by the belief that their participation could contribute to a positive benefit for their community. After hearing about NHES in the focus groups, they assumed NHES is a tool that policymakers use to allocate resources in education. This belief is a critical motivational factor for their participation in NHES. Some parents also have a sense of civic duty regarding the Census (i.e., an obligation to support the accuracy of the data) that could be leveraged for NHES.
For participating in the second-stage survey, parents are motivated by the prospect of receiving an incentive. They generally cited $25, either in cash or as a gift card, as a “fair” incentive for participation.
Does the U.S. Census Bureau “brand” help in motivating parents to complete the survey?
The Census brand is widely recognized, and most parents seem to understand and respect its importance. The credibility of the Census also helps in allaying privacy and security concerns. However, Census is not as interesting to parents as the topic of education. In some cases, parents see Census as a “tedious” burden that must be endured. Therefore, the package should emphasize the topic of education, which is of interest to parents, and highlight the role of the Department of Education, which parents seem to trust.
A small percentage of parents have a negative view of Census, largely associated with privacy concerns.
Does the $5 bill included in the survey motivate parents to complete it?
Incentives help in two ways: All participants indicated a willingness to look at the package more closely after finding the $5 bill, and some felt obligated to complete and return the survey in exchange for keeping the money. A few parents expressed concern about “wasting taxpayer dollars” if recipients of the package threw it away without opening it.
Participants suggested that they would be motivated to take the second-stage survey if they knew they would receive an incentive to take it. Therefore, providing information about the incentive within the information about the two-step process would be beneficial.
What are the perceived benefits of participation for schools?
After hearing about NHES in the focus groups, they assumed NHES is a tool that policymakers use to allocate resources in education. Specifically, parents hope the schools in their community will receive funding for services and programs based on decisions made from survey data. This belief is a critical motivational factor for their participation in NHES.
What current or potential NHES messages or materials do parents find relevant, useful, and informative?
The mail package could be revised to minimize barriers more effectively. Suggestions include:
Exterior Envelope
The exterior envelope containing the recruitment package would be more effective if it emphasizes the contents inside pertain to education. Including the Department of Education logo and a brief reference to an education survey would likely capture parents’ attention and interest.
Advance Letter
The advance letter requires the most extensive revisions out of all components in the package. Suggestions include:
Increase font size to improve readability.
Include essential information in bullets and bold font.
Revise the content to focus on what parents want to know about the survey:
Emphasize that the topic is education.
Clarify that it is being conducted by the Census on behalf of the Department of Education.
Describe the survey program’s purpose and how it will be used.
Briefly describe the two-step process to let parents know what to expect.
Retain information about the amount of time required to complete the survey.
Retain the telephone number and website address parents can use to get more information about the survey.
Delete reference to a “scientific study,” as parents were confused and skeptical of this. This reference does not enhance the credibility of the NHES.
Delete the appeal that parents “cannot be replaced,” as this is confusing for parents when they see the reference to random sampling that is in the survey questionnaire. The appeal, as currently worded, is not effective in motivating parents to participate.
Retain bilingual versions of the letter in English and Spanish, as parents find these useful.
Consider adding a separate page of information about the second-stage surveys that parents may be invited to complete. Briefly describe the purpose of the surveys and a website address where parents can get more information.
Screener Survey
Participants found the sample screener survey straightforward and easy to understand. However, parents universally suggested moving the Commonly Asked Questions (CAQ) content to a more prominent place in the package where parents would notice it more readily. The CAQ content answers most of the questions parents had about the NHES, and should be the first or second item they see in the package. Participants suggested either incorporating the content into the advance letter or moving the CAQ section to the page immediately following the survey title page. A combination of both options could be effective—e.g., keep the CAQ section intact at the front of the survey, while also highlighting some of its key points in the advance letter. In addition, the questionnaire may also provide an opportunity to briefly reiterate the two-step process for administering the NHES.
Return Envelope
A few participants noted that the return address to the Director of the Census Bureau seems impersonal and perhaps even illegitimate. Adding the name of the director, if feasible, could allay concerns about the official nature of the package.
What would be the most effective channels, formats, and materials to use to communicate with parents?
Many parents suggested distributing the survey through the schools. When explained that this is not feasible, parents generally preferred to learn about and receive NHES materials through the mail. Some suggested that raising awareness about the survey through the schools could be beneficial, as parents trust information they hear from their child’s teachers.
A few parents indicated they would actively go online to seek information about the NHES. Some parents also cited Facebook as a relevant channel for outreach about NHES.
What other groups or people would be effective in influencing parents on this topic, and what would be the best way to reach those influencers?
Teachers are key influencers on any topic pertaining to education.
Recommendations
Key recommendations for revisions to the package are outlined as follows:
Package Component |
Recommendation |
Exterior Envelope |
|
Advance Letter |
|
Screener Survey |
|
Return Envelope |
|
National Household Education Surveys Program (NHES) 2019 Early Childhood Cognitive Interviews (OMB# 1850-0803 v.198)
Findings presented below in the “National Household Education Surveys Program (NHES) 2019 Early Childhood Cognitive Interviews - Revised (OMB# 1850-0803 v.215)” section.
National Household Education Surveys Program (NHES) 2019 Early Childhood Cognitive Interviews - Revised (OMB# 1850-0803 v.215)
NHES ECPP 2017 Cognitive Interviews Report
Prepared for the National Center for Education Statistics (NCES)
March
23, 2018 Child
Trends Kristen
Darling, Manica Ramos, Bianca Faccio, Monica Arkin, Alexandra Cox,
Lina Guzman
Table of Contents
Rationale for Using Cognitive Interview Method 2
Survey Instruments and Administration 4
Findings & Summary of Revisions 8
Issues with Question Comprehension 9
Issues Recalling or Retrieving Information to Answer the Question 14
Issues Matching Internal Representation to Given Categories 15
Issues Navigating and Following Intended Skip Patterns 17
Revisions made to ensure consistency across NHES suite of surveys 21
Language Revisions Made to the Spanish Survey 22
Recap of Key Findings and Recommendations 24
Limitations of Study Design 26
Future Work (NHES: 2019 and 2022) 27
Items Needing Additional Monitoring or Development 27
Appendix 1. Tested and Revised ECPP Items 29
Appendix 2. ECPP Finding and Choosing Care for Your Child section 58
Appendix 3. ECPP 2017 Cognitive Interview Respondent List and Detailed Characteristics 60
Appendix 4. NHES ECPP Cognitive Interview Protocol 63
Appendix 5. Final ECPP English and Spanish Questionnaires 73
Introduction
Background on the NHES
The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields two to three topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.
Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. The same two surveys, along with the Adult Training and Education Survey (ATES), were fielded in NHES:2016. In 2019, the NHES will field the PFI and ECPP surveys along with the second administration of the ATES. This will be a two-stage study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within- household sampling will be performed. Finally, topical surveys will be administered to the selected household members.
The ECPP, previously conducted in 1991, 1995, 2001, 2005, 2012 and 2016, surveys families of children ages 6 or younger who are not yet enrolled in kindergarten and provides estimates of children’s participation in care by relatives and non-relatives in private homes and in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for nonparental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics.
Purpose of this Study
This report is based on the findings from 2017 cognitive testing of the Early Childhood Program Participation (ECPP) paper-based survey. These interviews were conducted because NHES no longer plans to rotate topical modules as was commonplace from 1991-2007 and NCES wanted to refresh the content of the ECPP for the NHES:2019survey instruments and beyond.
Prior to fielding, the research team conducted two activities to inform recommendations for items to include or adapt for the NHES:2019 instruments. These two activities were 1) a review of NHES questionnaires including past versions of the ECPP and the School Readiness Survey3; and 2) the January 2017 convening of a panel of twelve early care and education subject matter experts. These experts were convened by NCES to review how the NHES ECPP addressed the constructs of school readiness and child care decision making; in addition, during the meeting, the panel recommended including items about families’ experiences with early detection systems for developmental concerns.4 The following report details which items were added, adapted, or tested, and our findings.
The PFI was also tested as part of this contract. Findings for the PFI are discussed in a separate report. However, the rationale for some changes to the ECPP questionnaire was to achieve consistency between these two NHES surveys. The relation of item-changes to decisions made for the PFI is noted where applicable.
Rationale for Using Cognitive Interview Method
Early learning is the bedrock for children’s later learning and success in school. Young children who have access to high quality early childhood education are more likely to be ready for school. However, not all children have access to or attend high quality early care arrangements. Collecting national data on the types of care young children attend and their level of school readiness provides information to stakeholders interested in improving the education outcomes of young children. Parents also face challenges to accessing affordable high quality early care and education arrangements. These challenges can make children vulnerable to attending early care and education arrangements that are suboptimal for their early learning. Understanding the challenges and the decision-making processes by which parents choose children’s care is of utmost importance so that stakeholders can support parents in the best possible way. This is particularly important as public investment in ECE has increased and publicly funded preschool, as of 2016, has expanded to now 43 states, plus the District of Columbia and Guam.5
The 2019 NHES-ECPP survey will expand content related to childcare decision-making and school readiness. Thus, cognitive interviews were completed from August to November 2017 to assess the extent to which new NHES-ECPP survey items designed to capture data about childcare decision-making and school readiness can measure these early childhood phenomena accurately. Though no significant changes were planned to the design of the questionnaire from the 2016 NHES-ECPP administration, the interviews also evaluated respondents’ ability to navigate the survey instrument. Specifically, the research team ensured that the items were clear, easily understood, and interpreted the way they were intended, and that respondents had the information needed to answer the questions. Additionally, cognitive interviews aimed to confirm that the questions were applicable for racially, ethnically, and economically diverse populations and across care settings and age groups of children—including center-based, family-based, and Head Start settings and other publicly funded programs. The results presented below have been submitted as revisions to the ECPP survey for NHES:2019.
Research Team
The ECPP cognitive interview team was comprised of staff across two organizations partnered to administer this contract: Sanametrix and Child Trends. Both firms offered experience conducting cognitive interviews, as well as Spanish language competencies. The firms worked collaboratively to execute the study, with Sanametrix leading the majority of the recruitment efforts; Child Trends providing subject matter expertise, training oversight, and data quality control; and both firms sharing the field work and documentation requirements.
Research Methods
Timing
This project received OMB clearance in July of 2017. While anticipating approval, all team members that conducted cognitive interviews were trained twice on cognitive interview techniques and the study protocol, once in March of 2017 and again in June of 2017, using the draft materials. Recruitment and fielding began shortly after OMB clearance was obtained and ran from August 2017 through November 2017. The final recommendations for the ECPP survey based on the cognitive testing were submitted in December 2017.
Sample
We sought to conduct a total of 65 cognitive interviews with parents of young children to determine whether there are systematic problems in the ECPP items by type of care and/or child’s age. Given this objective, we selected a purposive sample to ensure respondents had experiences relevant to the study. Recruited parents were grouped by their child’s age and participation in different types of child care arrangements. At least 12 of the 65 interviews were to be conducted in Spanish with Spanish-language dominant participants, making sure that at least one Spanish-speaking participant was represented in each grouping. Additionally, we strategically sought to sample diverse participants with respect to race/ethnicity, education, and income. As such, tallies of race/ethnicity, education, and income of each recruited participant were monitored, ensuring representation of these demographic characteristics across the groupings.
The plan was that all respondents would be recruited using multiple sources, including company databases, social media/Craig’s List, personal and professional contacts, community-based centers, early education and Head Start programs, fliers posted in the community, and on-the-ground recruitment (handing out fliers and describing the study to potential participants – Spanish language only). However, recruitment was slower than we anticipated, so Child Trends and Sanametrix expanded efforts by sending out multiple emails to work and personal contacts and attending a neighborhood event to recruit community members. We also utilized snowball sampling, in which respondents were given the opportunity at the conclusion of their cognitive interview to refer friends who might be eligible.
Due to recruitment challenges, target numbers were modified, in consultation with the NCES. In total, the team successfully conducted 58 cognitive interviews across all types of care. Many respondents had more than one type of care. In total, 14 respondents reported a relative care arrangement, 21 respondents reported anon-relative care arrangement, and 39 respondents reported a center-based care arrangement. See Exhibit 1 below for a summary of the characteristics for our sample, and Appendix 3 for full details by respondent.
Exhibit 1. ECPP 2017 Cognitive Interview Respondent Characteristics
Interview Language |
Type of Care |
Race |
Hispanic/ Latino |
Education |
Income |
Survey Instruments and Administration
The primary purpose of the cognitive interviewing was to test question comprehension, and response and information retrieval. Our interview protocols were designed accordingly. Respondents were given a copy of the latest iteration of the ECPP questionnaire to complete. The interviewer followed along simultaneously with a copy of the ECPP questionnaire and interview protocol (edited after each round of interviews to match the latest iteration of the ECPP questionnaire), which included probes. See Appendix 4 for the protocol and Appendix 5 for the final questionnaires.
Interviews were divided into four rounds. This allowed for an iterative process of revising the surveys every few weeks. Between each round was a debriefing, where main findings and patterns in respondent responses were discussed with NCES. After each debriefing, the survey instruments were updated to resolve issues that were addressed in the debriefing.
Interviewers attended two trainings prior to the start of fielding. Trainings were held at the Child Trends office on March 23, 2017 and June 26, 2017 and were instructed by two Child Trends research scientists who had prior experience with cognitive interviewing. Additionally, interviewers were either accompanied by one of these two instructors at their first interview, or their recordings were reviewed following completion, to ensure that administration went smoothly and that data quality would be consistent. After that, each interview was conducted as a one-on-one conversation between the interviewer and respondent. Interviews were held in public areas such as libraries and cafes. The time reserved for each interview was one hour; some interviews ended early and some ran over the hour time limit. When interviews were not completed within sixty minutes, interviewers asked the respondents if they would like to continue or not.
During each interview, the interviewer took notes by hand on the interview protocol. Additionally, each interview was audio recorded. Within 24 hours of the conclusion of each interview, interviewees summarized the interview. These summaries were completed by going through hand-written notes and referring to the audio recording to fill in any missing information. These summaries were discussed at each debriefing with other team members to compare findings.
Interviews facilitated a “think-aloud” approach, accompanied with selected verbal probing on key issues. The following summary (adapted from Willis and Artino, 20131) describes these methods. Think-aloud interviewing asks respondents to actively verbalize their thoughts as they answer the survey questions. This method is recognized as less directive. It does however place more burden on the subjects. Interviewer supported this activity by prompting respondents to “say more” and recording verbalizations in their interview protocol for later analysis. Commonly used prompts in this study included:
Please keep talking.
Tell me what you are thinking.
How did you arrive at your answer?
Can you tell me more about that?
Verbal probing is when the cognitive interviewer actively administers questions designed to elicit additional information beyond that provided independently by respondents. This method may lead to the respondent providing more information than they would have independently. The majority of probes were developed prior to ECPP cognitive interviews and included in a written interview protocol. These probes were designed to help the research team understand very specific issues. Probes used consistently in this study included:
What is this question asking, in your own words?
How easy or difficult was it for you to answer this question?
Also, flexible probes were developed “ad hoc” in response to unique information provided by the respondents. For example, one respondent said the focal child’s program does not provide sick child care when the child is sick but does not have a fever and does not provide sick child care when this child is sick with a fever. The interviewer continued probing, asking if the program makes any distinction based on whether the child has a fever. The respondent explained that they were thinking about sick child care as special emergency sick care, which the program does not offer. However, if a child is sick and fever-free, they can still attend the child care arrangement. This specific, ad hoc probe provided additional information about how the respondent understood “sick child care.”
Overall, the research team found that both probing types worked well.
Spanish interviews followed the same protocol as English interviews; findings from Spanish interviews were discussed at the debriefing along with findings from English interviews. Most of the changes that were made between rounds were relevant to both versions of the questionnaire. However, sometimes Spanish-speaking respondents encountered additional translation issues. Therefore, the Spanish version of the survey sometimes underwent additional changes as we sought to find the clearest translation for each item. Summaries were written in English for the benefit of the full research team. The instruments were translated by native-Spanish speaking team members; one member translated the relevant items, while another member reviewed and either accepted, declined, or provided another suggestion, and a third member reviewed the discrepancies and made final decisions.
Approach to Analysis
As noted earlier, interviews were divided into four rounds to allow for an iterative process throughout fielding. Between each round a debriefing was held, where main findings and patterns in respondent responses were discussed. Debriefing #1 was held on September 8, 2017 and 15 interviews were discussed. Debriefing #2 was held on October 17, 2017 and 15 interviews were discussed. Debriefing #3 was held on November 8, 2017 and nine interviews were discussed. The final debriefing, Debriefing #4, was held on November 27, 2017 and 19 interviews were discussed. At debriefings, interviewers brought all of the summaries from interviews that had completed that round. In preparation for Debriefings #2, #3, and #4, Child Trends staff drafted a “summary of summaries,” which compiled findings across all summaries for that round of fielding. This was used to facilitate the flow of the debriefing: the leader of the meeting went through each item on the questionnaire in order, and interviewees reported whether respondents had issues to each item.
When issues were identified, the team discussed whether the item should be revised or not. There were multiple factors involved in this process of drawing conclusions and making recommendations. One important factor was the number of respondents who encountered/noted/for whom problems were detected with a particular item. If several respondents encountered the same issue on an item, it was considered a flag that the item may need to be revised. In cases where only one respondent voiced an issue, the team discussed whether the source of the issue was in the interview item itself that could be resolved, or a mistake on the behalf of the respondent that was not likely to occur in other respondents. Another factor in this process was whether issues were encountered repeatedly across rounds.
During debriefings one notetaker took notes on a word document while another followed along in an excel document. Each row of the excel document represented an item, and each set of columns represented a debriefing. This excel document allowed the team to look at findings across rounds for each item.
After each debriefing, the survey instruments were updated to resolve issues that were addressed in the debriefing. The only exception was during the final round: following the third debriefing, due to time constraints between the debriefing and the fielding end-date, the instrument received only limited revisions that could be completed manually, and not the full batch of recommended changes. All recommended, approved changes were implemented in the questionnaire following the end of fielding.
Decision-making Process
The decision-making process was a collaborative effort between the data-collectors and the client, NCES. As described above, the findings were compiled by round. This documentation was reviewed at a meeting at the conclusion of each round with all data-collectors and NCES representatives present on a virtual video-conference platform that allowed for as-close-to-in-person-interaction as possible (Web-Ex).
Child Trends prepared for these meetings by reviewing the findings and identifying patterns (or a lack thereof) in the data. At the start of each meeting, the facilitator described these themes, and then presented item-level evidence and examples for each topic. Attendees engaged in dialog to seek further clarification or develop a shared understanding of how to interpret the response(s).
In some cases, decisions about an appropriate course of action were made during the meeting (e.g., for an item with only one respondent presenting a comprehension issue, NCES advised us to monitor that item in the next round; for inconsistent wording between related items in different sections of the survey, desired terminology was agreed upon and documented). For the majority of issues, Child Trends took a reflective approach, deliberating on both the raw data and the meeting discussion, and made recommendations in the days following the meeting. Sometimes additional research was needed to inform a decision (e.g., on frequencies of how respondents answered selected questions; or how certain concepts were presented in other NHES instruments). This information was collected, relayed to NCES, and their guidance was then recorded in the final recommendations documentation. NCES then reviewed the recommendations and made the final decision for each item.
Three types of decisions were generally made:
Revise item wording, response options, and/or instructions or navigational supports
Monitor the item in future data collection
No change
Exhibit 2 below outlines the general decision-making process applied to this study. Determinations were made by first identifying whether an item proved problematic during testing. Issues were examined by type (“Inquiry”). Among extant items, selected time-series items (i.e., items collected in the past by NHES which provide historical comparisons across timepoints) identified by NCES were generally kept intact, barring small edits for consistency or clarity. Issues that arose for extant non-time series items were discussed and the decision to revise, monitor, or drop was applied. For new or revised items, the same “Inquiry” and decision process was applied. Some decisions were also influenced by NCES’s need to balance agency priorities and the “value-added” benefit of the potential change to our understanding of an issue. In instances where the findings were ambiguous (i.e., respondents did not clearly articulate a better alternative), or the information was perceived as less useful without more contextual information (e.g., when the given response options failed to capture the multitude of unique circumstances articulated by respondents, and further explication became unwieldy), these items were generally dropped, or returned to their original wording if they had been revised.
Exhibit 2. Decision tree for how to treat items presenting issues during cognitive interviews
Decisions for individual items, and the rationale for those decisions, are described below as well in Appendix 1.
Findings & Summary of Revisions
The findings of four rounds of cognitive interviews are summarized in the following sections. Interviews were conducted using an iterative approach with each round stemming from and being informed by the previous round. As previously mentioned, NHES’ Early Childhood Program Participation (ECPP) survey is designed to be completed by families using the following care arrangements: (1) relatives; (2) home-based care; or (3) center-based care. Skip patterns are used to direct respondents, based on care arrangement, to applicable sections and questions. Respondents using more than one type of care are directed to complete all applicable sections.
Though there are separate sections for each care arrangement type, duplicate or closely worded questions are asked in each section in order to solicit the same information using language that is most appropriate given the care arrangement. For example, “How many days each week does this child [receive care from this relative/person/go to this program]?”.
Key Findings
Excluding duplicate questions, 84 unique questions were tested through four rounds of cognitive interviews; of these, 24 percent of items (N=20) were new questions being tested for the first time. Of the tested items, we found that: about 19 percent of items (N=16) had no issues6; 39 percent of items (N=33) had issues that required revisions; and 42 percent of items (N=35) raised minor concerns that did not justify revisions. Overall, the results indicate that the question and instruction wording, format and design, and navigational changes made across the four rounds of cognitive interviews addressed the concerns identified through the cognitive interviews. Findings are grouped by the following types of issues encountered and specific examples are presented:
Issues with question comprehension (e.g., misinterpretation of question/question not interpreted as intended by survey writers);
Issues with recalling the information to answer the question (e.g., difficulty recalling the needed information);
Issues with matching internal representation to given categories (e.g., difficulty in accurately mapping the response to the provided categories); and
Issues with navigation and skip patterns (e.g., misguided navigation).
In addition, we present a list of newly developed items that performed well (i.e., no issues were encountered).
Following the Key Findings section, we describe revisions made to ensure consistency across NHES suite of surveys (e.g., terminology alignments).
In addition to testing the NHES’ ECPP survey in English, we also conducted cognitive interviews in Spanish for the Spanish language version of the survey. There were two sets of revisions made to the Spanish language survey: (1) revisions to survey questions based on findings from Spanish cognitive interviews; and (2) revisions made to the survey questions, instructions, and numbering to ensure that the Spanish survey was parallel to the most updated version of the English survey. Revisions that were based on cognitive interview findings are included in the sections listed above with notes that the finding refers to the Spanish language survey. Revisions made to achieve parity between the English and the Spanish survey are summarized later in this report.
Limitations of Study Design
Several notable limitations exist for this study. First, the goal of this round of cognitive testing for the ECPP was not to complete a redesign; rather, it was to refresh the instrument in anticipation of a major change to the mode of administration (from mail-based to mixed methods: web- and mail-based) in 2019. As such, the rationale for changes to the survey was not based solely on respondent feedback. While respondent data provided a starting point in most cases, some changes were indicated based on subject matter expert input, and some on NCES’ priorities. Overall, there was a goal to identify to most value-added changes based on all these factors, weighed against the burden of substantive changes both in terms of programming the survey instrument and disruptions to time series data.
Second, recruitment was challenging, resulting in a less diverse sample than desired. This was consistent with recent reports of low response rates for surveys in general, and federal surveys in particular, with Hispanic response rates at a historic low.7 The sample was overall higher income, and had fewer Black and Hispanic respondents than desired. Faced with time and resource constraints, the research team supplemented advertising with a convenience sampling approach, tapping into respondent and staff personal networks. Diverse recruitment across types of care – a primary focus at the planning stage – was achieved, although a larger sample size would have allowed for even greater variation, particularly across the many types of center-based care referenced in various sections of this report.
Third, again due to time and resource constraints, the Spanish language translations could not be conducted using the gold standard method of translation-back translation (i.e., translating the English items to Spanish, then back-translating the Spanish items to English to identify parity or disconnects). While we feel confident that the Spanish language recommendations are strong, future efforts should include plans to complete a thorough translation verification. Spanish survey item revisions were based on: (1) cognitive interviews findings; (2) native Spanish speaker recommendations to improve wording choice, grammar, and syntax; and (3) housekeeping changes made to parallel revisions made to the English survey and the PFI questionnaire. During fielding, Spanish survey items that were revised to parallel English survey items were one testing iteration behind all other items (i.e., all English survey items and Spanish survey items that were revised based on cognitive interviews) because those items were not revised until a round of cognitive interviews were complete and revisions were made. Note: We recommend completing more Spanish language cognitive interviews to ensure that all revised Spanish items are indeed culturally meaningful and work as intended.
Future Work (NHES: 2019 and 2022)
NCES should plan to address three types of issues in the future work on the ECPP: 1) a review of items identified as needing monitoring and/or additional cognitive laboratory work during this round of study; 2) monitoring of user success with navigational supports throughout the protocols, including instructions, section headings, and skip pattern instructions and symbols (e.g., arrows); and, 3) scrutiny of performance related to new survey administration methods.
Items Needing Additional Monitoring or Development
Due to time constraints, there were a number of items that were deferred for monitoring and/or future cognitive laboratory work. Retained items that warrant further examination are listed below (see notation in Appendix 1 for details).
5. Is this care provided in your home or another home?
6, 24, 52. About how long does it take to go from your home to this [relative’s home/ care provider's home/ program]?
18. Does this child have any other care arrangements with a relative on a regular basis?
28 (Spanish version). “¿Qué tan bien cubre este proveedor, que no es el pariente de este niño(a), las horas del cuidado que usted necesita para …"
41. Is this child now attending a day care center, preschool, or prekindergarten program not in a private home?
44. Does this program teach religious content to the children?
56. Do any of the following people, programs, or organizations help pay for this child to go to this program?
64. (NEW) Have you ever searched for care for this child?
65. How much difficulty did you have finding the type of child care or early childhood program you wanted for this child? (monitor new response options)
67. (NEW) Did you have a care arrangement for this child in the past year?
68. (NEW) What was the main reason your household chose the care arrangement(s) that you chose for this child?
69. How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time?
85: Has a health professional told you that this child has any of the following conditions? Mark one box for EACH item below. Monitor for comprehension of response options.
103. Including children, how many people live in this household?
104. We are interested in learning about how the people in your household are related to this child. How many of the following people live in this household with this child?
In addition, future researchers may wish to revisit some of the tested/dropped items listed in Appendix 1 to see if they can be improved upon. This includes items recommended by the January 2017 ECE subject matter expert panel that tap into important constructs of interest including: child care decision making; school readiness; and screening, assessment, and referrals for developmental concerns. Work to develop these items should continue.
Navigational Supports
Our data revealed a number of difficulties among respondents navigating the many different sections of this lengthy survey. Electronic administration methods may help resolve this matter, however, if mail-based methods (using pen and paper) will continue, this will be important to monitor. Many respondents read the instructions and articulated when survey navigation seem to contradict the instructions. For example, inside the cover page, instructions say “To answer a question, simply mark [X] the box that best represents your answer.” However, for a few items, open ended or “fill in” responses exist. Respondents also sometimes did not notice the different section headings or skip pattern instructions and symbols (e.g., arrows). Some of this was theorized to be related to the spacing and size of the text and symbols. At a minimum, for future paper-based administration, we suggest adding instructions explaining the concept skip patterns in the introductory section of the survey. Respondents may also benefit from an orientation to the three different types of care they will be asked to consider, prior to beginning the survey.
Appendix 1. Tested and Revised ECPP Items
2016 ECPP Item # |
Final Item #8 |
Original wording and response options |
Issues and # of Rs with issue (R=Respondent; FC=Focal Child) |
Decision9 & rationale |
Final item wording and response options |
Recommend future testing? |
1 |
1 |
Is this child now receiving care from a relative other than a parent or guardian on a regular basis, for example, from grandparents, brothers or sisters, or any other relatives? -No -Yes |
1 R had difficulty deciding what would quality as a “regular basis.” |
No change: this issue only occurred with one R |
Is this child now receiving care from a relative other than a parent or guardian on a regular basis, for example, from grandparents, brothers or sisters, or any other relatives? -Yes -No |
N/A |
2, 18 |
2, 21 |
Are any of these care arrangements regularly scheduled at least once a week? -No -Yes
|
Round 2: 3 Rs marked “no” because they have a contract with providers (nannies) and they don’t consider these “scheduled.” |
No change because it is part of a key item that has been asked the same way for a long time. |
Are any of these care arrangements regularly scheduled at least once a week? -Yes -No |
N/A |
3 |
3 |
These next questions are about the care that this child receives from the relative who provides the most care. How is that relative related to this child? -Grandmother/ Grandfather -Aunt/Uncle -Brother/Sister -Another relative |
2 Rs struggled when determining which relative to answer for (1 R in round 3, 1 R in round 4). |
We separated the instruction from the item stem.
We did not change the item stem itself.
|
How is that relative related to this child? -Grandmother/ Grandfather -Aunt/Uncle -Brother/Sister -Another relative |
N/A |
4 |
4 |
How old is the relative who provides the most care to this child? _____ age |
No issues from 2016 ECPP. |
No change. |
How old is this relative who provides the most care to this child? _____ age |
N/A |
5 |
5
|
Is this care provided in your home or another home? -Own home -Other home -Both |
1 R had an issue in round 2: FC had 50/50 custody arrangement between each parent’s home. While care was divided between two homes, both were the FC’s homes. |
NCES suggested we continue to monitor the item and revisit it for 2022 revisions. No change. |
Is this care provided in your home or another home? -Own home -Other home -Both |
Continue to monitor this item, potentially change in 2022. |
New item |
6, 24, 52 |
N/A |
Round 2: 2 Rs said that it depends on other factors such as traffic.
Round 3: 1 R suggested we change “on average” to “based on your regular commute.” |
After round 2, we added “on average” to the beginning of the question stem.
After round 3, we decided to change “on average” to “about.” |
About how long does it take to go from your home to this [relative’s home/ care provider's home/ program]? -Less than 10 minutes -About 10 to 20 minutes -About 20 to 30 minutes -About 30 minutes to 1 hour -More than 1 hour |
Monitor. |
New item |
Dropped |
N/A |
This was a new item that we introduced in round 1. Rs had difficulty with this item across all 4 rounds, since mileage is not salient to them. |
NCES determined we would drop the item. |
About how far is this [relative's home/care provider's home/program] from your home? - Less than 1 mile - Between 1 and 2 miles - Between 2 and 4 miles - Between 4 and 10 miles - Over 10 miles |
Item has been dropped. |
6, 21, 42 |
7, 25, 47 |
How many days each week does this child [receive care from this relative/person/go to this program]? ___ days each week |
Round 1: 1 R indicated variability in days per week that the child receives care from provider. Round 3: 2 Rs requested “on average” or “about” since time fluctuates. |
After discussion of how parents have fluctuating answers for estimate questions like this, the word “About” was added. |
About how many days each week does this child [receive care from this relative/person/go to this program]? ___ days each week |
N/A |
7, 22, 43 |
8, 26, 48 |
How many hours each week does this child [receive care from this relative/person/go to this program]? ____hours each week |
Round 3: 2 Rs requested “on average” or “about” since time fluctuates. One wrote in an answer in hours per day, missing the week instruction. Similar to questions 8/26/49, so similar solution implemented. |
After discussion of how parents have fluctuating answers for estimate questions like this, the word “About” was added. |
About how many hours each week does this child [receive care from this relative/person/go to this program]? ____hours each week |
N/A |
New item |
9, 27, 49 |
N/A |
New item added: “How well does this [relative care arrangement/ non-relative care arrangement/ program] cover the hours of care needed for... A. Work hours? B. Time to run errands? C. Free time?” with response options: “'Work Hours: -Not well -Somewhat well -Well -Very well -Not applicable Time to run errands/Free Time: -Not well -Somewhat well -Well -Very well -Not applicable: I do not expect this care arrangement to cover [time to run errands/my free time]”. Round 3: 1 R thought “work” was the only need. Another R found the question stem confusing. Across the parallel items, many Rs commented on this and noted they did not expect care arrangements to cover for free time/errands. They had difficulty selecting a response option because of this. |
NCES decided to remove the sub-items asking about errands and free time since Rs round this strange and difficult to answer. |
How well does this [relative care arrangement/ non-relative care arrangement/ program] cover the hours of care needed for work? -Not well -Somewhat well -Well -Very well -Not applicable |
N/A |
10, 27 |
12, 32, 54F&G |
Will this [relative/care provider/program] care for this child when the child is...? sick but does not have a fever? -Yes /No sick and has a fever? -Yes/No |
Round 2: R using relative care suggested adding a “don’t know.” Round 3: 2 Rs thought fevers were not a salient distinction, noting that the fever is not always the point at which there can be no child care. |
No change, instruction box added “Mark one box for EACH item below.” |
Will this [relative/care provider/program] care for this child when he or she is… Mark one box for EACH item below sick but does not have a fever? -Yes /No sick and has a fever? -Yes/No |
N/A |
New item |
13, 33 |
N/A |
New item added. Round 2: 1 R suggested adding “on average” to capture fluctuation in number of children. Round 3: 1 R was not sure whether they should be thinking of all children in the care arrangement or the exact number of children in each care provider’s supervision. |
No change. |
How many other children does this relative/provider care for while caring for this child? -None -1-2 -3-5 -6 or more |
N/A |
11, 29, 48 |
14, 35, 55 |
Is there any charge or fee for the care this child receives from this [relative/care provider/program], paid either by you or some other person or agency? -No -Yes |
Round 3: 1 R thought about a nanny agency that charged fees to find care. Round 4: 1 R was confused about whether to include fees for food and after care. |
Monitored and decided not to change the item. |
Is there any charge or fee for the care this child receives from this [relative/care provider/program], paid either by you or some other person or agency? -No -Yes
|
N/A |
12, 30, 49 |
15, 36, 56 |
Do the following people, programs, or organizations help pay this [relative/person/program] to care for this child?
-A relative of this child outside your household who provides money specifically for that care, not including general child support -Temporary Assistance for Needy Families, or TANF -Another social service, welfare, or child care agency -An employer, not including a tax-free spending account for child care -Someone else |
Round 3: 1 R missed “outside your household.” Another R emphasized importance of flexible spending account, no change needed. A separate R answered yes to “e” even though she was thinking of her spouse, who is in her household. |
Adjusted the language for response option b. |
Do any of the following people, programs, or organizations help pay this [relative/person/program] to care for this child?
-A relative of this child outside your household who provides money specifically for that care, not including general child support -your state welfare or family assistance program (this may be called Temporary Assistance for Needy Families [TANF], or something else) -Another social service, welfare, childcare, or other kind of agency -an employer, not including a tax-free spending account for child care -someone else |
N/A |
13, 31, 50 |
16, 37, 57 |
How much does your household pay for this [relative/person/program] to care for this child, not counting any money that may be received from others to help pay for care? Is that amount per… -Hour -Day -Week -Every 2 weeks -Month -Year -Other ‐ Specify: |
Round 2: 1 R asked about non-cash benefits, such as housing, given to provider. Round 4: 1 R was confused about whether to include after care and food fees. A few Rs had to go back and change their answer once they saw they could select the frequency. 1R said semester should be included as a payment frequency (for 61) |
No change: NCES staff looked back at past data to see if “semester” should be included. It was only brought up once during round 4, no other times. |
How much does your household pay for this [relative/person/program] to care for this child, not counting any money that may be received from others to help pay for care? Is that amount per… -Hour -Day -Week -Every 2 weeks -Month -Year -Other ‐ Specify: |
N/A |
15 |
18 |
Does this child have any other care arrangements with a relative on a regular basis? -No -Yes
¿Tiene este niño(a) algún otro arreglo de cuidado infantil con un pariente con regularidad? -No -Si |
Round 3: 1 R criticized "regular basis" as being ambiguous. This language makes people think--sometimes causes them to look ahead for clues or frames of reference in other items.
Spanish question: Respondents had to read the question multiple times and mentioned that it was long and unclear. |
No change. |
Does this child have any other care arrangements with a relative on a regular basis? -No -Yes
¿Algún otro pariente cuida a este niño(a) con regularidad? -No -Si |
Monitor how respondents interpret “regular basis.” |
19 |
22/23 |
These next questions are about the care that this child receives from someone who is not related to him or her who provides the most care. Is this care provided in your own home or another home? -Own home -Other home -Both |
Round 4: 1 R suggested we change the response option "other home" to instead say "another home," so that it matches the question stem. |
No change. |
These next questions are about the care that this child receives from someone who is not related to him or her who provides the most care. Is this care provided in your own home or another home? -Own home -Other home -Both
22. These next questions are about the care that this child receives from someone who is not related to him or her who provides the most care.
Does this person who cares for this child live in your household? -Yes -No
23. Is this care provided in your own home or another home? -Own home -Other home -Both |
N/A |
24 |
29 |
Was this care provider someone you already knew? -No -Yes |
N/A |
No change. |
Was this care provider someone you already knew? -Yes -No |
N/A |
35 |
41 |
Is this child now attending a day care center, preschool, or prekindergarten program not in a private home? -No -Yes |
Round 1: 1 R found it difficult because it sounded like a double negative when they answered no. |
Discussed changing it to "Is this child now attending a day care center, preschool, or prekindergarten program other than a private home?” However, this item is part of a time series. No change, monitor. |
Is this child now attending a day care center, preschool, or prekindergarten program not in a private home? -Yes -No |
Monitor. If problem persists, consider "For the next question, do not include care that takes place in a private home. Is this child now attending a day care center, preschool, or prekindergarten program? |
37 |
Dropped |
Is this child's current program a day care program, a preschool program, or a prekindergarten program? -Day care -Preschool -Prekindergarten |
Round 1: Respondents had trouble distinguishing between the 3 options.
Round 2: 4 Rs did not know how to distinguish between these types of care.
Round 3: 3 Rs had difficulty distinguishing types of care. |
Discussion about providing more information about the types of care listed and ultimately decided to drop this item, since the types of care listed were not salient for many Rs. |
N/A |
Item has been dropped. Future research needed around terminology to accurately classify the various types of early care and education. |
39 |
43 |
Where is this program located? In a church, synagogue, or other place of worship In a public elementary or secondary school In a private elementary or secondary school At a college or university At a community center At a public library In its own building, office space, or storefront Some other place—specify |
Round 3: 1 R struggled because child’s center is in a church, not affiliated with that church. |
No change. |
Where is this program located? In a church, synagogue, or other place of worship In a public elementary or secondary school In a private elementary or secondary school At a college or university At a community center At a public library In its own building, office space, or storefront Some other place--specify |
N/A |
New item |
44 |
N/A |
Round 2: 1 R did not know the answer. |
No change: we considered adding “don’t know” as a response item, but no other Rs had an issue. |
Does this program teach religious content to the children? -Yes -No |
Explore this and related items in future cognitive testing |
45 |
51 |
What language does this child's main care provider or teacher at this program speak most when caring for this child? -English -Spanish -A language other than English or Spanish -English and Spanish equally -English and another language equally |
Round 1: 1 R wrote in their own response because they spoke English 80% and Spanish 20%, which isn’t equal. R should have chosen English, since we are looking at language “most” spoken. |
No change. |
What language does this child's main care provider or teacher at this program speak most when caring for this child? -English -Spanish -English and Spanish equally -A language other than English or Spanish -English and another language equally |
N/A |
46 |
53 |
Would you recommend this program to another parent? -No -Yes |
Round 1: 2 Rs were not sure what “program” referred to. |
No change. |
Would you recommend this program to another parent? -Yes -No |
N/A |
47 |
54 |
Does this program provide any of the following services to this child or your family? Mark one box for EACH item below. -Hearing, speech, or vision testing -Physical examinations -Dental examinations -Formal testing for developmental or learning problems -Sick child care when this child is sick but does not have a fever -Sick child care when this child is sick and has a fever
(next to each response option are 2 boxes - yes/no) |
Round 1: Rs were answering whether the program provided the services in general, but not specifically to their child/family.
Round 3: 1 R said that the distinction between having a fever/not having a fever was not salient, since sick child care is more dependent on the type of sickness rather than fever grade. |
After round 1 we revised the question stem to clarify. |
Has this program provided any of the following services to this child? Mark one box for EACH item below. -Hearing, speech, or vision testing -Physical examinations -Dental examinations -Formal testing for developmental or learning problems -Medication administration -Sick child care when this child is sick but does not have a fever -Sick child care when this child is sick and has a fever
(next to each response option are 3 boxes - yes/no/don't know) |
N/A |
New item |
Dropped |
N/A |
Round 3: 2 Rs were not sure what constituted as a referral.
Round 4: 2 Rs had difficulty distinguishing between referrals for screening and exams required for enrollment. |
After round 3: added an information box before the question, “Sometimes programs suggest that children may need services outside the program. This suggestion is called a ‘referral’.”
After round 4 NCES requested we drop the item. |
Has this child been referred by the program to be screened for: Hearing, speech, or vision testing Physical examinations Dental examinations Formal testing for developmental or learning problems (yes/no columns) |
Item has been dropped. |
51 |
58 |
How many children from your household is this amount for, including this child? -This child only -2 children -3 children -4 children -5 or more children |
Round 4: 4 Rs objected to this item coming after the earlier questions as its placement would’ve influenced their comprehension and responses. |
No change: NCES said this item was added to prevent respondents from mistakenly reporting on all children and exclude such cases from analysis. |
How many children from your household is this amount for, including this child? -This child only -2 children -3 children -4 children -5 or more children |
N/A |
52 |
59 |
Does this child have any other care arrangements at a day care center or preschool on a regular basis? -No -Yes |
Round 1: 1 R was not sure whether to consider before- and after-care as different arrangements since they are in the same building as the pre-k. |
No change: after monitoring further rounds, no other Rs had issues. |
Does this child have any other care arrangements at a day care center or preschool on a regular basis? -Yes -No |
N/A |
53 |
60 |
How many total hours each week does this child spend at those day care centers or preschools? ____ hours each week |
Round 3: 1 R suggested making it clear that question is about additional arrangements. |
Changed: added the word “other” to question stem. |
How many total hours each week does this child spend at those other day care centers or preschools? ____ hours each week |
N/A |
55 |
62 |
What is the main reason your household wanted a care program for this child in the past year? Mark ONE only. -To provide care when a parent or guardian was at work or school. -To prepare this child for school -To provide cultural or language learning -To make time for running errands or relaxing -some other reason -did not have care in the past year |
Round 1: 2 Rs did not think question applied to them because of confusion over the word “program.” Round 2: Several Rs wanted to choose more than one option. Round 3: 1 R did not know which arrangement to think about, 1 R wanted to check more than one option, 1 R thought it was repetitive. Round 4: 4 Rs wanted to select more than 1 option, 2 Rs said reason changed over time. |
Round 1: changed “care program” to “care arrangement.”
After round 4: NCES made revisions to response options. |
What is the main reason your household wanted a care arrangement for this child in the past year? Mark ONE only. -Did not have care in the past year -To provide care when a parent or guardian was at work or school. -To prepare this child for school -To provide cultural or language learning -To make time for running errands or free time -Some other reason |
N/A |
New item |
Dropped |
N/A |
Round 1: 1 R did not select "religious orientation" as being important, but later said religious orientation was very important - it is very important that the program have no religious orientation. Round 2: 1 R did not know at what point in the search for care to consider. Round 3: 2 Rs had same issue with “religious orientation,” noting that they did not want a religiously affiliated program but they did collect that information. One R answered the way we wanted, the other R did not. Round 4: 1 R felt this was not applicable for relative care. |
NCES decided to drop this item. |
What information did you get about providers when you were searching for care? Mark one box for EACH item below. YES/NO: Location Cost Hours of care Qualifications of the staff Activities of children in care Religious orientation Other ‐ Specify: |
This item was dropped. |
56 |
63 |
Do you feel there are good choices for child care or early childhood programs where you live? -No -Yes -Don't know |
Round 1: 1 R struggled to answer the question because there are good options but there is a lottery system and you have to win the lottery to get placed in a good arrangement. Round 2: 5Rs had comments on this item. Some wanted to differentiate between good choices/quality/cost. Round 3: 1 R wanted an N/A response option, 1 R wondered if “where you live” was about neighborhood or city. Round 4: we received more "very few" responses since adding to this item, since Rs began considering affordability. |
After round 2 we tested new phrasing: How many of the options for child care or early childhood programs where you live do you feel are high-quality and affordable? -None -Very few -Several -Many -All
After round 4, NCES decided the wording would stay the same as the 2016 ECPP. |
See original wording, but change response option order to: -yes -no -don’t know (Tested wording was: Round 2: Of your options for child care or early childhood programs, how many were both high quality and affordable? Rounds 3 & 4: How many of the options for child care or early childhood programs where you live do you feel are high-quality and affordable? -None -Very few -Several -Many -All.) |
N/A |
New item |
64 |
N/A |
N/A |
NCES asked that we add a new item after #68 from 11/12 version |
Have you ever searched for care for this child? -Yes -No [SKIP to next section] |
New item to be added to final version of questionnaire. |
New item |
68 |
N/A for ECPP 2016, but wording during round 4 was: This next question asks about the reasons you chose the care arrangement where this child spends the most time. Did you choose the arrangement because… Mark one box for EACH item below. a. There were no other choices of child care providers available to you b. There were other choices of child care providers available, but they did not have slots c. Cost/affordability d. Hours/reliability e. Location f. Quality of care g. This child's sibling enrolled at the same care arrangement
|
Round 1: 1 R was confused because all of the options ask about why she chose her child's program, but f and g ask about why she did not choose a different program. The same R pointed out that the question begins with "what" but all of the options are yes/no. Round 2: Several Rs struggled to map their situations onto the response options. Round 3: 1 R did not detect difference in response option a and b. Round 4: Rs were confused about option i: age requirements. |
After round 1: rephrased item stem to present as a yes/no question. Made further revisions after round 2. After round 4 NCES decided to drop the item and replace with an open-ended item: What was the main reason your household chose the care arrangement(s) that you chose for this child? [Open-ended text box where Rs can fill in their answers]. This item was added in response to the lack of consensus among respondents about predetermined response categories for item 69. |
What was the main reason your household chose the care arrangement(s) that you chose for this child? Please write your response in the box below. [Open-ended text box where Rs can fill in their answers |
Monitor this new open-ended question. |
New item |
Dropped |
N/A |
Round 1: 1 R said her reasons may have changed over time. Round 3: 1 R skipped because found it too difficult to choose. |
NCES decided to drop item after round 4. |
Of the reasons that you chose the care arrangement where this child spends the most time, what was the most important reason you listed above?
Write the letter from question 68 for the most important reason you chose the care arrangement where this child spends the most time. ___ Enter letter from question 69 |
This item has been dropped. |
57 |
65 |
How much difficulty did you have finding the type of child care or early childhood program you wanted for this child? -Have not tried to find care -No difficulty -A little difficulty -Some difficulty -A lot of difficulty -Did not find the type of child care program I wanted |
Round 2: 1 R mentioned that "a little" and "some" were not both necessary |
After round 4: NCES requested we revise response options. |
How much difficulty did you have finding the type of child care or early childhood program you wanted for this child? -No difficulty -A little difficulty -Some difficulty -A lot of difficulty -Did not find the type of child care program I wanted |
Monitor new response options.
|
58 |
72 |
What was the primary reason for the difficulty finding child care or early childhood programs? Mark ONE only. -Cost -Location -Quality -Lack of open slots for new children -Needed a program for children with special needs -Other reason - specify: ____ |
Round 1: 1 R thought question asked for the primary reason they chose their arrangement. Round 2: 4 Rs had difficulty choosing only ONE reason. 1 suggested adding "Philosophy/ pedagogy.” |
After round 4, NCES said we would use 2016 ECPP wording. |
See original wording. (Tested wording was: What was the main reason for the difficulty finding child care or early childhood programs? Mark ONE only. -Cost/affordability -Hours/reliability -Location -Quality of care -No spaces available/lack of open slots -Needed a program for children with special needs -Other - Specify ____) |
N/A
|
New item |
67 |
N/A |
N/A |
After round 4, NCES requested we add a new item. |
Did you have a care arrangement for this child in the past year? -Yes -No |
Monitor new item. |
New item |
Dropped |
N/A |
No Rs mentioned QRIS. |
After round 3, NCES decided to drop item. |
Did you visit a website that offers information about multiple early childhood care providers and features of the care they provide? -Yes -No |
This item was dropped. |
59 |
69 |
How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time? A. The location of the arrangement B. The cost of the arrangement C. The reliability of the arrangement D. The learning activities at the arrangement E. The child spending time with other kids his/her age F. The times during the day that this caregiver is able to provide care G. The number of other children in this child's care group H. Ratings on a website I. Recommendations from friends and family J. The religious orientation of the program Under each subitem: -Not at all important -A little important -Somewhat important -Very important |
Round 1: item was difficult for Rs with multiple care arrangements. 1 R wanted a N/A option. Round 3: 1 R said options did not capture references from neighborhood lists and strangers. 1 R reported using reviews, not ratings. Round 4: Rs asked what learning activities are. |
After round 2, we added two new response items for h (Website ratings were not available) and I (Family and friends did not provide recommendations). After round 4, NCES requested we add 2 new subitems (j and k) |
How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time? a. The location of the arrangement b. The cost of the arrangement c. The reliability of the arrangement d. The learning activities at the arrangement e. The child spending time with other kids his or her age f. The times during the day that this caregiver is able to provide care g. The number of other children in this child's care group h. Ratings on a website i. Recommendations from friends and family j. Qualifications of the staff k. Whether or not the program teaches religious content |
Monitor new subitems (j and k) |
60, 61, 62 (counted as 3 unique items in totals) |
70, 71, 72 (counted as 3 unique items in totals) |
About how many books does this child have of his or her own, including those shared with brothers or sisters? ____number of books How many times have you or someone in your family read to this child in the past week? not at all; _____ times About how many minutes on each of those times did you or someone in your family read to this child? _____ minutes |
Round 2: Rs had trouble estimating. Round 3: 1 R didn’t understand “times.” |
No change. |
About how many books does this child have of his or her own, including those shared with brothers or sisters? ____number of books
How many times have you or someone in your family read to this child in the past week? not at all; _____ times
About how many minutes on each of those times did you or someone in your family read to this child? _____ minutes |
N/A |
66 |
76 |
In the past month, have you or someone in your family visited a bookstore with this child? -No -Yes
Durante el mes pasado, ¿ha ido usted o alguien de su familia a una librería con este niño(a)? -No -Si |
Round 1: 1 R said they purchase books online.
Spanish: Four respondents in the PFI survey said “librería” is the “same thing” as biblioteca”, but when asked about where they bought books they said “libraría”. The Project team added “tienda de libros” and no further respondents have had trouble. |
No change. |
In the past month, have you or someone in your family visited a bookstore with this child? -No -Yes
Durante el mes pasado, ¿ha ido usted o alguien de su familia a una librería o tienda de libros con este niño(a)? -No -Si |
N/A |
69 |
78 |
Can this child recognize the letters of the alphabet? -No -Yes, some of them -Yes, most of them -Yes, all of them |
No issues during fielding. |
The word “correctly” was added based on expert panel suggestion. |
Can this child correctly recognize the letters of the alphabet? -No -Yes, some of them -Yes, most of them -Yes, all of them |
N/A |
New item |
80 |
N/A |
No issues during fielding |
No change. |
Can this child recognize the beginning sound of a word? For example, can this child tell you that the word “ball” starts with the “buh” sound? -Yes -No |
N/A |
New item |
Dropped |
N/A |
Across all 4 rounds: multiple Rs did not have an accurate understanding of rhyming or whether their child was capable. |
After round 4, NCES decided to drop this item since it needs more development. |
Can this child rhyme words? -Yes -No -Don’t know |
This item was dropped. |
New item |
81 |
N/A |
Round 3: 1 R interpreted this as allowing for a child communicating needs (crying) vs. truly explaining things. |
No change. |
How often can this child explain things he or she has seen or done so that you get a very good idea of what happened? -Always -Usually -About half the time -Sometimes -Never |
N/A |
70 |
82 |
How high can this child count? -This child cannot count -Up to 5 -Up to 10 -Up to 20 -Up to 50 -Up to 100 or more |
Round 3: 1 R said response options did not match FC’s ability. |
No change. |
How high can this child count? -This child cannot count -Up to 5 -Up to 10 -Up to 20 -Up to 50 -Up to 100 or more |
N/A |
New item |
83 |
N/A |
No issues |
New item. |
Can this child identify basic shapes such as a triangle, rectangle, circle, or square? -No, none of them -Yes, some of them -Yes, most of them -Yes, all of them |
N/A |
75 |
84 |
In general, how would you describe this child's health? -Excellent -Very good -Good -Fair -Poor |
For rounds 1 and 2 we tested alternative phrasing: [If this child has a care arrangement] Has this child’s care provider referred this child to be screened for any of the following? a. Hearing, speech, or vision b. Physical examinations c. Dental examinations d. Developmental or learning problems e. Other ‐ Specify [BLANK FIELD] |
Alternative phrasing from rounds 1 and 2 was similar to another earlier item on the survey, so after round 2 the phrasing reverted back to the 2016 version. |
In general, how would you describe this child's health? -Excellent -Very good -Good -Fair -Poor |
N/A |
76 |
85 |
Has a health, education, or early intervention professional told you that this child has any of the following conditions? Mark one box for EACH item below. YES/NO: An intellectual disability (mental retardation) A speech or language impairment A serious emotional disturbance Deafness or another hearing impairment Blindness or another visual impairment that can't be corrected with glasses An orthopedic impairment Autism Pervasive Developmental Disorder (PDD) Attention Deficit Disorder, ADD or ADHD A specific learning disability A developmental delay Traumatic brain injury Another health impairment lasting 6 months or longer |
Round 2: Some Rs had difficulty knowing what some options meant (e.g., PDD). |
No change. |
Has a health professional told you that this child has any of the following conditions? Mark one box for EACH item below. YES/NO: An intellectual disability, formerly known as mental retardation A speech or language impairment A serious emotional disturbance Deafness or another hearing impairment Blindness or another visual impairment not corrected with glasses An orthopedic impairment Autism Pervasive Developmental Disorder (PDD) Attention Deficit Disorder, ADD or ADHD A specific learning disability A developmental delay Traumatic brain injury Another health impairment lasting 6 months or longer |
Monitor for comprehension of response options. |
79, 81 |
88 |
79. Is this child receiving services for his/her condition? -No -Yes
81. Are any of these services provided through an Individualized Family Service Plan (IFSP), Individualized Education Program (IEP), or services plan? -No -Yes |
In the 2016 ECPP, item 79 asked if the child was receiving services, 80 asked about which sources provided the services, and 81 asked if the services were provided through an IFSP, IEP, or services plan. Prior to fielding, it was determined that 80 would be eliminated and 79+81 would be merged into one item. There were no issues throughout fielding. |
No change. |
Is this child receiving any services through an Individualized Family Service Plan (IFSP), Individualized Education Program (IEP), or services plan? -Yes -No |
N/A |
83 |
89 |
Since September, how satisfied or dissatisfied have you been with the following aspects of this child’s IFSP, IEP or services plan? |
This item was revised in March 2017, prior to fielding. Throughout fielding there were no issues. |
No changes throughout fielding. |
Thinking about the child’s IFSP, IEP or services plan, how satisfied are you with the service provider’s or school’s communication with your family? -Very satisfied -Somewhat satisfied -Somewhat dissatisfied -Very dissatisfied -Does not apply |
N/A |
85 |
91 |
Does this child’s condition interfere with his/her ability to do any of the following things? Mark ONE box for each item below. -Learn -Participate in play with other children -Go on outings -Make friends
Next to each response option are two boxes for “Yes” and “No” |
This item was revised in March 2017, prior to fielding. Throughout fielding there were no issues. |
No changes throughout fielding. |
Does this child’s condition interfere with his/her ability to do any of the following things? Mark ONE box for each item below. -Learn -Play with other children -Go on outings -Make friends
Next to each response option are two boxes for “Yes” and “No” |
N/A |
New item |
92 |
N/A |
This item was added in March 2017, prior to fielding. Throughout fielding there were no issues. |
No changes throughout fielding. |
If your child goes to a care arrangement outside of your home, does this child’s condition interfere with his or her ability to attend child care? -Yes -No -This child is not in care outside of the home |
N/A |
88 |
95 |
How old was this child when he/she first moved to the 50 United States or the District of Columbia? ___ age |
Round 4: 1 R wanted to respond in months instead of years. |
After round 4 we added an instruction box that says, "if younger than 1, enter 0” because PFI has it. |
How old was this child when he or she first moved to the 50 United States or the District of Columbia? If younger than 1, write 0. ____ age |
N/A |
89 |
96 |
Is this child of Hispanic, Latino, or Spanish origin? -No -Yes |
Round 1: 1 R was not sure since FC was adopted. Round 2: 1 R objected to these classifications. Round 3: Rs had difficulty distinguishing between race/ethnicity/nationality. |
Before round 1, more specific response options were added. After round 3, we added an option to indicate multiple origins. |
Is this child of Hispanic, Latino, or Spanish origin? -No, not of Hispanic, Latino, or Spanish origin -Yes, Mexican, Mexican American, Chicano -Yes, Puerto Rican -Yes, Cuban -Yes, another Hispanic, Latino, or Spanish origin; or more than one Hispanic, Latino, or Spanish origin. |
N/A |
90 |
97 |
What is this child's race? You may mark one or more races. -American Indian or Alaska Native -Asian -Black or African American -Native Hawaiian or other Pacific Islander -White |
Round 1: 1 R was not sure since FC was adopted. Round 2: 1 R wanted different options, but these match categories used by federal agencies. Round 3: R wanted an "other" option. NCES is not at liberty to change this. |
Instruction clarified for consistency with PFI. |
What is this child's race? You may mark one or more races. Mark all that apply. -American Indian or Alaska Native -Asian -Black or African American -Native Hawaiian or other Pacific Islander -White |
N/A |
91 |
98 |
What is this child’s sex? -Male -Female |
Round 2: 1 R found these options limiting. |
No change. |
What is this child’s sex? -Male -Female |
N/A |
92 |
99 |
Does this child live at this address and another address (for example, because of a joint custody arrangement)? Do not include vacation properties. -No -Yes |
Round 2: 1 R suggested changing to “Does the child live at more than one address?” |
No change. |
Does this child live at this address and another address (for example, because of a joint custody arrangement)? Do not include vacation properties. -Yes -No |
N/A |
94 |
101 |
What language does this child speak most at home? -This child has not started to speak (Go to Q96) -English (Go to Q96) -Spanish -A language other than English or Spanish -English and Spanish equally -English and another language equally |
Round 3: 1 R was skipped out of this item but felt it applied. Round 4: 1 R learned more than one language as their first languages simultaneously but English was NOT one of them. |
After round 4 we switched response options 4 and 5 because they did that on PFI. |
What language does this child speak most at home? -This child has not started to speak -English -Spanish -English and Spanish equally -A language other than English or Spanish -English and another language equally |
N/A |
95 |
102 |
Is this child currently enrolled in English as a second language, bilingual education, or an English immersion program? -No -Yes |
Round 3: 1 R was skipped out of item but felt it applied to them (FC is enrolled in immersion program but doesn’t speak yet). |
No change. |
Is this child currently enrolled in English as a second language, bilingual education, or an English immersion program? -Yes -No |
N/A |
96 |
103 |
How many people live in this household? INFO BOX: Include adults and children who are temporarily away from home (for example, living in college housing) if they have no other permanent home. ___ people |
Round 3: 1 R did not include FC. |
After round 3 decided to add “including children” to beginning of question and delete info box. |
Including children, how many people live in this household? ___ people |
Monitor new phrasing. |
97 |
104 |
How many of the following people live in this household with this child? Do not include this child in your answer.
Example: Brother (s)……. 2
This child's... Write '0' if none. Number a. Brother(s).....__ b. Sister(s).....__ c. Mother (birth, adoptive, step, or foster).....__ d. Father (birth, adoptive, step, or foster).....__ e. Aunt(s).....__ f. Uncle(s).....__ g. Grandmother(s).....__ h. Grandfather(s).....__ i. Cousin(s).....__ j. Parent's girlfriend/boyfriend/partner.....__ k. Other relative(s).....__ l. Other non-relative(s).....__ |
None. |
After round 3, we deleted the instruction box and added a sentence before the question. |
We are interested in learning about how the people in your household are related to this child. How many of the following people live in this household with this child?
Example: Brother (s)……. 2
Write '0' if none. Number a. Brother(s).....__ b. Sister(s).....__ c. Mother (birth, adoptive, step, or foster).....__ d. Father (birth, adoptive, step, or foster).....__ e. Aunt(s).....__ f. Uncle(s).....__ g. Grandmother(s).....__ h. Grandfather(s).....__ i. Cousin(s).....__ j. Parent's girlfriend/boyfriend/partner.....__ k. Other relative(s).....__ l. Other non-relative(s).....__ |
Monitor new wording. |
99 |
106 |
Which language(s) are spoken at home by the adults in this household? Mark all that apply. English Spanish or Spanish Creole French (including Patois, Creole, Cajun) Chinese Other languages ‐ Specify: |
Round 4: Rs were confused about Spanish Creole, which is mentioned in this item but nowhere else. |
After round 4 we removed Spanish Creole. |
Which language(s) are spoken at home by the adults in this household? Mark all that apply. English Spanish French (including Patois, Creole, Cajun) Chinese Other languages ‐ Specify: |
N/A |
101, 120 |
108, 126 |
Is this person male or female? -Male -Female |
None |
After round 4 we changed “this person” to be “this parent or guardian” in order to match wording on PFI. |
Is this parent or guardian male or female? -Male -Female |
N/A |
100, 119 |
107, 125 |
Is this person this child's… Biological parent Adoptive parent Stepparent Foster parent Grandparent Other guardian |
None. |
After round 4 we changed the question phrasing to match the PFI. |
Who is this parent or guardian in relation to this child? Biological parent Adoptive parent Stepparent Foster parent Grandparent Other guardian |
N/A |
102, 121 |
109, 127 |
What is this person's current marital status? Mark ONE only. -Now married -Widowed -Divorced -Separated -Never married |
Round 2: 1 R mistook “now married” as “not married.” Round 3: 2 Rs thought “now married” sounded odd. |
No change. |
What is this person's current marital status? Mark ONE only. -Now married -Widowed -Divorced -Separated -Never married |
N/A |
104, 123 |
111, 129 |
What was the first language this parent or guardian learned to speak? -English -Spanish -A language other than English or Spanish -English and Spanish equally -English and another language equally |
None
|
After round 4 we decided to rearrange the order of response options to match PFI. |
What was the first language this parent or guardian learned to speak? Mark one only. -English -Spanish -English and Spanish equally -A language other than English or Spanish -English and another language equally |
N/A |
105, 124 |
112, 130 |
What language does this parent or guardian speak most at home now? -English -Spanish -A language other than English or Spanish -English and Spanish equally -English and another language equally |
None. |
After round 4 we decided to rearrange the order of response options to match PFI. |
What language does this parent or guardian speak most at home now? -English -Spanish -English and Spanish equally -A language other than English or Spanish -English and another language equally |
N/A |
107, 126 |
114, 132 |
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia? ___ age |
None. |
After round 4 we added an instruction to write 0 if FC is younger than 1; did this to match PFI. |
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia? If younger than 1, write 0. ___ age |
N/A |
108, 127 |
115, 133 |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? -No -Yes |
None. |
New response categories to match rest of the survey. |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? -No, not of Hispanic, Latino, or Spanish origin -Yes, Mexican, Mexican American, Chicano -Yes, Puerto Rican -Yes, Cuban -Yes, another Hispanic, Latino, or Spanish origin; or more than one Hispanic, Latino, or Spanish origin. |
N/A |
109, 128 |
116, 134 |
What is this parent or guardian’s race? You may mark one or more races. American Indian or Alaska Native Asian Black or African American Native Hawaiian or other Pacific Islander White |
Throughout fielding, the addition that read “you may mark one or more races” was removed. |
After round 4, we added in another sentence to the question, “you may select more than one race.” |
What is this parent or guardian’s race? You may select more than one races. Mark all that apply. American Indian or Alaska Native Asian Black or African American Native Hawaiian or other Pacific Islander White |
N/A |
110, 129 |
117, 135 |
What is the highest grade or level of school that this parent or guardian completed? 8th grade or less High school, but no diploma High school diploma or equivalent (GED) Vocational diploma after high school Some college, but no degree Associate's degree (AA, AS) Bachelor's degree (BA, BS) Some graduate or professional education, but no degree Master's degree (MA, MS) Doctorate degree (PhD, EdD) Professional degree beyond bachelor's degree (MD, DDS, JD, LLB) |
Round 3: 1 R answered one way out loud, but selected a different response option on the questionnaire. |
No change. |
What is the highest grade or level of school that this parent or guardian completed? 8th grade or less High school, but no diploma High school diploma or equivalent (GED) Vocational diploma after high school Some college, but no degree Associate's degree (AA, AS) Bachelor's degree (BA, BS) Some graduate or professional education, but no degree Master's degree (MA, MS) Doctorate degree (PhD, EdD) Professional degree beyond bachelor's degree (MD, DDS, JD, LLB) |
N/A |
112, 131 |
119, 137 |
Which of the following best describes this person's employment status? Employed for pay or income Self-employed Unemployed or out of work Full-time student Stay at home parent Retired Disabled or unable to work |
Round 3: 1 R on maternity leave struggled with response options. 1 R said self-employed is the same as employed for pay or income. |
No change. |
Which of the following best describes this parent or guardian's employment status? Mark one only. Employed for pay or income Self-employed Unemployed or out of work Full-time student Stay at home parent Retired Disabled or unable to work |
N/A |
113, 132 |
120, 138 |
(If employed or self-employed) About how many hours per week does he or she usually work for pay or income, counting all jobs? ____ hours |
Round 3: Rs were confused by parenthetical text at beginning of question. |
After round 3 we removed the parenthetical text. |
About how many hours per week does this parent or guardian usually work for pay or income, counting all jobs? ____ hours |
N/A |
114, 133 |
121, 139 |
(If unemployed) Has this parent or guardian been actively looking for work in the past 4 weeks? -No -Yes |
None. |
After round 3 we removed the parenthetical text. |
Has this parent or guardian been actively looking for work in the past 4 weeks? -Yes -No |
N/A |
137 |
142 |
In the past 12 months, did your family ever receive benefits from any of the following programs? YES/NO: Temporary Assistance for Needy Families, or TANF Your state welfare or family assistance program Women, Infants, and Children, or WIC Food Stamps Medicaid Child Health Insurance Program (CHIP) Section 8 housing assistance |
Round 4: 1 R was unsure how to count employment. |
Response options were revised before round 1. |
In the past 12 months, did your family ever receive benefits from any of the following programs? YES/NO: Your state welfare or family assistance program (this may be called Temporary Assistance for Needy Families [TANF], or something else) Women, Infants, and Children, or WIC SNAP benefits, also known as Food Stamps Medicaid Child Health Insurance Program (CHIP) Housing assistance through a voucher or Section 8 |
|
138 |
143 |
Which category best fits the total income of all persons in your household over the past 12 months? Include your own income. Include money from jobs or other earnings, pensions, interest, rent, Social Security payments, and so on.
'$0 to $10,000 *** several more response options up until…*** $250,001 or more |
Round 4: 1 R asked pre or post tax? |
No change. |
Which category best fits the total income of all persons in your household over the past 12 months? Include your own income. Include money from jobs or other earnings, pensions, interest, rent, Social Security payments, and so on.
'$0 to $10,000 *** several more response options up until…*** $250,001 or more |
N/A |
140 |
144 |
Is this house… -Owned, or being bought by someone in this household, -Rented by someone in this household, or -Occupied by some other arrangement? |
Round 1: Some Rs didn’t like the use of the word house since not everyone lives in a house. Round 2: Some Rs thought question sounded awkward. |
After round 1 revised to read “house or apartment.” |
Is this house or apartment… -Owned, or being bought by someone in this household, -Rented by someone in this household? -Occupied by some other arrangement? |
N/A |
141 |
145 |
Do you have Internet access on a cell phone? -No -Yes |
Round 1: 1 R found item “weird” and asked if we were asking about a data plan. Round 4: 1 R asked about supervised or independent use. |
No change. |
Do you have Internet access on a cell phone? -Yes -No |
N/A |
142 |
146 |
Do you have Internet access at home on a computer or tablet? -No -Yes |
Round 1: 1 R found item “weird” and asked if we were asking about a wifi. Round 4: 1 R asked about supervised or independent use. |
No change. |
Do you have Internet access at home on a computer or tablet? -Yes -No |
N/A |
New item |
147 |
N/A |
Round 2: 1 R observed that if they select “never” they should not have to answer the next question about how child accesses internet. Round 3: 1 R suggested adding a question about duration of use by the child. Round 4: Rs asked what counted as learning activities. 1 R asked about supervised or independent use. |
After round 2, added a skip pattern so Rs who select never do not answer following question. |
How often does this child use the Internet at home for learning activities? -Every day -A few times a week -A few times a month -A few times a year -Never |
N/A |
New item |
148 |
N/A |
Round 2: phrasing read “does the child access the internet” – 1 R suggested changing “access” to “use.” Round 3: 1 R suggested adding a question about duration of use by the child. Round 4: Rs asked what counted as learning activities. |
After round 2, changed “access” to “use.” |
Does the child use the Internet for learning activities on… Mark one box for EACH item below. YES/NO: Computer? Tablet? Cell phone? |
N/A |
Appendix 2. ECPP Finding and Choosing Care for Your Child section
Reorganized Order Following Completion of Cognitive Interviews
Numbering reflects the starting number of this section in the final round of cognitive testing, not the final questionnaire. Skips in this appendix refer to numbering in this appendix not final order of questions.
65. Has this child ever attended a Head Start or Early Head Start program?
Yes
No
Don’t know
66. What is the main reason your household wanted a care program for this child in the past year? Mark ONE only.
Did not have care in the past year
To provide care when a parent or guardian was at work or school
To prepare this child for school
To provide cultural or language learning
To make time for running errands or free time
Some other reason
67. Do you feel there are good choices for child care or early childhood programs where you live?
Yes
No
Don’t know
Note: This is the same as the 2016 ECPP (tested alternative item dropped).
NEW: 68. Have you ever searched for care for this child?
Yes
No GO TO question 74
69. How much difficulty did you have finding the type of child care or early childhood program you wanted for this child?
No difficulty GO TO question 71
A little difficulty
Some difficulty
A lot of difficulty
Did not find the type of child care program I wanted
70. What was the primary reason for the difficulty finding child care or early childhood programs? Mark ONE only.
Cost
Location
Quality
Lack of open slots for new children
Needed a program for children with special needs
Other reason - Specify ____
Note: This is the same as the 2016 ECPP (tested alternative item dropped).
NEW: 71. Did you have a care arrangement for this child in the past year?
Yes
No GO TO question 74
72. What was the main reason your household chose the care arrangement(s) that you chose for this child?
73. How important was each of these reasons when you chose the child care arrangement or program where this child spends the most time?
The location of the arrangement
The cost of the arrangement
The reliability of the arrangement
The learning activities at the arrangement
The child spending time with other kids his or her age
The times during the day that this caregiver is able to provide care
The number of other children in this child's care group
Ratings on a website
Recommendations from friends and family
Qualifications of the staff
Whether or not the program teaches religious content
Note: Response options j and k are new to this item. Former response option j (“The religious orientation of the program?”) was dropped and is now addressed as a stand-alone option.
National Household Education Surveys Program (NHES) 2019 Types of Schooling Cognitive Interviews (OMB# 1850-0803 v.200)
Findings presented below in the “National Household Education Surveys Program (NHES) 2019 Types of Schooling Cognitive Interviews – Revision 2 (OMB# 1850-0803 v.216)” section.
National Household Education Surveys Program (NHES) 2019 Types of Schooling Cognitive Interviews - Revised (OMB# 1850-0803 v.212)
Findings presented below in the “National Household Education Surveys Program (NHES) 2019 Types of Schooling Cognitive Interviews – Revision 2 (OMB# 1850-0803 v.216)” section.
National Household Education Surveys Program (NHES) 2019 Types of Schooling Cognitive Interviews – Revision 2 (OMB# 1850-0803 v.216)
Parent and Family Involvement in Education (PFI):
2017 Cognitive Interviews Final Report
National Center for Education Statistics
Prepared by Sanametrix, Inc.
March 23, 2018
Table of Contents
1.1 Cognitive Interview Methodology 1
1.3 Data Collection and Analytic Approach 5
1.3.3. Measure Iterations and Versions 6
2.1.1 Findings from Cognitive Interviews on Navigation 7
2.1.3 Child’s Homeschooling 10
Appendix A. Navigation Interview Protocol A-1
Appendix B. New Content Interview Protocol B-1
Appendix C. PFI Fielding First Draft C-1
Appendix D. PFI English and Spanish Final Questionnaires D-1
1. Introduction
In the summer and fall of 2017, the National Household Education Surveys Program fielded the 2019 Parent and Family Involvement in Education Survey (PFI) cognitive interview study. This report details the findings from the study which was conducted in order to assess the feasibility of a combined instrument; previous years had utilized the Parent and Family Involvement in Education Survey, Enrolled (PFI-E) and the Parent and Family Involvement in Education Survey, Homeschooled (PFI-H) questionnaires separately. As some families elect to involve their children in more than one type of educational program, and as alternate educational formats are growing in popularity (i.e., online learning), NHES moved to combine instruments into a single questionnaire that asks parents about all types of educational options. As such, the PFI has necessitated a new organizational format and includes new items not in the previous 2016 versions. In order to assess how respondents were able to understand items in the questionnaire and navigate through the measure, a set of cognitive interviews were conducted with participants who fit the demographics of future respondents of the 2019 survey.
Cognitive interviews were conducted in the summer and fall of 2017. These interviews were utilized in order to assess what items were problematic or misunderstood by respondents, and to test the new iterations of these items as the measures were updated throughout the fielding. Thus the fielding was a dynamic process in which respondent feedback was used to modify item text, and these revised items were then tested with additional respondents in order to determine whether these questions remained problematic.
The final PFI consists of 146 items. These items included questions about homeschooling, online learning, the child’s school, and family background. Interviews were conducted in two waves, with the initial wave focused on respondent navigation through the survey, and the second focused on the understanding of item content. These two waves were structured as such in order to determine that respondents could successfully navigate through the combined version of the questionnaire before proceeding with the combined instrument. Once this was determined successful, the second wave focusing on item content commenced. All interviews were conducted between August 9, 2017 and November 22, 2017.
1.1 Cognitive Interview Methodology
Cognitive interviews were structured to last for about 60 minutes, although actual interview length varied. Respondents were told during the screening and consent process that interviews would last for about 60 minutes. Interviewers were instructed to allow each respondent to continue participation for as long as they felt comfortable, but to bring their attention to the time when interviews went over the estimated time by approximately 10-15 minutes.
Interviews were conducted using retrospective probing and the “think-aloud” approach with concurrent probing. Two protocols were created: one for the initial set of navigation interviews, and one for the second set of item content interviews (see: Appendices A for the protocol specific to navigation, and B for the protocol specific to the testing of new content). These protocols were created in order to identify when respondents incorrectly navigated through the survey based on the skip logic, and in order to identify misunderstanding and misinterpretation of item wording. Trained interviewers from Sanametrix, Inc. and Activate Research conducted all interviews. Training was conducted in order to instruct interviewers on cognitive interview techniques generally and the PFI protocols in particular. The interview training was conducted by a senior researcher at Child Trends on cognitive interview techniques, and by senior researchers at Activate Research on the interview protocols. Interviewers with novice experience were observed and provided guidance by senior interviewers on the research team.
Interviews took place at a location of the respondent’s choosing, typically at a coffee shop, public library, or a quiet location within or close to their office space. Three of the 50 completed interviews were conducted remotely through web conferencing software. At the beginning of the meeting, the interviewer explained that the interview and the information that the respondent would share would help the research team to make sure that other parents or guardians like the respondent understood the questionnaire measure. Participants were given the consent form to review and sign and assured that their participation was completely voluntary and that the information they disclosed would be used for research purposes only. In the interviews conducted with the item content protocol (after the first ten navigation interviews were complete), interviewers explained the think-aloud process to the respondents and went through a practice think-aloud question and answer. Interviewers were trained to focus more attention on respondents’ experiences with all questionnaire items that were new or revised than on items that were identical to NHES:2016 items. The first draft of the 2019 PFI questionnaire is included as appendix C.
The first set of ten navigation interviews were conducted by allowing participants to move through the questionnaire uninterrupted until the last navigation item of interest was answered, and the participant had moved on to the next question. At that point, interviewers used a retrospective verbal probing technique and asked participants about specific items in the questionnaire. Interviewers used probes provided to them in the interview protocol and used their own attention and observation to identify when to continue probing respondents and when to ask for clarification.
1.2 Sample
Participants in the cognitive interviews were recruited nationally, with a majority of respondents being recruited from the DC metro area. As was noted previously, the initial ten interviews focused on navigation were conducted in-person. This was done so that interviewers could fully observe how respondents moved through the survey, paying attention to non-verbal cues. As such, recruitment was initially more concentrated on the DC metro area so that interviewers would be able to travel to these respondents. However, recruitment was also open nationally as a large portion of the sampling population (i.e., parents of homeschooled children) was determined to be more difficult to recruit.
Initial recruitment efforts commenced in June 2017 and focused on the investigation and compilation of local and national organizations that represented parents of children who are homeschooled, children who attend virtual school, and children who attend a school other than their designated public school option (i.e., school choice). Outreach to leaders and administrators at these national organizations were made with requests for passing on study information to the parents engaged with their organizations. Not all organizations were responsive, but many of those that responded agreed to send study information out to the organization’s listserv or to post it in a newsletter to parents. Another recruitment effort was asking contractor employees to send information about the study to their friends, acquaintances, and family. In addition, interviewers asked participants to pass on study information to other parents they know who may also be eligible and interested in participating in the study. The most beneficial recruitment efforts for this project were snowball recruitment strategies from both personal/professional contacts and from participant contacts. Specifically, these contacts would spread the study information by messages sent to parent listservs or by contact to their own acquaintances. This then resulted in potential participants reaching out to the recruiters and expressing interest in participating in the study.
In exchange for their time, respondents were offered a monetary incentive, which was distributed in cash for in-person interviews and by check for remote/virtual interviews. At the start of recruitment, the incentive was $40. This was then increased to $50, and finally was increased to $60. In addition, after the initial set of ten navigation interviews, respondents could choose to conduct the interview remotely if desired or needed. This was especially beneficial in the recruitment of parents whose children were either homeschooled or enrolled in virtual school. Remote interviews were either conducted over the phone, or by a secure WebEx web conference.
Recruitment resulted in 50 completed participant interviews for the PFI. These included interviews with parents of homeschooled children, parents of children enrolled in private school or public charter schools that they elected in place of the district-assigned school, parents of children attending full-time virtual school, and parents whose children attend their district-assigned school. The full demographics overview may be seen below in Table 1.
Table 1. Demographics Overview of All Respondents
Version |
Type of School |
Education |
Income |
Race/ethnicity |
1.3 Data Collection and Analytic Approach
1.3.1 Data Collection
Interviewers met respondents at a location of their choosing, or via a phone call or WebEx secure web conference based on their preference. Interviewers arrived at each cognitive interview equipped with an audio recorder, the interview protocol, questionnaire measures for the interviewer and respondent, and note-taking materials. Interviewers greeted participants and explained how their participation would help in the study, their rights as participants, and the confidentiality of their responses. Interviewers provided participants with the consent form and received a signed copy from the participants prior to the interview. Interviewers also explained that the interview would be audio-recorded in order to help the research team learn more from the respondent’s answers. All participants agreed to be recorded.
During the cognitive interview, the audio recorders recorded participant speech. Interviewers observed participant speech and behavior and made their own notes of behavior or responses of relevance. Post-interview, interviewers reviewed and expanded on their notes, and summarized the cognitive interview using an interview summary template that was created to summarize the interview with regard to each questionnaire subsection, as well as general notes about the respondent’s talkativeness, level of interest, and contextual data quality.
For the navigation interviews, the interview process was to have the respondents complete the questionnaire up until the item following the last item of navigational concern. After that item, interviewers asked the respondent to please stop there and to turn back to the beginning of the survey, and then asked respondents about the specific items of interest. For each of these items, interviewers used the probes provided in the interview protocol, and used their own judgment to ask ad hoc questions when they believed they needed further explanation based on participant response. For the content interviews, respondents were given an example question prior to starting the questionnaire in order to practice the “think-aloud” process. This question asked participants to count the number of windows in their home. During this interview, respondents were asked to think-aloud through their responses to each item. Concurrent probing was used for the specific items of interest, and interviewers used the probes provided in the interview protocol as well as ad hoc questions for further clarification and explanation.
Hard-copy interview data were stored in secure locations within research team offices. Audio recordings of interviews were uploaded and kept securely on password-protected computers with secure servers used by the research team. After interviews were complete, interviewers entered summary notes into a standardized template. These notes were also kept securely on password-protected computers and secure servers.
1.3.2 Data Analysis Methods
Interviewers reviewed their notes and participant data following the interview to identify problematic items. A series of debriefing meetings were held throughout the data collection process, after a set number of interviews had been completed, in order to enumerate the item issues and discuss how and why items were problematic for respondents. Navigation specific debriefings were held after each initial set of five navigation interviews (n = 10) were conducted, with the purpose of deciding whether the combined version of the PFI was navigable and thus should be retained for 2019 data collection. The final debriefings was held after the remaining interviews were conducted (n = 40).
These debriefings allowed for the corroboration of item issues across interviews. Interviewers identified which items were problematic and entered item issues into a collaborative spreadsheet. These issues were summed across the total set of participant interviews to determine which items were persistently problematic across respondents. During the debriefing meetings, items that were commonly problematic across respondents were discussed in depth in order to illustrate how and why respondents found these items problematic. Based on these discussions, item wordings were either changed in order to provide more clarity based on respondent concerns or retained as-is with instructions for interviewers to monitor these items in future interviews. Specific item changes from the initiation of the study to the final version are outlined in further detail in the Discussion section below.
1.3.3. Measure Iterations and Versions
The data collection was an iterative process that yielded rolling questionnaire and protocol changes. The English questionnaire measure existed in three different versions during data collection. Relatedly, the English interview protocol existed in two different versions for navigation, and three different versions for new content. The Spanish version of the PFI existed in three different versions. The Spanish navigation protocol existed in one version, and the Spanish new content protocol went through three different versions.
Items in this report are largely based on English cognitive interviews. However, important questionnaire changes occurred based on the Spanish cognitive interviews. For example, in Question 30 in the Child’s School section, the wording for “including a charter school” was dropped for the first response item, “A public school located in a physical building”, as numerous Spanish speaking respondents found this wording confusing and answered no based on their child’s non-attendance at a charter school.
Other important feedback from the Spanish-language cognitive interviews included amendments to the items that ask about when a parent or guardian moved to the United States. Specifically, several respondents indicated that the person in question came to the United States when they were younger than 1 year old, while the question was formulated to only write in an age in years. This item has now been revised in both the English and Spanish version to note that respondents should write in “0” if the parent who came to the United States did so when they were less than a year old. In addition, cognitive interviews conducted in Spanish revealed that these respondents had trouble with parentheticals for items in the Household Members section. As such, these parentheticals were removed in both the Spanish version and the English version.
2. Findings
2.1 The PFI
Items in the PFI that were of great importance for this fielding were those that directed respondents through the survey in order to answer items in subsections that were applicable to the focus child (e.g., homeschooling questions for a child educated at home), and to prevent respondents answering items that did not apply to the focus child. As such, one other change to the PFI beyond the question revisions was the insertion of an informational bullet alerting respondents to watch for text boxes and arrows to direct them through the survey, with the inclusion of image examples. The other items in the PFI that were of high importance are those that were new additions to the 2019 questionnaire. Finally, items that overlap with the Early Childhood Programs and Participation questionnaire (ECPP) were revised to be congruous across these two surveys. That is to say, when item issues arose in one survey during fielding for questions that belong to both surveys, any final changes to those items were also implemented in the alternate questionnaire.
2.1.1 Findings from Cognitive Interviews on Navigation
Ten interviews were conducted in order to assess navigational troubles that respondents experienced while answering the combined version of the PFI. Although later PFI cognitive interviews could be conducted remotely, these ten interviews were all conducted in person in order for interviewers to attend to non-verbal signals and any potential cues of confusion or miscomprehension. These navigation-focused cognitive interviews revealed some important concerns. Specifically, several respondents expressed confusion about items related to the focal child’s brick and mortar school, as their child was primarily homeschooled, and they felt that these didn’t apply. Other respondents expressed confusion and frustration at answering duplicate questions as they moved through different subsections related to homeschooling and the child’s school. In addition, several respondents felt that they could not identify the type of school(s) that their child attends in the available options in the item asking respondents to identify this (Q2). One respondent initially selected one type of school for her child and progressed through the questionnaire before turning back to the item and questioning her response. One respondent mistakenly skipped over the entire subsection on Child’s Schooling due to a navigational error based on misreading the skip directions. Only during the retrospective probing did this respondent realize she should have answered the questions in this section.
The navigation-centered interviews ultimately determined that the PFI-C was navigable by respondents, but also identified important modifications to the format of the questionnaire. A significant format change was to move the Child’s Homeschooling section to the beginning of the questionnaire, after the Child’s Schooling section. In addition, it was decided to shorten the Child’s Schooling section, and to direct respondents onward to the applicable schooling section immediately after asking them to identify the type of school that the focal child attends. After identifying the type of school(s), the respondents would then be directed either to the Child’s Homeschooling section for homeschooled children, or the Child’s School section for enrolled children, or to the Child’s Homeschooling and then onward to the Child’s School section for children both schooled at home and in a school. In addition, several directional questions were added to the beginning and end of schooling subsections as a navigational check, in order to help ensure that respondents were in the correct subsection and responding to the appropriate questions.
2.1.2 Child’s Schooling
This section asks respondents to identify what grade their child is in, as well as the type of school that the child attends (Q2, in Table 2 below). Item Q2 was of navigational concern, as parents would either be directed to the section on homeschooling or on enrolled schooling based on the respondent’s answer. This item confused many respondents. One respondent didn’t realize that they could select more than one option. Many other respondents reported that the question wording, especially the text for homeschool items, confused them. Specifically, one respondent reported that the language for “AND partially enrolled in another type of school from the list above” was confusing. In addition, several respondents had trouble with the wording of “including co-ops”. The final wording for this item reflected these considerations. Item Q3 was problematic for several respondents, to the extent that one respondent reported that they didn’t see a “homeschool only” option, and another reported that only option e applies to her child, and not h or i, believing that she had to endorse either h or i based on the item phrasing. This item also changed wording based on these considerations. In addition, some homeschool respondents expressed confusion about answering items pertaining to the child’s school or answering repeat items based on the different types of schooling items in which their child was involved.
Table 2. Identification of Focal Child’s Type of School(s)
Item # |
Tested Item Wording |
Final Wording |
Q2 |
Students today take part in many different types of schools and education settings. What type of school does this child attend? Mark X one box for EACH item below.
a. A public school located in a physical building, including charter school b. A private Catholic school located in a physical building c. A private, religious but not Catholic school located in a physical building d. A private, not religious school located in a physical building e. Online, virtual, or cyber school for grades kindergarten through 12 f. College, community college, or university that is online, virtual, or cyber g. College, community college, or university located in a physical building h. Student is partially educated at home (homeschooled) AND is partially enrolled in another type of school from the list above i. Student does not attend a school in a physical building AND is not enrolled in an online, virtual, or cyber school (homeschooled only) |
Students today take part in many different types of schools and education settings. What type of school does this child attend? Mark X one box for EACH item below.
a. A public school located in a physical building, including charter school b. A private Catholic school located in a physical building c. A private, religious but not Catholic school located in a physical building d. A private, not religious school located in a physical building e. Full-time online, virtual, or cyber school for grades kindergarten through 12 f. College, community college, or university that is online, virtual, or cyber g. College, community college, or university located in a physical building h. Student is homeschooled, including co-ops |
When the Child’s Schooling section was extended, in the initial draft of the PFI-C, fielding revealed that respondents who answered questions about a focal child that was both homeschooled and enrolled in another type of schooling were frustrated about responding to redundant items. Based on these findings, a format modification was implemented which shortened the Child’s Schooling section, whereas previously the expanded Child’s Schooling section had items concerning schooling that were redundant with other sections. In addition, the subsections were rearranged so that the Child’s Homeschooling section directly follows the Child’s Schooling section. Item Q3, which ask respondents whether they marked Yes to “h. Homeschooled” from the list in Q3, now directs respondents to Q4 at the start of the Child’s Homeschooling section if they respond Yes, and directs them to Q30 at the start of Child’s School section if they respond No.
Table 3. Item Changes for Child’s Schooling Section
Item # |
Original Wording |
Final Wording |
Q1, information box |
If this child is not assigned a specific grade, mark the grade he/she would be in at a school with regular grades. |
If this child is not assigned a specific grade or is homeschooled, mark the grade level of the curriculum that the child receives. |
Q2, e |
Online, virtual, or cyber school for grades kindergarten through 12 |
Full-time online, virtual, or cyber school for grades kindergarten through 12 |
Q2, h |
Student is partially educated at home (homeschooled) AND is partially enrolled in another type of school from the list above |
Student is homeschooled, including co-ops |
Q2, i |
Student does not attend a school in a physical building AND is not enrolled in an online, virtual, or cyber school (homeschooled only) |
Item removed |
Q3 |
Did you mark Yes to “i. Homeschooled only” from the list in question 2 above? |
Did you mark Yes to “h. Homeschooled” from the list in question 2 above? |
Q4 |
Did you mark Yes to “a. A public school located in a physical building, including charter school” from the list in question 2 above? |
Item removed |
2.1.3 Child’s Homeschooling
This section asks respondents about the child’s homeschooling, including the amount of homeschool classes, who provides instruction, the resources used, and whether the child is also enrolled in virtual school. Respondents reported having trouble with the items that ask about who provides the instruction, about where they receive physical and online curriculum materials, and whether and how online classes and classes out of the home should be included as part of the focal child’s homeschool instruction.
In the final PFI, the wordings of several items were revised to reflect the inclusion of online and virtual classes in the child’s homeschool curriculum, and the inclusion of the instructor of online and virtual classes. Several other items were changed to help with clarity and reduce confusion. A new item directing respondents onward at the end of this section, Q29, asks parents whether or not the focus child is enrolled in a school; responding Yes routes them to the Child’s School section, while marking No routes them to the Families and School section.
Table 4. Item Changes for Child’s Homeschooling Section
Item # |
Original Wording |
Final Wording |
Q5, first response option |
This child is homeschooled for all classes or subject areas |
This child is homeschooled for all classes or subject areas, which may include coops, virtual/cyber/online courses, and home instruction provided by a private tutor or teacher |
Q7 |
Who is the person that mainly provides this child’s home instruction? * Mother * Father * Grandparent * Brother/sister * Another person – Who is that? |
Who is the person that mainly provides this child’s home instruction? * Mother * Father * Grandparent * Brother/sister * Teacher of online, virtual, or cyber school * Another person – Who is that? |
Q9 |
Is this child enrolled in any online, virtual, or cyber courses? * Yes, all the child’s courses are online, virtual, or cyber * Yes, more than half of the child’s courses are online, virtual, or cyber * Yes, less than half of the child’s courses are online, virtual, or cyber * No, none of this child’s courses are online, virtual, or cyber |
Is this child enrolled in any online, virtual, or cyber courses? * Yes, all the child’s courses are online, virtual, or cyber * Yes, about half or more than half of the child’s courses are online, virtual, or cyber * Yes, less than half of the child’s courses are online, virtual, or cyber * No, none of this child’s courses are online, virtual, or cyber |
Q18 |
In this question, we are interested in the online, virtual, or cyber resources that you use. Where do you get the online curriculum and materials you use to homeschool this child (for example, courses on the computer, streaming instructional videos, downloaded course materials)? Since September, have you used materials from… Mark X ONE box for EACH item below |
In this question, we are interested in the online, virtual, or cyber resources that are used in your home when a parent is providing instruction (e.g., streaming instructional videos, downloaded course materials)? Since September, have you used materials from… Mark X ONE box for EACH item below |
|
In this question, we are interested in the physical resources that you use. Where do you get the physical curriculum and materials you use to homeschool this child (for example, worksheets, textbooks, fiction/nonfiction books, DVDs, or videos)? Since September, have you used materials from… Mark X ONE box for EACH item below |
In this question, we are interested in the physical resources that you use. Where do you get the physical curriculum and materials you use to homeschool this child (e.g., worksheets, textbooks, fiction/nonfiction books, DVDs, or videos)? Since September, have you used materials from… Mark X ONE box for EACH item below |
Q22, response option j |
This child has a unique learning style? |
Item removed, new text: You have another reason for homeschooling this child? - Specify: |
Q22, response option k |
The schools that you would want this child to attend are too far from home? |
Item removed |
Q22, response option l |
You have another reason for homeschooling this child? - Specify: |
Item removed (moved to option j) |
Q29 |
New item |
Is this homeschooled child also enrolled in a school? * Yes * No |
2.1.4 Child’s School
This section asks respondents about the school that the child attends, including what the type of school is, whether it was their district-assigned school, how important different characteristics of the school were to the respondent, and whether the child is enrolled in virtual classes.
Respondents commonly expressed confusion or extended deliberation on the items that asked about the degree of choice they had in selecting the focal child’s school. Many respondents replied that they felt as though they had some choice, in that they had the option to try for a different school via the lottery system; respondents also expressed that the ultimate decision was out of their hands as it was left to chance. Respondents also commonly expressed confusion regarding the amount they spent on the focal child’s school fees, books and materials for school, and after school activities. Similarly, respondents were commonly unaware of what an Educational Savings Account is, what a school voucher is, or what a tax credit scholarship is. As such, these items were dropped from the final version of the questionnaire.
Item 30 is a new item that helps direct respondents to different items. If respondents select the first option, “A public school located in a physical building”, they are directed to GO TO item 31. If respondents select the second option, “A private school located in a physical building”, they are directed to GO TO item 35. If respondents select the third option, “An online, virtual, or cyber school”, they are directed to GO TO item 32. If respondents select the fourth option, “This child is homeschooled only”, they are directed to GO TO item 59.
Table 5. Item Changes for Child’s School Section
Item # |
Original Wording |
Final Wording |
Q30 |
New item |
This question helps to route you to questions appropriate for this child. Which best describes the physical or online/virtual/cyber school where this child is enrolled for the most hours? Mark X ONE only * A public school located in a physical building * A private school located in a physical building * An online, virtual, or cyber school * This child is homeschooled only |
Q38, item k |
k. Class size |
k. Number of students in class |
Q39 |
How did you find out about this child’s school? |
How did you find out about this child’s school? Mark X all that apply. |
Q39, last response item |
* None of the above |
* Other – Specify: (write in box inserted) |
Q42 |
About how many hours does this child attend a school each week? |
About how many hours does this child attend a school each week? |
Q44, item e |
New item |
This child needed extra help in a course or subject offered at his or her physical school |
Old item Q59 |
About how much did you spend last year on this child’s… School tuition? School fees? Books and materials for school? After-school activities? |
Item removed |
Old item Q60 |
During a typical week, what are the afterschool arrangements for this child? How about… Mark X one box for EACH item below. * Receiving care from a relative other than parent? * Receiving care in your home or another home from someone not related to this child? * Attending an after-school program at a school or in a center, either on a scheduled or a drop-in basis? * Spending time responsible for his or herself? |
Item removed |
Old item Q61 |
This school year, did your household use an Education Savings Account (ESA) from your state to purchase educational needs for this child, like private school tuition, tutoring, educational therapy, or textbooks? * Yes * No |
Item removed |
Old item Q62 |
This school year, did your household use a school voucher for this child? * Yes * No |
Item removed |
Old item Q63 |
This school year, did your household use a tax credit scholarship for this child’s private school tuition? * Yes * No |
Item removed |
2.1.5 Families and School
This section asks about household adults’ engagement in activities at the focal child’s school, communication received from the child’s school, and the respondent’s satisfaction with the child’s school. This section is specifically for children who are enrolled in a physical school building or enrolled in online or virtual school.
Item 59 is a new item at the start of this section that assists in directing respondents to different items. If respondents select the first option, “Homeschool for ALL subject areas”, they are directed to GO TO item 72. If respondents select the second option, “Full-time online, virtual, or cyber school”, they are directed to GO TO item 62. If respondents select the third option, “Any other type of school”, they are directed to GO TO item 60.
Table 6. Item Changes for Families and School Section
Item # |
Original Wording |
Final Wording |
Text at the start of this section |
If this child is homeschooled for ALL subject areas, GO TO question 71. If this child is enrolled only in online, virtual, or cyber school, GO TO question 62. For all others, please answer the following questions about the school located in a physical building where this child is enrolled for the most hours. |
Removed |
Q59 |
New item |
The questions in this section are about the physical or online/virtual/cyber school where this child is enrolled for the most hours. Which best describes that school? * Homeschool for ALL subject areas *Full-time online, virtual, or cyber school *Any other type of school |
2.1.6 Homework
This section asks about the amount of homework the focal child has, how the respondent and child feel about the amount of homework and helping the child with homework. No changes were made in this section from previous administrations.
2.1.7 Family Activities
This section asks about what activities the family engage in together that are educational or developmentally beneficial in nature, such as making arts and crafts, playing a sport or exercising, visiting a bookstore, or visiting a museum. There were no modifications to items in this section.
2.1.8 Child’s Health
This section asks about the child’s overall health, whether the child has been diagnosed with a disability or condition, and whether the child receives IEP services. One respondent was strongly opposed to the language “formerly known as mental retardation” in item Q76, but no other strong reactions to item wordings arose. There were no modifications to items in this section.
2.1.9 Child’s Background
This section asks about when the focal child was born, where the focal child was born, the focal child’s race and ethnicity, and language spoken at home. Item modifications include clarification of what to write in if the child was younger than 1 year old when moving to the United States and creating an option for respondents to report that the child identifies as more than one Hispanic or Latino ethnicity.
Table 7. Item Changes for Child’s Background Section
Item # |
Original Wording |
Final Wording |
Q84 |
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia?
|
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia? If younger than 1 write “0”. |
Q85 |
Is this child of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin |
Is this child of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin; or more than one Hispanic, Latino, or Spanish origin |
Q90 |
What language does this child speak most at home? Mark X ONE only. * English * Spanish * A language other than English or Spanish * English and Spanish equally * English and another language equally * This child is not able to speak |
What language does this child speak most at home? Mark X ONE only. * English * Spanish * English and Spanish equally * A language other than English or Spanish * English and another language equally * This child is not able to speak |
2.1.10 Household Members
This section asks about other people living in the household where the child resides. Respondents reported confusion about whether to include the focus child in number of people in the household, and confusion over the different requirements to include and then not include the child across items Q92 and Q93. In addition, one respondent reported that they did not understand the example provided for Q93. These item wordings were revised to provide greater clarification to respondents and are detailed below in Table 8.
Table 8. Item Changes for Household Members Section
Item # |
Original Wording |
Final Wording |
Q92 |
How many people live in this household? |
Including children, how many people live in this household?
|
Q93 |
How many of the following people live in this household with this child? Do not include this child in your answer. |
We are interested in learning about how the people in your household are related to this child. How many of the following people live in this household with this child? |
Q95 |
Which language(s) are spoken at home by the adults in this household? * English * Spanish or Spanish Creole * French (including Patois, Creole, Cajun) * Chinese * Other languages – Specify |
Which language(s) are spoken at home by the adults in this household? * English * Spanish * French (including Patois, Creole, Cajun) * Chinese * Other languages – Specify |
2.1.11 Child’s Family
This section asks about up to two parents and guardians living in the focal child’s household. Items that were changed include the removal of parentheticals in questions asking about parent and guardian employment as suggested by the Spanish language interviews. Several respondents reported that either they or the other household parent or guardian are both a stay at home parent and employed for pay, but this item was retained as is, with respondents being directed to mark one response option only for employment status. Some item changes arose based on considerations from the ECPP cognitive interview process. One wording change, to amend “this person” to “this parent or guardian” in all questions asking about parents one and two, was implemented in response to an NCES expert review of the questionnaire. This change ensures comparability in wording across questions, provides clarification, and aids in comprehension for these questions. Item changes made in this section are illustrated in the table below.
Table 9. Item Changes for Child’s Family Section
Item # |
Original Wording |
Final Wording |
Q100 |
What was the first language this person learned to speak? |
What was the first language this parent or guardian learned to speak? |
Q106 |
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia?
|
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia? If younger than 1 write “0”. |
Q107 |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin; or more than one Hispanic, Latino, or Spanish origin |
Q109 |
Where was this person born? * One of the 50 United States or the District of Columbia * One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands) * Another country |
Where was this parent or guardian born? * One of the 50 United States or the District of Columbia * One of the U.S. territories (Puerto Rico, Guam, American Samoa, U.S. Virgin Islands, or Mariana Islands) * Another country |
Q112 |
(If employed or self-employed) About how many hours per week does this parent or guardian usually work for pay or income, counting all jobs? |
About how many hours per week does this parent or guardian usually work for pay or income, counting all jobs? |
Q113 |
(If unemployed or out of work) Has this person been actively looking for work in the past 4 weeks? |
Has this person been actively looking for work in the past 4 weeks? |
Q127 |
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia?
|
How old was this parent or guardian when he or she first moved to the 50 United States or the District of Columbia? If younger than 1 write “0”. |
Q128 |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin |
Is this parent or guardian of Hispanic, Latino, or Spanish origin? * No, not of Hispanic, Latino, or Spanish origin * Yes, Mexican, Mexican American, Chicano * Yes, Puerto Rican * Yes, Cuban * Yes, another Hispanic, Latino, or Spanish origin; or more than one Hispanic, Latino, or Spanish origin |
Q133 |
(If employed or self-employed) About how many hours per week does this parent or guardian usually work for pay or income, counting all jobs? |
About how many hours per week does this parent or guardian usually work for pay or income, counting all jobs? |
Q134 |
(If unemployed or out of work) Has this person been actively looking for work in the past 4 weeks? |
Has this person been actively looking for work in the past 4 weeks? |
2.1.12 Your Household
This section asks about the household income, the type of housing resided in, and Internet use. Items in this section asking about Internet use are new for the 2019 questionnaire. Respondents expressed some confusion about Internet use and what constituted a learning activity, but ultimately made their own decision of what they believed this represents. Otherwise, the novel items for Internet used seemed to test well among respondents. Some respondents also mentioned that the phrasing of the question asking about whether the house is owned or rented awkward, and that the assumption in the item that the household must be a house was insensitive. This item was changed to reflect that respondents might reside in a house or apartment.
Table 10. Item Changes for Your Household Section
Item # |
Original Wording |
Final Wording |
Q137, item a |
a. Temporary Assistance for Needy Families or TANF b. Your state welfare or family assistance program |
a. Temporary Assistance for Needy Families (TANF) or your state welfare or family assistance program (combined into single item) |
Q139 |
Is this house… |
Is this house or apartment… |
Q140 |
New item |
Do you have Internet access on a cell phone? * Yes * No |
Q141 |
New item |
Do you have Internet access at home on a computer or table? * Yes * No |
Q142 |
New item |
How often does this child use the Internet at home for learning activities?
- Every day - A few times a week - A few times a month - A few times a year - Never |
Q143 |
New item |
Does this child access the Internet for learning activities on… Mark X one box for each item below. a. Computer? b. Tablet? c. Cell phone? |
Q144 |
Item moved here from earlier section |
How far do you expect this child to go in his or her education? Mark X ONE only. * Complete less than a high school diploma * Graduate from high school * Attend a vocational or technical school after high school * Attend two or more years of college * Earn a bachelor’s degree * Earn a graduate degree or professional degree beyond a bachelor’s |
3. Discussion
3.1 Summary of Findings
The cognitive interview process revealed valuable insight regarding how respondents understand and interpret items within the PFI questionnaire. The cognitive interviews were conducted with a diverse set of parents with school-aged children, including those who homeschooled their child, those who sent their child to the district assigned public school, those who elected to send their child to a private school or a public charter school, and those whose child was enrolled in either part-time or full-time virtual school.
Perhaps the most important finding from the PFI cognitive interview process was that the combined version of the Parent and Family Involvement in Education Survey is an instrument that respondents are able to navigate without too much difficulty. Issues initially arose in the cognitive interview process with parents of homeschooled children who felt as though they were repeating questions and answering questions that didn’t apply to them based on their choice of schooling for their child. As such, a significant restructuring change was made to the questionnaire. This change entailed creating a separate section for Child’s School, in which parents of enrolled children could answer questions about the brick and mortar or virtual school that their child attends. It also included reorganizing the survey so that the Homeschool section directly follows the initial questions in the Child’s Schooling section, allowing homeschool parents to navigate directly onward and answer questions that specifically apply to their child’s schooling.
Other changes pertaining to homeschooling options include condensing the choices in item Q2 for homeschooled children, removing the need to differentiate between “types” of homeschooling and reducing the cognitive burden and confusion for parents. Other changes to items in the Homeschool section include the addition and rewording of items asking about the amount of virtual and online classes and the type of person providing home instruction, including online and virtual teachers.
The cognitive interviews conducted revealed that certain items that ask about financial information did not test well with parents. Specifically, parents expressed confusion as to how to differentiate between school fees, costs for materials, and costs for activities. In addition, many parents were unaware what an Educational Savings account is, what a school voucher is, or what a tax credit scholarship is. As such, these items were dropped from the questionnaire.
Other key changes were based on the Spanish language cognitive interviews and sensitivity to respondent demographics and ethnicity. Specifically, respondents in both the PFI and the ECPP cognitive interview data collection indicated that a response option should be added to capture data about respondents with more than one Hispanic or Latino ethnic origin. In addition, several respondents reported that they had moved to the United States prior to the age of one, although the questions asking about this reflected a write-in box for years only. Additional text was added to this item with instructions for respondents who moved prior to the age of one year. Further, parentheticals were removed from items as Spanish-speaking respondents reported confusion based on their inclusion. An important change was also made to the item that asks about the household in which respondents reside, and now notes that this may be a house or an apartment.
3.2 Future Considerations
The changes detailed in the tables and summarized above represent changes that were deemed critical to implement in order to maximize respondent comprehension while answering the PFI based on the cognitive interview process. Other items indicated minor and idiosyncratic concerns, but none that warrant continued monitoring in future administrations.
National Household Education Surveys Program 2019 (NHES:2019) Spanish Recruitment Materials Focus Groups (OMB#1850-0803 v.225)
National Center for Education Statistics
National Household Education Surveys
Report of Focus Groups
Among Spanish-Speaking Parents and Non-Parent Adults
April 2018
Prepared by Hager Sharp
Table of Contents
Summary of Highlights and Conclusions 6
Detailed Findings by Location 11
Rockville, MD, April 5, 2018 11
Appendix A: Moderator’s Guides 36
Overview and Methodology
Overview
The National Household Education Surveys Program (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields two to three topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.
Modeling response propensity suggested that addresses from which NHES is unlikely to get a response share similar demographic characteristics, suggesting that response bias is an issue that may affect data quality. Analyses of the demographics of low-response propensity addresses indicate that occupants of these addresses are more likely to fall into at least one of the following groups:
Be age 35 or younger
Rent rather than own
Have less than a high school education
Have an income below $15,000
Identify as Hispanic
Anyone matching any one of these characteristics is less likely to respond. The response propensity model is a regression model that showed that, holding all other variables constant, each of the above characteristics is significantly related to responding at lower rates to previous NHES administrations. Therefore, NCES determined the need to focus research on predominantly Spanish-speaking parents and non-parent adults.
NCES hopes to improve NHES response rates and reduce response bias for the 2019 NHES by better understanding factors that motivate Spanish-speaking parents and non-parent adults to complete (or decline to complete) the NHES questionnaire. NCES has focused specifically on understanding motivations to complete the survey among occupants in sampled addresses that are modeled to be unlikely to respond to the survey. The findings highlighted in this report will be used to refine the materials and strategies used for NHES recruitment.
Methodology
Hager Sharp conducted a total of four in-person focus groups with Spanish-speaking parents and non-parent adults from different states and school districts to understand their perceptions of the NHES study and how they would respond if selected for NHES. We worked with recruiters at local focus group facilities in Rockville, MD (EurekaFacts) and Denver, CO (Fieldwork Denver) to recruit for and host the in-person focus groups. Focus group facility staff contacted people from their databases to recruit respondents who met the following eligibility criteria:
Two groups of Spanish-speaking parents (one in each location):
Parent of at least one child under age 18 in household
Indicated they speak “mostly Spanish” at home
Two groups of Spanish-speaking non-parents (one in each location):
Not a parent of a child under age 18 in household
Indicated they speak “mostly Spanish” at home
In addition to the screening criteria, we recruited for diversity in household income level (ensuring at least two participants per group have a household income of $40,000 or less), education level, participant age, and gender.
The configuration of the four focus groups is as follows:
Market |
Language |
Parent or Non-Parent |
Ethnicity |
Date |
Time |
Rockville, MD |
Spanish |
Non-Parent |
Hispanic |
April 5, 2018 |
6:00-7:30 pm |
Rockville, MD |
Spanish |
Parent |
Hispanic |
April 5, 2018 |
8:00-9:30 pm |
Denver, CO |
Spanish |
Parent |
Hispanic |
April 10, 2018 |
5:30-7:00 pm |
Denver, CO |
Spanish |
Non-Parent |
Hispanic |
April 10, 2018 |
7:30-9:00 pm |
We recruited 10 participants for each focus group, and five to eight showed in each group. Topics of the discussions focused on identifying the benefits parents and non-parents associate with NHES participation, the barriers they perceive to participating, which NHES advance materials they believe are most and least useful, and their suggestions for factors that may increase response rates for NHES. The focus group sessions lasted approximately 90 minutes, and all sessions were audio and video recorded. Hager Sharp will destroy the recordings upon final approval of this report. This report includes no personally identifiable information regarding participants. To facilitate recruitment for the focus groups and thank participants for their time, we gave each participant a $75 incentive for attending.
During the focus groups, using a moderator’s guide (Appendix A), a moderator led the participating parents through a discussion. Key research questions for this project are outlined as follows:
All participants:
How do members of these households interact with mail? What mail gets opened in the household? When? By whom? What mail gets thrown out? Why?
How do members of these households decide to participate in a survey? The survey’s length? The incentive? The importance?
Would members of these households prefer to complete a survey by web or by paper or by calling into an operator? Why? What are the positive and negative aspects of each?
Are there specific endorsements, colors, or photos for the front of the survey that would influence these household members toward participating?
Parent participants:
What is important to parents of young children about care arrangements? What messages would be most salient to parents of young children about the unique data from the Early Childhood Program Participation Survey?
What is important to parents of school-aged children about their relationship with their child’s school? What message would be most salient to parents of school-aged children about the unique data from the Parent and Family Involvement in Education Survey?
Non-parent participants:
What language and/or visuals will help non-parents understand that it is important for them to complete and return the survey even though they do not have children in school?
If they see this is an education survey, what will make them read further and complete the survey?
The moderator showed participants several documents that are proposed for use in NHES recruitment, and participants discussed each. The materials included:
Advance letter for recruitment (in Census envelope) – Spanish
Invitation letter to take the screener survey online - Spanish
Sample screener survey – English
Sample screener survey – Spanish
Bank of visual elements (photographs)
Short statements about NHES – Spanish
Second (reminder) letter to take the screener survey – Spanish
Third letter (reminder) to take the screener survey – Spanish
To prepare this report of findings, Hager Sharp worked from transcripts of all groups to analyze trends and key themes. We have included verbatim comments from participants to illustrate and provide examples of themes and trends. We have also noted key areas of consensus and areas where participants differ in their views.
Summary of Highlights and Conclusions
Highlights and conclusions from the focus groups are organized by the key research questions NCES identified when planning the research:
All participants:
How do members of these households interact with mail? What mail gets opened in the household? When? By whom? What mail gets thrown out? Why?
When asked who usually opens their household’s mail, non-parents cited a variety of people within the household, including father, niece, sister, and “whoever gets home first.” They tend to open bills and throw away ads and marketing materials:
When asked who usually opens their household’s mail, parents cited usually a husband or wife, but sometimes a child. As with non-parents, parents tend to open bills and throw away ads and marketing materials.
Key Takeaways: It is essential that the exterior envelope for the survey look official, important, and relevant. Participants generally respond well to the US Census logo on the survey. Some find the Department of Commerce address confusing. If possible, it may help to make it clear that the US Census is part of the Department of Commerce, perhaps by including US Census in the address. The official nature of the envelope is important as well because participants were vocal about security concerns. Many had been victims of scams, fraud, or attempted fraud.
How do members of these households decide to participate in a survey? The survey’s length? The incentive? The importance?
Non-parents struggle to find relevance for themselves in the survey. Because they do not have children, they are not especially motivated by the topic of education. However, they respond favorably to the US Census, and they are somewhat motivated to help their communities.
The length of the survey is important to them. If they understand immediately that the survey will be simple and brief for them because they do not have children, they will be more likely to complete it. Many are also appreciative of the $5 incentive.
Parents are strongly motivated by contributing to a better education for their children. They respond favorably to messages that state how the survey results may be used to improve education in communities. They also responded favorably to elements of the third letter, including interesting statistics about education and the appeal to preserve the accuracy of the survey data.
Many parents were confused by the nature of the screening survey. They felt a disconnect after seeing the screener survey questions, as many expected to be asked questions about their views on education.
Key Takeaways: Materials related to the survey should more explicitly state that non-parents need to complete and return the survey, and that it is easy to so. Materials should also include stronger messages about the purpose and benefits of the survey. The wide time range of 3-30 minutes referenced in the advance letter should be explained – e.g., “The survey should take 3 minutes for households without children, and 5-20 minutes for households with children.” No participants thought the survey would require 30 minutes to complete.
Many parent and non-parent participants were confused by the translation of the survey title, National Household Education Surveys. Some interpreted the survey to be about “domestic education.” The translation needs to be refined and it should be used consistently in all materials.
Would members of these households prefer to complete a survey by web or by paper or by calling into an operator? Why? What are the positive and negative aspects of each?
Roughly half of the participants said they would prefer to take the survey in paper format because:
“Paper would be less distracting than internet.”
“I prefer paper – you have the envelope with the stamp, so I pay nothing.”
“I prefer paper, and I want it to look official.”
Half said they would prefer the online option because:
“It is faster and you don’t lose the envelope.”
“Online is easier.”
“Online is safest.”
Several participants said their preference in format depends on the length of the survey and how much time is required to complete it.
Key Takeaways: Some participants have definite preferences for either paper or online, and most appreciate the option to do either.
Are there specific endorsements, colors, or photos for the front of the survey that would influence these household members toward participating?
Many participants expressed confusion about the separate government agencies and offices involved in administering the survey. They specifically wondered why the Department of Commerce would be involved in an education survey.
All participants agreed that it is essential that the exterior envelope bears an official government seal and address. Most responded favorably to the Census Bureau. Several said they would take the Department of Commerce address seriously, but again, some expressed confusion about the role of the Department of Commerce. Most participants did not understand that the US Census is within the Department of Commerce. A few participants expressed concerns about the federal government targeting the Hispanic community regarding immigration issues. The US Census and the Department of Commerce were more alarming in this regard than the department of Education.
Key Takeaways: The US Census is probably the single government entity that participants will understand and view favorably. Including too many different logos of government offices and agencies will likely confuse some participants.
Parents generally responded favorably to the Department of Education, although many strongly desired the involvement of their local schools.
Parent participants:
What is important to parents of young children about care arrangements? What messages would be most salient to parents of young children about the unique data from the Early Childhood Program Participation Survey?
Parents suggested several factors that are important to them, including safety, patience, “for the kids to be supported – so they can study better,” and to have their children “treated well…not yelled at.”
Key Takeaway: Parents would be receptive to data and messages related to these factors.
What is important to parents of school-aged children about their relationship with their child’s school? What message would be most salient to parents of school-aged children about the unique data from the Parent and Family Involvement in Education Survey?
Parents suggested a variety of factors that are important to them, including safety, an environment that is conducive to learning, and the quality of teachers and principals.
Key Takeaway: Parents would be receptive to data and messages related to these factors.
Non-parent participants:
What language and/or visuals will help non-parents understand that it is important for them to complete and return the survey even though they do not have children in school?
Non-parents appreciated the visual images of families, particularly images of parents helping their children with homework. However, they did not necessarily see relevance in these images for themselves.
Key Takeaways: The images are less important for non-parent participants. They should communicate various facets of education to reinforce the concept that the survey is about education. However, more importantly for non-parents, the materials need to explicitly state that non-parents need to complete and return the survey and that it is simple to do so.
If they see this is an education survey, what will make them read further and complete the survey?
Many participants did not understand until the third letter that non-parents need to complete and return the survey. Many struggled to understand the relevance of the survey for themselves as they read the advance, invitation and second letters.
Key Takeaway: There should be explicit language in the first communication that non-parents need to complete and return the survey, and that it should take about three minutes for them to do so.
Recommendations
Key recommendations for revisions to the package are outlined as follows:
Package Component |
Recommendation |
Exterior Envelope |
|
Advance Letter |
|
Invitation Letter |
|
Screener Survey |
|
Visual Elements |
|
Short Statements about NHES |
|
Second (Reminder) Letter |
|
Third (Reminder) Letter |
|
NHES 2019 Spanish Materials Focus Groups Round 2 - Recruitment (OMB# 1850-0803 v.229)
This submission gained approval for three focus groups designed to test a set of recruitment materials tailored specifically for Spanish speakers. This package included the justification and focus group recruitment materials only. Results are provided below under the entry for OMB #1850-0803 v.232.
NHES 2019 Spanish Materials Focus Groups Round 2 (OMB# 1850-0803 v.232)
Focus Groups with Spanish Speakers for Development of Tailored Recruitment Materials
Summary of Focus Group Results
Conducted under OMB #1850-0803 v. 229 and v. 232
National Center for Education Statistics
June 2018
Prepared by
American Institutes for Research and Hager Sharp
Background
The National Household Education Surveys Program (NHES) is conducted by the National Center for Education Statistics (NCES) and provides descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Surveys are administered in English and in Spanish. Modeling of response propensity in prior NHES administrations suggested that addresses from which NHES is unlikely to get a response share similar demographic characteristics, hence, response bias is an issue that may affect data quality. The response propensity modelling showed that household members of low responding addresses are more likely to fall into one or more of the following groups:
Be age 35 or younger
Rent rather than own
Have less than a high school education
Have an income below $15,000
Identify as Hispanic
Anyone matching any one of these characteristics is less likely to respond. The response propensity model is a regression model that showed that, holding all other variables constant, each of the above characteristics is significantly related to responding at lower rates to previous NHES administrations. Therefore, NCES determined the need to focus research on predominantly Spanish-speaking parents and non-parent adults.
NCES hopes to improve NHES response rates and reduce response bias for the 2019 NHES by better understanding factors that motivate Spanish-speaking parents and non-parent adults to complete (or decline to complete) the NHES questionnaire. NCES has focused specifically on understanding motivations to complete the survey among occupants in sampled addresses that are modeled to be unlikely to respond to the survey. The findings highlighted in this report were used to refine the materials and strategies used for NHES recruitment.
Methodology
Hager Sharp and the American Institutes for Research (AIR) partnered to conduct a total of seven in-person focus groups with Spanish-speaking parents and non-parent adults to understand their perceptions of the NHES study and how they would respond if selected for NHES, and their reactions to different key messages, images, and written materials. Focus groups were conducted in Rockville, MD and Denver, CO with respondents who met the following eligibility criteria:
Spanish-speaking parents:
Parent of at least one child under age 18 in household
Indicated they speak “mostly Spanish” at home
Spanish-speaking non-parents:
Not a parent of a child under age 18 in household
Indicated they speak “mostly Spanish” at home
In addition to the screening criteria, we recruited for diversity in household income level (ensuring at least two participants per group have a household income of $40,000 or less), education level, participant age, and gender. The configuration of the seven focus groups is as follows in Exhibit 1.
Exhibit 1. Location, parent/non-parent configuration, and date of focus groups: April – June 2018
Location |
Parent or Non-Parent |
Date |
Rockville, MD |
Non-Parents |
April 5, 2018 |
Rockville, MD |
Parents |
April 5, 2018 |
Denver, CO |
Parents |
April 10, 2018 |
Denver, CO |
Non-Parents |
April 10, 2018 |
Rockville, MD |
Parents |
May 31, 2018 |
Denver, CO |
Parents |
June 4, 2018 |
Denver, CO |
Non-Parents |
June 4, 2018 |
In the first round of focus groups in April, the research focused on current materials and key messages. The moderator showed participants several documents that are proposed for use in NHES recruitment, and participants discussed each. The materials included:
Advance letter for recruitment (in Census envelope) – Spanish
Invitation letter to take the screener survey online - Spanish
Sample screener survey – English
Sample screener survey – Spanish
Bank of visual elements (photographs)
Short statements about NHES – Spanish
Second (reminder) letter to take the screener survey – Spanish
Third letter (reminder) to take the screener survey – Spanish
Based on the results of the first round of focus groups, we developed new materials to test in a second round of focus groups. The key features of the materials that were tested are shown in Exhibit 2.
Exhibit 2. Features and materials tested in the round 2 targeted materials focus groups: May-June 2018
|
Materials tested |
|||
Feature tested |
Advance Letter |
Invitation Letter |
Envelope |
Screener |
Images depicting Hispanic families |
|
|
|
|
Key messages (voluntary, about your community, trusted) |
|
|
|
|
Icons with key messages |
|
|
|
|
Bilingual text with Spanish first |
|
|
|
|
Bilingual addressee line |
|
|
|
|
Text communicating participants opinion matters |
|
|
|
|
Endorsements |
|
|
|
|
Findings
Images depicting Hispanic families
Participants recognized themselves, their families, and their communities in the images and in one group, there was discussion about whether or not the survey was a survey specific to Latinos. This indicates that the images were effective at reflecting the target population. The images that gained the most positive feedback from participants were images of:
A child in a wheelchair seated at school table
Graduation
Grandparents and children
Fathers and children
One image did not gain positive feedback; this was the image of the bus in front of an urban school.
In the letters and envelopes, participants preferred the versions that used the icons over the versions that used the pictures. Some participants commented that pictures on an envelope made them think of marketing materials or junk mail. The single image used in one version of the invitation letter was interpreted too literally without the context of the survey cover or other images along with it. One respondent asked if it was the grandparent showing the child how to fill out the Census.
Icons and Key messages
All respondents reacted positively to the icons and key messages (voluntary, about your community, trusted) on the envelope and in the invitation letter. They also felt these provided more information than pictures or a blank envelope.
Bilingual text/addressee line
All respondents appreciated the bilingual text and addressee line and did not indicate any problems with translation. However, one of the reasons they prefer bilingual materials is because sometimes the Spanish translation is poor and they rely on the English to help establish meaning. This rationale for the bilingual text has been a recurring comment in the NHES qualitative research with Spanish speakers.
Text communicating participant’s opinion matters
The text “Make your experience count. Take the National Household Education Survey now.” was included on the invitation letter and participants reacted positively toward it. It stood out because it was bold and the message resonated with them, as also evidenced from the round 1 focus groups.
Endorsements
The endorsements were Hispanic-affiliated organizations plus the National Women, Infants and Children's program (WIC). All participants liked the endorsements and felt they added legitimacy to the survey and made them feel more positive about filling it out. In addition, some participants commented that the endorsements indicated what kind of organizations may use the information they provided.
The local Hispanic Chamber of Commerce was the most salient organization. In Rockville, participants also recognized The League of United Latin American Citizens (LULAC) and in Denver, WIC.
Recommendations
Based on the second round of focus group findings, the recommended sequential contact materials for use in the NHES:2019 targeted mailing are described below and in Exhibit 3:
Envelope – from the 1st mailing onwards, use the envelope with the icons and key messages.
Advance mailings – use general population mailing materials.
1st mailing (invitation letter) – use letter with icons, key messages, opinion matters text, and endorsements. Use icon envelope. Note: NCES intends to seek endorsements from LULAC, WIC, and the US Hispanic Chamber of Commerce
2nd mailing – use general population mailing materials and icon envelope.
3rd mailing (contains paper screener) – use letter with icons, key messages, opinion matters text, and screener with images of graduation, child in a wheelchair, a grandparent, and a father. Use icon envelope.
4th mailing – use general population mailing materials.
Exhibit 3. Features and materials recommended for targeting in the NHES:2019
|
Materials recommended for targeting |
||||
Feature tested |
Advance Letter |
Invitation Letter |
Envelope |
Screener |
Third mailing letter |
Images depicting Hispanic families |
|
|
|
|
|
Key messages (voluntary, about your community, trusted) |
|
|
|
|
|
Icons with key messages |
|
|
|
|
|
Bilingual text with Spanish first |
|
|
|
|
|
Bilingual addressee line |
|
|
|
|
|
Text communicating participants opinion matters |
|
|
|
|
|
Endorsements |
|
|
|
|
|
NHES 2019 Screener Cognitive Interviews - Recruitment (OMB# 1850-0803 v.230)
This submission gained approval to test a new, experimental version of the screener instrument which is designed to allow respondents without children to mark that there are no children in the household and return the survey without having to open it. This package included a justification and recruitment materials for the cognitive interviews. Results are provided below under the entry for OMB #1850-0803 v.231
NHES 2019 Screener Cognitive Interviews (OMB# 1850-0803 v.231)
National Household Education Surveys Program (NHES)
Screener Opt-out Cognitive Interviews
Summary of Cognitive Interview Results
Conducted under OMB #1850-0803 v. 231
National Center for Education Statistics
June 2018
Prepared by
American Institutes for Research
Background
In NHES:2016 and in the NHES 2017 web-pilot, the screener enumerated the entire household including adults and children because the Adult Training and Education Survey (ATES) was among the second-stage surveys fielded. NHES:2019 will not field ATES. NHES:2019 will include only the Early Childhood Program Participation (ECPP) survey and the Parent and Family Involvement in Education (PFI) survey. Therefore, the NHES:2019 screener was returned back to the NHES:2012 screener to only ask to enumerate the children in the household. A concern for NHES:2019 is that only enumerating children in the screener could potentially result in nonresponse from households without children that elect not to complete the screener because they may feel it is not applicable. Response from households without children is important to the study because it provides data necessary to calculate accurate prevalence rates and because it indicates which households are or are not eligible for a topical survey.
For NHES:2019, NCES plans to experiment with the use of an “opt-out” paper screener and cover letter which allows households without children (about 60 percent of all sampled households) to indicate on the front cover of the screener that there are no children in the household without having to open the survey to complete their response. Our goal is to increase the response rate to the paper screener by making it easier for households without children to quickly realize the low burden required to complete the instrument and thus for these households to respond.
In preparation for the NHES:2019 administration, cognitive interviews were conducted in English and Spanish to test this adapted version of the screener. These interviews included testing three different formats of the screener questionnaire: 1) a longer version that includes all the instructions and wording that are part of the original screener on the cover, 2) a shorter version that highlights the question on the cover as to whether or not the household has children, and 3) a screener that emphasizes that if the household has no children they only need to answer this one question and return the survey.
Methodology
American Institutes for Research (AIR) conducted a total of 25 cognitive interviews with parents and non-parent adults. The following table outlines the characteristics of the respondents to the cognitive interviews.
|
Summary of Screener Opt-out Cognitive Interviews |
||||||
|
Total |
Language |
Gender |
Age |
Education |
Children |
Hispanic |
Completed |
25 |
English: 20
|
Male: 7 Female: 13 |
18-24: 2 25-34: 5 35-44: 4 45-64: 6 65+: 3 |
<HS: 0 HS: 4 Some college: 9 College: 6 Graduate: 1 |
Children: 6 No children: 14 |
Yes: 0 No: 20 |
Spanish: 5 |
Male: 1 Female: 4 |
18-24: 0 25-34: 1 35-44: 2 45-64: 1 65+: 1 |
<HS: 0 HS: 0 Some college: 3 College: 1 Graduate: 1 |
Children: 3 No children: 2 |
Yes: 5 No: 0 |
||
The cognitive interviews took place mid-May through June. The interviews took about 30 minutes to test the following materials to ensure that including a question on the survey was appealing and easy to navigate:
Cover letter to the survey package that included text about if the household did not have any children it would be one question.
Three different opt-out survey designs
Long – this included all the instructions that were before the first question on the original screener and question 1 on the cover.
Short – only the question 1 on the cover (no instructions).
Attention grabbing – same as the Short version but included a “call-out” box before the question highlighting if the household does not have children they just have to answer this one question on the cover.
These cognitive interviews used retrospective probing to gain the following information about the materials being tested:
verify respondents’ interpretation of the screener questions and instructions;
check respondents’ understanding of the meaning of specific terms or phrases used in the questions and instructions; and
explore respondents’ reaction to different formats and constructions of the opt-out screener.
Findings
The following table provides a summary of findings from testing the different covers of the surveys. This includes the number of respondents that most preferred the type of cover by language the survey was in. It then provides a list of comments from respondents on each type of cover.
|
Summary of Findings - Covers |
|||
|
Cover A (Long) |
Cover B (Short) |
Cover C (Attention grabbing) |
|
Preferred Cover |
English |
Number of respondents that liked A the most: 5 |
Number of respondents that liked B the most: 12 |
Number of respondents that liked C the most: 3 |
Spanish |
Number of respondents that liked A the most: 1 |
Number of respondents that liked B the most: 4 |
Number of respondents that liked C the most: 0 |
|
Respondent Comments |
|
Pros (10): -More organized and official looking (likes that DOE was mentioned). -More serious, greater impetus to respond -More informative -Clear & easy to understand. -Liked Diversity -Liked “start here instructions” -Nice that it tells you what you’re signing up for
Cons (15): -Not bad, but didn’t feel instructions were necessary. -Not clear who, in relation to the child, should complete the survey (parent, grandparent, and caregiver). -More complicated than cover B. (too busy) -Took more time to process (but still easy to follow). -Square box is “harsh” |
Pros (20): -Straightforward, simple, clear, & clean. -Don’t need the instructions from other versions. -Bigger pictures. -Easy to read. -Eye-catching -Likes that it specifically says to include small children, babies, and those in college housing. -Pictures are welcoming. -Liked Diversity. -Good flow and softness to the rounded ‘box’ -Really easy and clear if you don’t have kids
Cons (5): -Pictures of children makes it seem that it wouldn’t be relevant for households without children. -Too simple. -Pictures of children are all older, not sure if it is relevant for younger children (e.g., 5) -Not enough information -Looks like a comic book for kids |
Pros (7): -Clear and organized. -Call out box makes it very clear. -Especially clear for people without kids. -Liked Diversity -Tells you, you are done if you answer “no”
Cons (18): -Call out box is a little awkward and unnecessary. -Didn’t like call out box, thinks it has a rude introduction. -Thought it was asking about the household of the children pictured. -Too cluttered (due to the call out box) -Instructions about foster children and babies were confusing. Not sure what to do when child goes to college and doesn’t live at home. -Not enough information -Looks like a comic book for kids |
Summary of findings - Mailing back screener
Two respondents said that they probably wouldn’t mail the screener back, three respondents didn’t say whether they would mail it back or not, and one respondent said they could probably only mail it back if they got Cover A. The rest of the respondents said they would mail the screener back.
Summary of findings – Letter.
Most respondents didn’t have any problems with the letter, though four respondents didn’t seem to really understand what the information would be used for. Three respondents just said that it wasn’t clear what the information would be used for and thought it was something in the general education realm, while the other respondent assumed it was about counting children (because of the connection to the Census) and that it was geared toward kids that don’t have papers and might not be going to school. One respondent mentioned that they thought the letter was a little boring and would like to have something that really catches your eye. One respondent mentioned that they didn’t like the term “harm” in the letter. They said it felt very threatening and thought “compromise” or “impair” might be a better fit. This respondent stressed this concern throughout the entire interview. In contrast, several respondents commented that they noticed and liked that the letter told them it was important for them to respond and that their failure to do so would be detrimental to the survey – it made them feel like their participation was important and urgent.
Recommendations
Based on the twenty-five cognitive interview findings, the recommended changes to the screener are described below:
Letter – no changes but ensuring that translation of Spanish words is consistent throughout letters.
Screener – recommend using the “Short” version of the survey with some modifications:
Remove the call-out shape and turn it into a rounded box
Participants given the form with a rounded box preferred this over the call-out.
Emphasize the word “Finished” by capitalizing all letters and underlining (make same change to Spanish survey)
Ensure the cover images are the same used in the regular screener.
1 CSM is currently planning a research study to compare user performance on iOS and Android dropdown boxes on mobile devices. Once the research is finished we hope to have some evidence based guidelines on how to design dropdowns for mobile.
2All mentions of item numbers refer to the combined PFI questionnaire contained in the OMB package associated with ESSIN 33 Option Year 1 Task 15.
3 NHES questionnaires are available at http://nces.ed.gov/nhes/questionnaires.asp.
4 For additional information on the specific recommendations of this panel, contact Sarah Grady: [email protected].
5 Barnett, W. S., Friedman-Krauss, A. H., Weisenfeld, G. G., Horowitz, M., Kasmin, R., & Squires, J. H. (2017). The State of Preschool 2016: State Preschool Yearbook. New Brunswick, NJ: National Institute for Early Education Research.
6 Revised items with no issues raised are not discussed in this section. A full list of items, including those that worked as intended and required no changes can be found in Appendix 1.
7 For more information, including recommendations, see Using Behavioral Science to Improve Survey Response: An Experiment with the National Beneficiary Survey (In Focus Brief; Nov. 2017) and Respondent Confidentiality Concerns and Possible Effects on Response Rates and Data Quality for the 2020 Census (Nov. 2017).
8 ECPP version 11/12/2017
9 Change/no change/ examine in 2019 or 2022
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | NHES ECPP 2017 Cognitive Interviews Report |
| Subject | Prepared for the National Center for Education Statistics (NCES) |
| Author | Stephen Wenck |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |
© 2024 OMB.report | Privacy Policy