National Household Education Survey 2019 (NHES:2019)
National Household Education Survey 2019 (NHES:2019)
Appendix 4 Study of NHES 2019 Nonresponding Households
National Household Education Survey 2019 (NHES:2019)
OMB: 1850-0768
National Household Education Survey 2019 (NHES:2019)
Full-scale Data Collection
OMB# 1850-0768 v.16
Appendix 4 – In-Person Study of NHES:2019 Nonresponding Households
May 2018
revised October 2018
Attachment B.2. will provide the Spanish versions of the materials that will be used to interact with households sampled for the qualitative interview. These materials will be provided to OMB as a change request in November 2018.
Contents
Description and Justification 1
1.2 The utility of conducting follow-up studies 1
1.3 The challenges associated with using address-based sampling (ABS) frames 2
1.4 The benefits of collecting interviewer observations 2
2.1 Qualitative interview methodology 4
2.2. Address/neighborhood observation 7
3. Looking ahead: making use of the study findings 8
B.1.2. First Reminder Postcard 17
B.1.3. Second Reminder Postcard 18
B.1.4. Phone Follow-up Recruitment Script 19
B.1.5. “Sorry We Missed You!” Card 21
B.1.6. In-Person Recruitment Script 22
B.1.7. Commonly Asked Questions 25
B.1.8. Informed Consent Form 26
B.1.11. Incentive Receipt Acknowledgment Form 35
B.2.1. Spanish Invitation Letter 37
B.2.2. Spanish First Reminder Postcard 38
B.2.3. Spanish Second Reminder Postcard 39
B.2.4. Spanish Phone Follow-up Recruitment Script 40
B.2.5. Spanish “Sorry We Missed You!” Card 41
B.2.6. Spanish In-Person Recruitment Script 42
B.2.7. Spanish Commonly Asked Questions 43
B.2.8. Spanish Informed Consent Form 44
B.2.9. Spanish Interview Protocol 45
B.2.10. Spanish Income Show Card 46
B.2.11. Spanish Incentive Receipt Acknowledgment Form 47
B.3. Observation Instrument 48
Description and Justification
Given continued declines in response rates to both the National Household Education Surveys (NHES) and to household surveys more broadly – and the growing challenges associated with conducting cost-efficient, high-quality, representative data collections, NCES will conduct the In-Person Study of NHES:2019 Nonresponding Households during the NHES:2019 administration. This study will primarily address the following research questions:
What are the barriers to response among screener nonrespondents to NHES? For example, do screener nonrespondents remember receiving the survey mailings, and, if so, do they open them? What stops some households from receiving or opening the mailings?
What are the characteristics of nonresponding households? Is there information about nonresponding households that could be used to tailor contact protocols in future collections?
Is the information available on the address-based sampling frame for nonresponding households accurate to ensure variables used for tailoring materials or predicting data collection protocols are reliable?
Are there variables related to household survey nonresponse that have not been previously observed or measured?
To address these research questions we will conduct approximately: (a) 100 of 90-minute interviews using an unstructured, qualitative interview protocol and (b) 750 address and neighborhood observations. This document discusses the importance of conducting this study and provides an overview of its methods.
1. Background
Survey nonresponse has been increasing for several decades (Brick and Williams 2013; Czajka and Beyler 2016). NHES has not escaped this trend; between the first mail-based NHES administration in 2012 and the most recent one in 2016, the screener response rate has dropped from 74 percent to 66 percent (McPhee et al. 2015; McPhee et al. 2018). In addition, NHES nonrespondents tend to be younger, less educated, and have lower incomes than respondents; they are also less likely to be White or to be married than are respondents.
Given the global nature of this phenomenon, researchers have devoted considerable attention to understanding the factors that contribute to survey nonresponse. Proposed individual-level explanations for nonresponse include privacy concerns, anti-government sentiment, busyness and fatigue, concerns about response burden, lack of interest in the topic, and low levels of civic engagement or community integration (Abraham et al. 2006; Amaya and Harring 2017; ASA Task Force on Improving the Climate for Surveys 2017; Groves et al. 1992; Groves et al. 2004; Singer 2016). Investigators also have suggested that societal-level changes, such as the increasing number of survey and solicitation requests, declining confidence in public institutions, and growing concerns about security and identity theft are contributing to declining response rates (ASA Task Force on Improving the Climate for Surveys 2017; Czajka and Beyler 2016; Presser and McCulloch 2011).
However, much remains unknown about sample members’ reasons for nonresponse and the best way to counteract those concerns. Recent task force reports focused on survey nonresponse argue there is a need for more research about the current survey climate and the various factors that could improve survey participation (AAPOR Task Force on Survey Refusals 2014; ASA Task Force on Improving the Climate for Surveys 2017). Due to the emphasis on telephone surveys in the latter decades of the 1900s, little research has been carried out since the 1970s about the response decision process associated with mail surveys like NHES (Singer 2016). Questions remain about whether research on nonresponse within mail collections from previous decades holds up in today’s survey environment.
Studies that have followed up with survey sample members after the survey request has been made have proven to be a useful tool for understanding the reasons for response (or lack thereof). In-depth follow-up interviews with respondents to an earlier survey found that materials that clearly communicated the topic and benefits of the study were particularly important for gaining response (Harcomb et al. 2011). In another study, semi-structured follow-up interviews with sample members identified altruism, perceived personal benefit, understanding the study, and being committed to its success as primary reasons for responding (Nakash et al. 2008). A focus group of immigrant and ethnic minority nonrespondents to a Danish national health survey discovered that linguistic and educational limitations, alienation resulting from sensitive questions and cultural assumptions, and concerns about anonymity were the primary cause of nonresponse among that group (Ahlmark et al. 2015).
The Bureau of Labor Statistics’ Response Analysis Survey used qualitative telephone interviews to examine nonresponse (refusals) to the American Time Use Survey (ATUS) (O’Neill and Sincavage 2004). The study found that the advance materials were ineffective at communicating the differences between the Current Population Survey, which served as the sampling frame for the ATUS, and the ATUS. It concluded that how respondents perceive the survey sponsor is an important factor in whether to participate and that the use of incentives should be explored to encourage participation.
The American Community Survey (ACS) conducts an in-person computer-assisted personal interview (CAPI) follow-up for a subsample of nonrespondents. Special emphasis is placed on collecting data from many languages (e.g., providing language assistance guides to the interviewers in Chinese and Korean, special training). Language availability and the mandatory nature of the survey keep the response rates high (United States Census Bureau 2014). CAPI interviewers achieve about a 95.5% interview rate. Cases that showed initial resistance or had at least one coded refusal had interview rates of 94.5% and 26.8% respectively (Zelnack et al. 2013). The ACS also uses the Contact History Instrument (CHI) to monitor field performance and collect paradata during in-person interviews. The purpose of the interview is to collect data from nonrespondents.
Despite the research cited here, there are relatively few studies that attempt to explore the reasons for nonresponse in American household surveys at the scale or depth planned for the In-Person Study of NHES:2019 Nonresponding Households, making this an important and unique contribution to the survey literature and a potential resource for researchers facing the same challenges in other national, federal government household surveys.
The NHES sample is comprised of addresses drawn from an ABS frame. The quality of the information about these addresses on the frame is another factor that could contribute to nonresponse. For example, vacancy or seasonal status indicators available on the frame could be inaccurate and lead to addresses being sampled where there is no one available to respond (AAPOR Task Force on Address-Based Sampling 2016). In addition, if the information available about the individuals living at sampled addresses (such as whether they prefer to speak Spanish) is inaccurate, this creates challenges for designing survey protocols that are targeted at sampled addresses’ response preferences (such as whether to send Spanish-language materials). The NHES frame is not immune to this problem; analysis of NHES:2016 data shows that only about 55% of the screener respondent households flagged on the commercial vendor frame as having children ended up reporting a child on the screener. Assessing the quality of additional sampling frame variables through interview questions and observations will help to assess the extent to which inaccurate frame data may be hindering efforts to gain response to NHES and making recruitment efforts inefficient.
Interviewer observations are a low-burden tool for learning about the context in which sample members receive the survey request and for verifying ABS frame data. Little and Vartivarian (2005) suggest that interviewer observations of variables related to the likelihood to participate in a survey and to the survey’s key estimates can be used to adjust for nonresponse bias. Several large-scale, in-person surveys collect observations using tools such as the Contact History Instrument or Neighborhood Observation Instrument (NOI). Since 2013, National Health Interview Survey (NHIS) interviewers have used the NOI to observe neighborhood characteristics, such as the condition of units or the presence of barriers to access, and to make inferences about sample member characteristics of interest, such as income, employment, and the presence of children (United States Census Bureau 2017). NHIS researchers have found that some, but not all, NOI measures are positively associated with refusal to participate (Walsh et al. 2013). The National Survey of Family Growth (NSFG) also collects interviewer observations about the neighborhood, housing unit, and sampled persons (User’s Guide for Paradata File n.d.). NSFG researchers again find that some, but not all, of the recorded observations are associated with lower propensity to respond to the in-person interview request (Lepkowski et al. 2013). These findings highlight the potential benefit of collecting observations, as well as the need for further research into which neighborhood and address characteristics are related to responsiveness to a mailed survey instead of an in-person survey.
2. The proposed study
The proposed In-Person Study of NHES:2019 Nonresponding Households will use qualitative in-person interviews, as well as address and neighborhood observations to assess whether the existing theories surrounding household survey nonresponse are applicable to NHES. The study is designed to provide additional actionable information about how to combat the growing nonresponse problem in NHES. It is expected that the results of this study will be used to improve the design of NHES, with the goal of increasing the response rate and the representativeness of the respondents. Potential changes could include modifications to the nonresponse follow-up protocol, the type of incentives offered, and the presentation of contact materials.
Screener nonrespondents, rather than topical nonrespondents, are the focus of the study because of the impact that screener nonresponse has on both the screener response rate and each NHES survey’s overall response rate and because NHES screener response rates are lower than topical response rates.1 Larger numbers of screener nonrespondents are anticipated to be clustered in metropolitan areas than in more rural locales. Exhibit 1 plots the location of NHES:2016 screener nonrespondents. As expected, screener nonrespondents are widely dispersed, but there is some clustering around large cities.
Exhibit 1. Location of NHES:2016 Screener Nonrespondents
Moreover, in NHES:2016, household in tracts with 40% of more Hispanic persons, high rise apartments, and renters had lower weighted screener response rates, 55.9%, 58.9%, and 56.6% respectively (see Attachment A in this document for screener response rates by additional subgroups) (McPhee et al. 2018).
Analyses of the resulting data are expected to illuminate some of the factors associated with nonresponse and inform revisions to the content and design of recruitment materials to be used in NHES:2022.
Qualitative research is particularly useful for trying to understand how people make sense of their lives and experiences. To gain a deeper understanding of why some sample members do not respond to the NHES screener, approximately 100 unstructured, qualitative interviews will be conducted with screener nonrespondents. These interviews are designed to provide rich and nuanced information about survey nonresponse.
The NHES:2019 screener nonrespondents will be stratified to focus on a handful of key household characteristics available on the sampling frame that are likely to drive differences in reasons for nonresponse and could be used for targeting NHES materials and contact protocols in future administrations, such as (1) whether the household is Hispanic or is expected to prefer to respond in Spanish, and (2) in what part of the United States the address is located. Cases will be sampled from three or more sites and the cases sampled for each site will be clustered within an area with approximately a 30-mile radius.2 This design will allow for sampling of some rural addresses, while still maintaining a manageable driving distance for the interviewers. Approximately 750 addresses will be sampled from among NHES:2019 early screener nonrespondents for address and neighborhood observations. Of these, 500 will be selected for and invited to take part in the qualitative interview study (to yield approximately 100 completed 90-minute interviews). If nonresponse clustering patterns result in lower than estimated cost of observing and interviewing the target numbers of addresses, more addresses will be selected for the address/neighborhood observations.
The cases included in the nonresponse follow-up study will be selected using frame variables that will be identified based on the observed patterns of nonresponse to early NHES:2019 screener mailings. Because NHES:2019 will be the first NHES administration that uses a web-push mixed-mode design (whereas recent NHES administrations used a paper-only design), we are not certain what patterns of screener nonresponse will be observed and thus must wait to determine the actual selection criteria for the nonresponse follow-up study until NHES:2019 screener data collection has begun and patterns of the early NHES:2019 screener nonresponse become known. However, patterns of screener nonresponse to NHES:2016 give some indication of address characteristics that are likely to drive screener nonresponse in 2019, such as: being in an area with a higher percentage of Black or Hispanic residents, being in an area with a higher poverty rate, being in the south, being in a rented (vs. owned) dwelling, being in a multiunit (vs. a single unit) dwelling, not having a phone number available on the frame, or lacking demographic information about the head of the household (such as gender, age, or marital status).
Once we determine the subgroups of address characteristics that show the highest likelihood of nonresponse to early NHES:2019 screener mailings, we will identify three or more geographic clusters, of approximately 30-mile-radius each, that will include large enough numbers of nonrespondents from the address characteristics subgroups to meet the recruiting targets for the nonresponse follow-up study. Due to differences in mail delivery challenges in urban versus rural areas (which are key for understanding potential reasons for nonresponse to a mailed survey such as NHES), urbanicity will be one of the factors used in the selection of sites. In addition, we will ensure that each of the at least three clusters will include a diverse group of addresses by characteristics such as region, address characteristics, demographics, etc. Initially, in each selected cluster, address/neighborhood observations will be conducted with the nonresponding households selected from that cluster. From the observed addresses, across all clusters, a subset of 500 addresses will be selected for recruitment for the target 100 in-person qualitative interviews. At this time, the cost of observations and interviews will be determined and, budget permitting, additional NHES:2019 screener nonrespondent addresses may be selected for additional address/neighborhood observations.
2.1.1 Recruitment
The following recruitment protocol will be used after the NHES:2019 screener data collection closes in late May 2019:
Sampled cases will be sent an invitation letter that includes $5 cash, invites them to participate in the in-person study, informs them of the contingent incentive, and provides them with NHES staff contact information for letting the researchers know that they would like to participate. All sampled cases will be offered an additional $120 cash incentive for completing the interview and this incentive will be mentioned in all recruitment materials. This incentive is designed to demonstrate to participants that their time and participation is valued and takes into account that the sample is comprised of households that have already shown reluctance to participate in the survey (see below for additional discussion of the incentive amount).
Up to two reminder post cards will be sent to each sampled household that has not yet responded to the nonresponse follow-up study invitation. The first will be sent in the early part of the recruitment period, approximately one week after the invitation letter. A second will be sent to the remaining follow-up nonrespondents about halfway through the recruitment period. These postcards will invite sampled households to participate in the study, remind them about the contingent incentive, provide them with NHES staff contact information for letting the researchers know that they would like to participate, and inform them that an interviewer will be visiting their home to try to schedule an interview.
Recruitment phone calls will occur in two windows for sampled households that have phone numbers available on the sampling frame and have not yet responded to the nonresponse follow-up study contacts. The first will be early in the recruitment period (shortly after the first reminder postcard) and the other will be about halfway through the recruitment period. During each window, each household will be called up to two times. One voicemail will be left per calling window if the household does not answer the phone.
An in-person recruitment period will also take place, where all households that have neither agreed nor declined to participate will be visited by an interviewer. Interviewers will have an identification badge, which they will show to each household along with study forms to indicate that they are part of the study.
The phone and in-person recruitment will encourage the household to participate in the in-person interview, screen for eligibility (e.g., the participant needs to have been living at the address when the NHES:2019 materials were sent), and set up a time to do the interview. The interviewer will attempt to speak with an adult living at this address who usually opens the mail. Should that person not be available, the interviewer will attempt to speak to another available adult living in the household. If, when the interviewer arrives, no one answers the door, the interviewer will leave a “Sorry We Missed You” card at the address. The card will indicate that someone stopped by and will return at a later day and time. It will also contain a phone number and e-mail address the sample member can contact to learn more about the study and/or schedule and interview.
All materials will be branded as coming from the National Center for Education Statistics (NCES). For sampled addresses that are expected to have a high likelihood of being Spanish speaking, the mailed recruitment materials will include both English and Spanish, and bilingual phone and in-person staff will be assigned to these cases.
To maintain engagement and increase the likelihood that recruited households keep their scheduled appointments, the interviewers also will ask recruited households to provide contact information that can be used to keep them engaged and aware of their upcoming interview appointment. Depending on the type of contact information provided, recruited households will receive: (1) a confirmation email or text shortly after scheduling the appointment (or an appointment reminder card if they are recruited in-person), and (2) a reminder email and/or phone call the day before the interview.
2.1.1.1 Use of Incentives
Given the length of the interview and the fact that the target population is comprised of survey nonrespondents, an incentive will be offered to households who complete the interview. Some examples of other incentives in in-person data collections follow:
In a comparison study of incentive groups in qualitative research, 3 monetary groups ($25, $50, and $75) all resulted in greater willingness to participate in the survey than no incentive or a nonmonetary incentive (Kelly et al 2017).
Similar incentive amounts were used in experiments as part of the National Survey of Family Growth (NSFG). In one experiment, a preincentive ($10 or $40 was mailed to sample members) and an additional $40 was offered to a small sub-set of nonresponders in later weeks in the data collection. The $80 total incentive increased response rates in both the screener and main survey by 10 and 12 percentage points respectively, and was more successful at recruiting different people (busy, college-educated, childless women, high-income men, and Hispanic men) into the sample than the $50 total incentive (Lepkowski et al. 2013).
The 2012/2014 Program for the International Assessment of Adult Competencies (PIAAC), an international household survey of adult skills conducted in-person, offered $5 to screener respondents. PIACC offered an additional $50 to households who then completed the assessment in-person. PIAAC achieved an overall weighted response rate of 70.3% for the main study (Hogan et al. 2016).
The in-person household component of the Medical Expenditure Panel Survey (MEPS) conducted an experiment beginning in 2008 for 5 rounds of the panel data collection. Households received $30, $50, or $70 for completing the survey. Response rates were higher for both the $50 and $70 incentive group relative to the $30 group, and the difference between the $70 (71.13%) and $50 (66.74%) groups was also statistically significant. In 2011, OMB approved the new $50 incentive for the MEPS collection.
The National Health and Nutrition Examination Survey (NHANES) involves multiple health exams and tests for respondents. NHANES has various incentive amounts for different subgroups of people. The maximum incentive for adults ages 16 and older who agree to participate in the exam component at a preselected time is $125. The amount decreases as cooperation decreases (e.g., adults ages 16 and older who refuse the exam component at a preselected time is $90). Additional follow-ups include additional $30-$50 incentives. NHANES respondents from low-income households often perceive the medical exams as a tangible, direct benefit to their well-being, which will not be true of the NHES interviews, wherein the benefits of participation are civic engagement and volunteerism, which are comparatively indirect benefits.
In the National Food Acquisition and Purchase Survey (FoodAPS), all contacted households were offered $5. The week-long data collection which included training, a food acquisition diary with scanners, two CAPI interviews, one computer-assisted telephone interview (CATI), and 3 respondent initiated call-in periods, offered eligible households a multi-part incentive designed to encourage initial agreement to participate. A base of $100 was offered with additional $10-$20 to other household members who agreed to participate. The average household incentive was $180 with a range of $130-$230 depending on household size. A previous experiment was conducted using $50 as the base incentive with the average household incentive receiving $130 and a range of $80-$180 depending on household size. Results showed that providing the larger base incentive yielded higher response rates.
The Survey of Consumer Finance (SCF) analyzed field effort outcomes between 2007 and 2010 after the base incentive increased from $20 in 2007 to $50 in 2010. On average, the $50 group needed four fewer attempted contacts before agreeing to participate relative to families receiving no incentive (To 2015).
Though many large studies use in-person data collection methods and/or burden respondents in excess of 1 hour, no study is a good match in the same type of sample members, burden, methods, and respondent experience to the proposed study of nonresponding households. To fully understand this unique target population, it is crucial to ensure a large enough, representative, and cooperative sample. Additionally, the need for high quality data from these sample members is of paramount importance for understanding reasons of nonresponse in studies like NHES. Based on the varying literature on higher incentive amounts and the unique target population of nonrespondents, all households will be offered a $120 cash incentive for completing the interview and this incentive will be mentioned in all recruitment materials.
2.1.2 Qualitative interview protocol
Qualitative interviews will be voluntary and will last 90 minutes to allow time to obtain consent, give participants sufficient time to reflect on the more open-ended questions included in this protocol, and conduct observations. Because this is exploratory research (trying to find out why something is happening as opposed to testing a hypothesis) maximizing contact time with the participants and giving them enough space to discuss their experience is crucial. Interviews will be conducted in both English and Spanish. All interviews will be audio-recorded with participants’ permission. They will consist of the following kinds of questions:
Qualitative interview questions that aim to gain a deep understanding of the household’s reasons for nonresponse and what might convince them to respond (covering domains such as understanding of surveys, privacy concerns, attitudes toward the government, attitudes toward education, time use, and civic engagement);
More structured questions, such as asking about the participant’s demographic characteristics, or asking if the participant remembers receiving the NHES mailings;
A mail interaction discussion, using a packet of example mail where the interviewer asks the participant to model how they sort mail and make decisions about what to keep, read, or throw away, while the interviewer probes about that decision process; and
A request for feedback on NHES:2019 screener mailed materials.
2.1.3 Observations
The interview will conclude with the interviewer making observations about the home and/or household members, beyond any observations recorded while interacting with the participant during the interview. Observations are important to include in qualitative research because interview questions can often only address attitudes or actions of which participants are cognitively aware. There are likely other observable aspects of participants’ lives that they may not explicitly connect to their lack of participation (e.g., how they organize their mail), as well as observable qualities that nonresponding households may have in common (e.g., whether they display children’s educational achievements). These observations will be used to capture such more nuanced indicators.
The finalized interview protocol and contact materials for the In-Person Study of NHES:2019 Nonresponding Households qualitative interviews can be found in the Attachment B of this document.
2.2. Address/neighborhood observation
Approximately 750 addresses will be sampled for address and neighborhood observations. Of these, 500 will be also invited to take part in the qualitative interview study (see above), and for the remaining 250 households, only observations will be collected. The objective of these observations is to determine the types of addresses that are prone to nonresponse or having their NHES mailings be undeliverable, and to assess the accuracy of the information available on the frame for such addresses.
The objective of these observations, which will be carried out by staff contracted for NHES, is to determine the types of addresses that are prone to nonresponse or having their NHES mailings be undeliverable and to assess the accuracy of the information available on the frame for such addresses. The instrument has been programmed to be administered on a tablet or computer to ease data collection efforts and minimize error. Each observer will receive a tablet to complete their observations. The 7-minute observation instrument is designed to evaluate variables from the address-based sample frame. Several of the items are based on existing instruments (e.g., the Neighborhood Observation Instrument (NOI), Contact History Instrument (CHI), the Los Angeles Family and Neighborhood Survey). The instrument also includes a series of characteristics that interviewers will look for and mark off when they observe them for that address or neighborhood. There is also space for entering notes that might provide helpful context about the address or neighborhood. Exhibit 2 below displays a summary of the proposed measures, the frame variables that will be evaluated, and the potential impacts those findings could have on the design of future NHES administrations. To increase efficiency, observers may audio record their observations and transcribe them onto the instrument later.
Exhibit 2. Summary of proposed measures, frame variables evaluated, and potential impact on NHES design
Measure |
Frame Variable(s) evaluated |
Potential Impact |
Structure type |
|
Mailing method (e.g., use courier service such as FedEx) |
Contact materials (e.g., use larger font for address to minimize sorting issues for multi-unit addresses, tailor language of contact materials) |
||
Incentive use (e.g., use varied incentives by structure type) |
||
Frame accuracy (e.g., evaluate frame information to improve adaptive design models) |
||
Occupancy |
|
Frame accuracy (e.g., evaluate frame information to improve adaptive design models) |
Eligibility (e.g., confirm whether the postmaster ‘vacant household’ status is accurate) |
||
Eligibility |
|
Topical Eligibility (e.g., assess whether the nonresponse process differs based on topical eligibility) |
Frame accuracy (e.g., evaluate frame information particularly to improve adaptive design models) |
||
Contact materials (e.g., tailor language used in survey invitation to alleviate privacy concerns of certain households such as households with children) |
||
Mail Access |
|
Mailing method (e.g., using courier services that provide door-to-door service) |
Contact materials (e.g., use bolder envelope designs such as larger font for addresses where mail mix-up is likely) |
||
Frame accuracy (e.g., evaluate frame information particularly to improve adaptive design models) |
||
Household income |
|
Incentive use (e.g., use varying incentive structures by income level) |
Frame accuracy (e.g., evaluate frame information particularly to improve adaptive design models) |
||
Additional address / neighborhood observations |
None (these observations will be used to assess whether additional census planning database variables should be used for making data collection protocol changes) |
Contact materials (e.g., tailor emphasis of individual (incentive) versus communal (policy impact) advantages of NHES participation) |
As part of the 7-minute observation, observers will also take a picture of the sampled address to validate that the observer was indeed present at the address, to document a time and location of the visit, and to clarify any issues related to the observation the observer may have recorded. The photos will only be taken from public spaces such as the street, and will not be taken while on someone’s private property. The finalized observation protocol for the In-Person Study of NHES:2019 Nonresponding Households is provided in Attachment B.3 of this document.
The priority for analysis will be on identifying actionable responses to the themes identified in the interviewer observations and the participant responses and comments. The protocols for this study are written with that aim in mind. Some global changes to NHES are likely to result from the In-Person Study of NHES:2019 Nonresponding Households, but it is anticipated that most findings will be targeted at specific priority or hard-to-reach subgroups that can be identified prior to the start of data collection. This may include, for example, changing the timing or mode of nonresponse follow-up for certain subgroups, altering the wording or presentation of contact materials, or changing the value or timing of incentives.
To help ensure that the results of this study reach the maximum possible audience, contribute to the survey methodology literature, and serve as a resource for others conducting federal government household surveys (particularly those using ABS frames or contacting household by mail), it is anticipated that the findings may be shared through the following outlets:
An official NCES publication posted on the NCES website that summarizes the methodology and key findings of the study. This report will identify key themes from the interviews and propose actionable strategies that could be implemented in future administrations of NHES and other household surveys.
A manuscript, based on the above report, that is submitted to a leading survey methodology journal, such as Public Opinion Quarterly, the Journal of Survey Statistics and Methodology, or Field Methods.
At least two presentations submitted to conferences, such as the American Association for Public Opinion Research national conference or the Federal Committee on Statistical Methodology Research and Policy conference.
References
AAPOR Task Force on Survey Refusals. (2014). Current knowledge and considerations regarding survey refusals. American Association for Public Opinion Research.
AAPOR Task Force on Address-Based Sampling. (2016). Address-based sampling. American Association for Public Opinion Research.
Abraham, K., Maitland, A., and Bianchi, S. (2006). Nonresponse in the American Time Use Survey: Who is missing from the data and how much does it matter? Public Opinion Quarterly, 70, 676-703.
Ahlmark, N., Algren, M., Holmberg, T., Norredam, M., Nielsen, S., Blom, A., Bo, A., and Juel, K. (2015). Survey nonresponse among ethnic minorities in a national health survey -- a mixed-method study of participation, barriers, and potentials. Ethnicity & Health, 20, 611-632.
Amaya, A., and Harring, J. (2017). Assessing the effect of social integration on unit nonresponse in household surveys. Journal of Survey Statistics and Methodology, 2017, 480-508.
ASA Task Force on Improving the Climate for Surveys. (2017). Interim report. American Association for Public Opinion Research.
Bates, N. (2007). The Simplified Questionnaire Report. Results from the Debriefing Interviews. Accesses 4/10/18 from https://www.census.gov/srd/papers/pdf/rsm2007-03.pdf
Brick, J., and Williams, D. (2013). Explaining rising nonresponse rates in cross-sectional surveys. Annals of the American Academy of Political and Social Science, 645, 36-59.
Czajka, J., and Beyler, A. (2016). Declining response rates in federal surveys. Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services. Washington, DC.
Groves, R., Cialdini, R., and Couper, M. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475-495.
Groves, R., Presser, S., Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68, 2-31.
Harcombe, H., Derrett, S., Herbison, P., and McBride, D. (2011). "Do I really want to do this?" Longitudinal cohort survey participants' perspectives on postal survey design: a qualitative study. BMC Medical Research Methodology, 11, 1-9.
Hogan, J., Thornton, N., Diaz-Hoffmann, L., Mohadjer, L., Krenzke, T., Li, J., VanDeKerckhove, W., Yamamoto, K., and Khorramdel, L. (2016). U.S. Program for the International Assessment of Adult Competencies (PIAAC) 2012/2014: Main Study and National Supplement Technical Report (NCES 2016-036REV). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Available from http://nces.ed.gov/pubsearch
Kelly, B., Margolis, M., McCormack, L., LeBaron, P.A., and Chowdhury, D. (2017). What affects people’s willingness to participate in qualitative research? An experimental comparison of five incentives. In Field Methods (29).
Kulka, R. A., Holt, N. A., Carter, W., and Dowd, K. L. (1991). Self-reports of time pressures,
concerns for privacy, and participation in the 1990 mail Census. Proceedings of the Bureau of the Census Annual Research Conference, Arlington, VA.
Lepkowski, J., Mosher, W., Groves, R., West, B., Wagner, J., and Gu, H. (2013). Responsive design weighting, and variance estimation in the 2006-2010 National Survey of Family Growth. 2(158). U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. Retrieved from https://stacks.cdc.gov/view/cdc/22069
Little, R. and Vartivarian, S. (2005). Does weighting for nonresponse increase the variance of survey means? Survey Methodology, 31, 161-168.
McPhee, C., Bielick, S., Masterton, M., Flores, L., Parmer, R., Masica, S., Shin, H., Stern, S., and McGowan, H. (2015). National Household Education Surveys Program of 2012: Data file user’s manual (NCES 2015-030). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.
McPhee, C., Jackson, M., Bielick, S., Masterton, M., Battle, D., McQuiggan, M., Payri, M., Cox, C., and Medway, R. (2018). National Household Education Surveys Program of 2016: Data file user’s manual (NCES 2017-100). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.
Nakash, R., Hutton, J., Lamb, S., Gates, S., and Fisher, J. (2008). Response and non-response to postal questionnaire follow-up in a clinical trial – a qualitative study of the patient's perspective. Journal of Evaluation in Clinical Practice, 14, 226-235.
O’Neill, G. and Sincavage, J. (2004). Response Analysis Survey: A Qualitative look at Response and Nonresponse in the American Time Use Survey. Retrieved from https://www.bls.gov/ore/pdf/st040140.pdf Washington, DC: Bureau of Labor Statistics.
Presser, S., and McCulloch, S. (2011). The growth of survey research in the United States: Government-sponsored surveys, 1984–2004. Social Science Research, 40, 1019-1024.
Singer, E., and Presser, S. (2008). Privacy, confidentiality, and respondent burden as factors in telephone survey nonresponse. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. de Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, and R. L. Sangster (Eds.), Advances in telephone survey methodology (449–470). New York: John Wiley.
Singer, E. (2016). Reflections on surveys’ past and future. Journal of Statistics and Survey Methodology, 4, 463-475.
To, N. (2015). Review of Federal Survey Program Experiences with Incentives. Washington, DC: Bureau of Labor Statistics. Retrieved from: https://www.bls.gov/cex/research_papers/pdf/Review-of-Incentive-Experiences-Report.pdf
U.S. Census Bureau. (2014). Design and Methodology: American Community Survey. Washington, DC: U.S. Census Bureau. Retrieved from: https://www.census.gov/programs-surveys/acs/methodology/design-and-methodology.html.
United States Census Bureau. (2017). CAPI Manual for NHIS field representatives. Retrieved from https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Survey_Questionnaires/NHIS/2017/frmanual.pdf
User’s guide for the paradata file and codebook documentation: Special file covering 2011-2015 of the National Survey of Family Growth. (n.d.). Retrieved from https://www.cdc.gov/nchs/data/nsfg/NSFG2011_2015_Paradata_UserGuide.pdf
Walsh, R., Dahlhamer, J., and Bates, N. (2013). Assessing interviewer observations in the NHIS. Retrieved from https://www.researchgate.net/publication/296847278_Assessing_Interviewer_Observations_in_the_NHIS
Zelnack, M. and David, M. (2013). Impact of Multiple Contacts by Computer-Assisted Telephone Interview and Computer-Assisted Personal Interview on Final Interview Outcome in the American Community Survey. Washington, DC: U.S. Census Bureau. Retrieved from: https://www.census.gov/content/dam/Census/library/working-papers/2013/acs/2013_Zelenak_01.pdf
Attachment A. Count of sampled households by response status, and weighted screener response rate, by selected household characteristics: NHES:2016
Table A-1. Count of sampled households by response status, and weighted screener response rate, by selected household characteristics
Household characteristic |
Count of sampled households |
Weighted screener response rate |
||||
Total |
Responded |
Refused |
Ineligible |
Unknown eligibility |
||
Total |
206,000 |
115,342 |
1,606 |
19,136 |
69,916 |
66.4 |
Frame variables |
|
|
|
|
|
|
Sampling stratum |
|
|
|
|
|
|
Tracts with 25% or more Black persons |
41,200 |
18,593 |
330 |
5,472 |
16,805 |
58.2 |
Tracts with 40% or more Hispanic persons |
30,900 |
13,906 |
206 |
3,404 |
13,384 |
55.9 |
All other tracts |
133,900 |
82,843 |
1,070 |
10,260 |
39,727 |
69.4 |
Tract poverty rate |
|
|
|
|
|
|
20% or higher |
64,760 |
29,030 |
468 |
9,025 |
26,237 |
59.5 |
Below 20% |
141,240 |
86,312 |
1,138 |
10,111 |
43,679 |
68.9 |
Census region1 |
|
|
|
|
|
|
Northeast |
35,398 |
20,341 |
281 |
3,019 |
11,757 |
67.2 |
South |
82,478 |
43,580 |
662 |
8,500 |
29,736 |
64.0 |
Midwest |
42,817 |
25,859 |
336 |
4,056 |
12,566 |
70.6 |
West |
45,307 |
25,562 |
327 |
3,561 |
15,857 |
65.6 |
Route type |
|
|
|
|
|
|
City style/street |
155,113 |
93,178 |
1,252 |
11,302 |
49,381 |
68.4 |
P.O. box |
2,181 |
785 |
19 |
845 |
532 |
74.5 |
High rise |
48,507 |
21,276 |
333 |
6,958 |
19,940 |
58.9 |
Rural route |
199 |
103 |
2 |
31 |
63 |
67.8 |
Delivery point is a drop point |
|
|
|
|
|
|
Yes |
3,784 |
1,558 |
50 |
573 |
1,603 |
57.5 |
No |
202,216 |
113,784 |
1,556 |
18,563 |
68,313 |
66.5 |
Dwelling type |
|
|
|
|
|
|
Single family |
150,545 |
91,419 |
1,219 |
10,650 |
47,257 |
68.9 |
Multiple unit |
53,274 |
23,138 |
368 |
7,641 |
22,127 |
58.5 |
Dwelling type unknown |
2,181 |
785 |
19 |
845 |
532 |
74.5 |
Number of adults in household |
|
|
|
|
|
|
1 |
78,318 |
39,090 |
589 |
8,008 |
30,631 |
61.1 |
2 |
50,592 |
32,743 |
395 |
2,388 |
15,066 |
70.6 |
3 or 4 |
38,440 |
26,435 |
288 |
1,225 |
10,492 |
73.1 |
5–8 |
9,200 |
6,321 |
63 |
260 |
2,556 |
72.7 |
Number of adults unknown |
29,450 |
10,753 |
271 |
7,255 |
11,171 |
61.6 |
Children in household |
|
|
|
|
|
|
Yes |
47,236 |
28,146 |
297 |
2,415 |
16,378 |
65.8 |
No/unknown |
158,764 |
87,196 |
1,309 |
16,721 |
53,538 |
66.5 |
See notes at end of table.
Table A-1. Count of sampled households by response status, and weighted screener response rate, by selected household characteristics—Continued
Household characteristic |
Count of sampled households |
Weighted screener response rate |
||||
Total |
Responded |
Refused |
Ineligible |
Unknown eligibility |
||
Phone number matched |
|
|
|
|
|
|
Yes |
134,574 |
84,010 |
1,099 |
7,687 |
41,778 |
69.4 |
No |
71,426 |
31,332 |
507 |
11,449 |
28,138 |
60.6 |
Home tenure |
|
|
|
|
|
|
Rent |
52,220 |
23,091 |
330 |
6,037 |
22,762 |
56.6 |
Own |
120,115 |
79,806 |
988 |
5,165 |
34,156 |
71.6 |
Home tenure unknown |
33,665 |
12,445 |
288 |
7,934 |
12,998 |
61.0 |
Income |
|
|
|
|
|
|
$50,000 or less |
83,949 |
43,740 |
673 |
8,065 |
31,471 |
62.8 |
$50,001–$100,000 |
55,967 |
36,271 |
401 |
2,446 |
16,849 |
70.2 |
$100,001–$150,000 |
21,839 |
14,376 |
147 |
822 |
6,494 |
70.5 |
$150,001 or more |
14,772 |
10,189 |
114 |
544 |
3,925 |
73.3 |
Income unknown |
29,473 |
10,766 |
271 |
7,259 |
11,177 |
61.6 |
Treatment variables |
|
|
|
|
|
|
Assigned mode at first screener mailing |
|
|
|
|
|
|
Paper |
171,000 |
97,315 |
1,256 |
15,819 |
56,610 |
67.2 |
$5 incentive |
126,000 |
72,064 |
914 |
11,593 |
41,429 |
67.5 |
Other incentive ($2 or modeled) |
45,000 |
25,251 |
342 |
4,226 |
15,181 |
66.5 |
Web2 |
35,000 |
18,027 |
350 |
3,317 |
13,306 |
62.1 |
Screener incentive |
|
|
|
|
|
|
$5 |
161,000 |
90,091 |
1,264 |
14,910 |
54,735 |
66.3 |
$2 |
10,000 |
5,449 |
71 |
922 |
3,558 |
64.9 |
Modeled incentives3 |
35,000 |
19,802 |
271 |
3,304 |
11,623 |
67.0 |
$0 |
1,750 |
1,393 |
18 |
55 |
284 |
82.8 |
$2 |
6,996 |
5,176 |
63 |
210 |
1,547 |
77.2 |
$5 |
21,007 |
11,502 |
150 |
1,932 |
7,423 |
64.2 |
$10 |
5,247 |
1,731 |
40 |
1,107 |
2,369 |
54.0 |
1 Northeast includes Pennsylvania, New York, New Jersey, Connecticut, Rhode Island, Massachusetts, Vermont, New Hampshire, and Maine. South includes Florida, Georgia, South Carolina, North Carolina, Virginia, District of Columbia, Maryland, Delaware, West Virginia, Alabama, Mississippi, Tennessee, Kentucky, Arkansas, Louisiana, Texas, and Oklahoma. Midwest includes North Dakota, South Dakota, Nebraska, Kansas, Missouri, Iowa, Minnesota, Wisconsin, Illinois, Indiana, Michigan, and Ohio. West includes New Mexico, Colorado, Wyoming, Montana, Idaho, Utah, Arizona, Nevada, Washington, Oregon, California, Hawaii, and Alaska.
2All households assigned to the web mode received a $5 screener incentive.
3Incentives in the modeled group were assigned according to predicted response propensity, with households with a higher predicted response propensity receiving a lower incentive.
NOTE: Weighted screener response rate is calculated using the single-eligibility formula (AAPOR RR3).
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Household Education Surveys Program (NHES) of 2016.
Attachment B. In-Person Study of NHES:2019 Nonresponding Households Qualitative Interview Materials
B.1. English Materials
Attachment B.1. presents the English version of the materials that will be used to interact with households sampled for the qualitative interview. These materials include:
First Reminder Postcard
Second Reminder Postcard
Phone Follow-up Recruitment Script
“Sorry We Missed You!” Card
In-Person Recruitment Script
Commonly Asked Questions
Interview Consent Form
Interview Protocol
Income Show Card
B.1.1. Invitation Letter
<Mailing_add1>
<Mailing_add2>
Dear <City> Household:
The National Center for Education Statistics (NCES) and American Institutes for Research (AIR) are doing an important study in your area.
This study will help us understand how to make NCES research more useful and helpful. We would like to get your feedback on a variety of topics, such as how you like to spend your time, how you go about day-to-day tasks like sorting your mail, and how you feel about participating in research.
This voluntary, 90-minute conversation takes place in your home at a convenient time for you.
We know that you are busy, and we have enclosed $5 as a thank you for your time. Because your participation is so important, you also will get $120 cash as a thank you for sharing your opinions and experiences with us.
To see if you are eligible to participate, please do one of the following:
Call us toll-free at 1-XXX-XXX-XXXX
E-mail us at [email protected]
Thank you in advance for participating. We look forward to hearing from you.
Sincerely,
[SIGNATURE]
James Lynn Woodworth
Commissioner, National Center for Education Statistics
Authorization and Confidentiality of Information
American Institutes for Research is administering this voluntary study on behalf of the National Center for Education Statistics (NCES). There are no penalties should you choose not to participate in this study. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The National Center for Education Statistics (NCES), within the U.S. Department of Education, is authorized to conduct the National Household Education Survey (NHES) by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).
B.1.2. First Reminder Postcard
Would you like $120 for sharing your opinions?
The National Center for Education Statistics (NCES) and American Institutes for Research (AIR) are doing an important study in your area. This study will help us understand how to make NCES research more useful and helpful.
Because your participation is so critical, you will get $120 cash as a thank you for sharing your opinions and experiences with us.
This voluntary, 90-minute conversation takes place in your home at a convenient time for you.
We would like to get your feedback on a variety of topics, such as how you like to spend your time, how you go about day-to-day tasks like sorting your mail, and how you feel about participating in research.
A researcher from AIR will stop by in the next few weeks to invite you to participate.
To find out sooner if you are eligible to participate: call us toll-free at 1-XXX-XXX-XXXX or e-mail us at [email protected].
Thank you in advance for participating. We look forward to hearing from you!
B.1.3. Second Reminder Postcard
Chat with us to help
make research more useful!
The National Center for Education Statistics (NCES) and American Institutes for Research (AIR) are doing an important study in your area. This study will help us understand how to make NCES research more useful and helpful.
An AIR researcher will stop by in the next few weeks to invite you to participate.
To find out sooner if you are eligible to participate: call us toll-free at 1-XXX-XXX-XXX
or e-mail us at [email protected].
We would like to get your feedback on a variety of topics, such as how you like to spend your time, how you go about day-to-day tasks like sorting your mail, and how you feel about participating in research.
Because your participation is so important, you will get $120 cash as a thank you for sharing your opinions and experiences with us.
This voluntary, 90-minute conversation takes place in your home at a convenient time for you.
Thank you in advance for your participation. We look forward to hearing from you!
B.1.4. Phone Follow-up Recruitment Script
IF NO ONE ANSWERS THE PHONE, LEAVE THE FOLLOWING MESSAGE:
Hello, this is [YOUR NAME] from American Institutes for Research, calling on behalf of the National Center for Education Statistics. We are doing an important study in your area. We would like to meet with you to get your feedback on a variety of topics, such as how you like to spend your time and how you go about daily tasks like sorting mail. Because your participation is so critical, you will get $120 cash as a thank you for speaking with us. If you would like to participate, please call us toll free at 1-xxx-xxx-xxxx.
IF SOMEONE ANSWERS THE PHONE:
Hello, this is [NAME] from American Institutes for Research, calling on behalf of the National Center for Education Statistics. Am I speaking with a resident of [ADDRESS]?
If yes, continue with the next paragraph.
If no, say: I am sorry. I have the wrong number. Thank you for your time. Have a nice day. End phone call.
I am calling to follow up on a letter that was recently mailed to your house. The National Center for Education Statistics and American Institutes for Research are conducting an important study in your area that will help us understand how to make our research more useful and helpful for people like you. We would like to meet with you to get your feedback on a variety of topics, such as how you like to spend your time and go about daily tasks like sorting mail. If you complete a 90-minute conversation with us in your home, we will give you $120 cash to show our thanks.
Are you interested in this opportunity?
If yes, continue with the next paragraph.
If no, try to address their concerns. If still not interested, say: No problem. Thank you for your time. Have a good day. End phone call.
Great. First, I need to ask a few questions to make sure you are eligible to participate.
If yes, ask if you can speak with them.
When that person is on the phone, go back to the study introduction paragraph and then restart the screener.
If the person is not available right now, ask when would be a good time for you to call back, thank them for their time, and then end the call. Try to get the other person’s first name to facilitate the return call.
If no one else lived here during that time period, thank them for their time, end phone call and do not call again.
Ok, great. Can you please confirm your age?
If 18 or older, continue with next screener item.
If 17 or younger, say: Unfortunately, only adults can participate. Is there an adult at home with whom I can speak?
When the adult is on the phone, go back to the study introduction paragraph and then restart the screener.
If an adult is not available right now, ask when would be a good time for you to call back to speak with one, thank them for their time, and then end the call. Try to get the other person’s first name to facilitate the return call.
Great, thanks. Are you involved in handling your household’s postal mail - for example, collecting, reviewing, or responding to the mail?
If yes, continue with next screener item.
If no, say: Unfortunately, we can only meet with one of the people who handles your household’s mail. Can I please speak to one of those people?
When that person is on the phone, go back to the study introduction paragraph and then restart the screener.
If the person is not available right now, ask when would be a good time for you to call back, thank them for their time, and then end the call. Try to get the other person’s first name to facilitate the return call.
Great, thank you! You are eligible to participate in the study. What days [this/next] week would be convenient for you to meet with us?
Looking at when interview team is available in that area, work with participant to schedule the interview. Our preference is to do interviews sooner than later.
Once interview date/time are decided, record them and say: We have you scheduled on [DATE] at [TIME]. You will be meeting with [INTERVIEWER NAME].
We would like to send you a confirmation message with the date and time of your interview. Do you prefer that we text you or that we e-mail you?
If willing to receive confirmation message: And what is the best [phone number/e-mail address] for reaching you?
If not willing to receive confirmation message, reassure them that this information will only be used for communicating with them about the interview. If still not willing, say: Ok, no problem.
We would also like to reach out to you the day before the meeting to confirm that this time still works for you. Do you prefer that we call you or that we e-mail you?
If willing to receive reminder message and request a different mode than for the confirmation message: And what is the best [phone number/e-mail address] for reaching you?
If not willing to receive reminder message, reassure them that this information will only be used for communicating with them about the interview. If still not willing, say: Ok, no problem.
Thank you for participating! If any conflicts come up between now and [DATE OF INTERVIEW] please call our toll-free number at 1-XXX-XXX-XXXX or email us at [email protected].
We look forward to seeing you soon. Bye. End call.
B.1.5. “Sorry We Missed You!” Card
Sorry We Missed You!
The National Center for Education Statistics (NCES) and American Institutes for Research (AIR) are doing an important study in your area. We stopped by to invite you to participate.
Contact us at 1-xxx-xxx-xxxx or [email protected] to see if you are eligible! |
B.1.6. In-Person Recruitment Script
INTRODUCTION
If the person who answers the door is clearly a child, ask about speaking to an adult in the house. Otherwise, say:
Hi, my name is [YOUR NAME] from American Institutes for Research. We recently sent a letter to your house about an important study we are doing with the National Center for Education Statistics. We would like to have a conversation with you to get your opinions on a variety of topics, such as how you like to spend your time, how you go about day-to-day activities like sorting your mail, and how you feel about participating in research. The conversation lasts 90 minutes, and we will give you $120 cash as a thank you for your time. Are you interested in this opportunity?
If yes, continue with the next section.
If no, try to address their concerns. If still not interested, say: No problem, I understand. Is anyone else from this household available to speak with us?
If yes and available, repeat the introduction once that person comes to the door.
If yes but not available right now, say: No problem. We can come back at a more convenient time for them. What would be a good time for us to come back? [record date/time, and if possible the other household member’s first name to facilitate the return visit] Here is a card with more information about the study that you can give to them. It also lists our contact information in case they would like to contact us sooner about participating. [hand Sorry We Missed You card to household member] Thank you for your time! Have a nice day.
If no one else lives here or not interested in you speaking with anyone else, say: I understand. Would you be willing to answer a few quick questions instead? It will take less than five minutes.
If no: No problem. Thank you for your time. Have a nice day. (end interview)
If yes, go to Short Interview with Full-Interview Refusals: Great, thank you.
SHORT INTERVIEW WITH FULL-INTERVIEW REFUSALS
First, can you please confirm that you are at least 18 years old?
Yes (continue)
No (go to end interview)
Don’t know (go to end interview)
Refused (go to end interview)
Can you also confirm that we are at [STREET ADDRESS]?
Yes (continue)
No (go to end interview)
Don’t know (go to end interview)
Refused (go to end interview)
Did you live or stay here most of the time between January and April 2019?
Yes (continue)
No (go to end interview)
Don’t know (go to end interview)
Refused (go to end interview)
Do you handle your home’s mail – for example, do you collect, review, or respond to the mail?
Yes (continue)
No (go to end interview)
Don’t know (go to end interview)
Refused (go to end interview)
Have you gotten any mail from the Department of Commerce or Census Bureau since January 1st, 2019?
Yes (continue)
No (continue)
Don’t know (continue)
Refused (continue)
How about any mail about the National Household Education Survey? Have you gotten any mail like this since January 1st, 2019?
Yes (ask QG)
No (go to QH)
Don’t know (go to QH)
Refused (go to QH)
Would you be willing to share some of the reasons you did not complete this survey? (Allow respondent a moment to think.)
Are any of the people who live here age 20 or less?
Yes (go to end interview)
No (go to end interview)
Don’t know (go to end interview)
Refused (go to end interview)
End interview: Those are all my questions. Thank you for your time. Have a nice day!
SCREENER
Once the interested household member is at the door:
Great! First, I need to ask a few quick questions to make sure you are eligible to participate.
Can you please confirm that we are at [STREET ADDRESS]?
If yes, continue with next screener item.
If no, say: Unfortunately, I can only meet with someone who lives at [STREET ADDRESS]. I am sorry; I must have the wrong house. Thank you for your time. Have a nice day. End conversation.
Did you live or stay here most of the time between January and April 2019?
If yes, continue with next screener item.
If no, say: Unfortunately, we can only meet with people who lived or stayed at this address most of the time between January and April 2019. Is there anyone else living here who lived or stayed here most of that time?
If yes and available, repeat the introduction once that person comes to the door.
If yes but not available right now, say: No problem. We can come back at a more convenient time for them. What would be a good time for us to come back? [record date/time, and if possible the other household member’s first name to facilitate return visit] Here is a card with more information about the study that you can give to them. It also lists our contact information in case they would like to contact us sooner about participating. [hand Sorry We Missed You card to household member] Thank you for your time! Have a nice day.
If no one else lives here or not interested in you speaking with anyone else, say: No problem. Thank you for your time. Have a good day.
Ok, great. Can you please confirm your age? [do not ask this if respondent is clearly much older than 18, but if there is any uncertainty make sure you confirm their age]
If 18 or older, continue with next screener item.
If 17 or younger, say: Unfortunately, only adults can participate. Is there an adult at home with whom I can speak?
If yes and available, repeat the introduction once that person comes to the door.
If yes but not available right now, say: No problem. We can come back at a more convenient time for them. What would be a good time for us to come back? [record date/time, and if possible the other household member’s first name] Here is a card with more information about the study that you can give to them. It also lists our contact information in case they would like to contact us sooner about participating. [hand Sorry We Missed You card to household member] Thank you for your time! Have a nice day.
If no one else lives here or not interested in you speaking with anyone else, say: No problem. Thank you for your time. Have a good day.
One last question: Are you involved in handling your household’s postal mail - for example, collecting, reviewing, or responding to the mail?
If yes, continue with next screener item.
If no, say: Unfortunately, we need to have the conversation with one of the people who handles your household’s mail. Can I please speak to one of those people?
If yes and available, repeat the introduction once that person comes to the door.
If yes but not available right now, say: No problem. We can come back at a more convenient time for them. What would be a good time for us to come back? [record date/time, and if possible the other household member’s first name to facilitate return visit] Here is a card with more information about the study that you can give to them. It also lists our contact information in case they would like to contact us sooner about participating. [hand Sorry We Missed You card to household member] Thank you for your time! Have a nice day.
If no one else lives here or not interested in you speaking with anyone else, say: No problem. Thank you for your time. Have a good day.
SCHEDULING
Great, thank you! You are eligible to participate in the study. We can meet right now if it is convenient for you. The conversation will take about 90 minutes.
If they are available now, start the interview.
If they are not available now, say: No problem. We can come back at a time that is more convenient for you. What days this week would be convenient for you to meet?
Based on your availability, work with the respondent to find a time for the interview.
Record the date/time for yourself, and on an Appointment card, and say: We have you scheduled on [DATE] at [TIME]. Here’s an Appointment confirmation card for you to keep as a reminder.
Hand them the card and then say: And could I have your first name for my appointment calendar?
Record name, then say: We would like to reach out to you the day before the conversation to confirm that this time still works for you. Do you prefer that we call you or that we e-mail you?
Once they respond, say: And what is the best [phone number/email address] for reaching you? If they express concern about providing this information, let them know this will only be used to remind them about the appointment and confirm they are still available.
Record answer for your records and say: Thank you for participating! If any questions or conflicts come up between now and [DATE OF INTERVIEW], please call or email us using the information listed on your Appointment card. I look forward to seeing you soon. Have a nice day.
B.1.7. Commonly Asked Questions
Commonly
Asked Questions
What is this study about? What will you do with my responses?
The focus of the study is to understand how to improve research done by the National Center for Education Statistics, which is part of the Department of Education. We would like to hear about your opinions and experiences with a variety of topics. For example, we would like to know what kinds of things in your everyday life are important to you. We would also like to know about how you handle certain tasks, like collecting and reviewing your home’s postal mail. Finally, we would like to get your feedback on some of our research materials. Ultimately, this information will help us improve the National Household Education Surveys and assure that they are relevant to all household in the United States.
Who is conducting this study?
The Department of Education, specifically the National Center for Education Statistics (NCES), is conducting this study. You can learn more about NCES by going to www.nces.ed.gov.
American Institutes for Research (AIR), a not-for-profit behavioral and social science research firm, is carrying out this study on behalf of NCES. You can learn more about AIR by going to www.air.org.
How much time will this take?
The conversation should take about 90 minutes to complete.
Where will this take place?
The conversation will take place at your home at a time that is convenient for you. We would like to speak with you in your home because we want you to share your experiences and opinions in a setting that is comfortable and familiar to you.
Who will know if I participate in the study?
Only the study team at AIR will know who participated in the study. The information collected during our conversation will be stored on AIR’s secure computer servers. We will not link what you say to anything identifiable about you, such as your name or address.
What do I get if I meet with you?
You will receive $120 cash as a thank you for your participation.
Who can I contact if I have questions about participating in the study?
You can call us toll-free at 1-XXX-XXX-XXXX or e-mail us at [email protected].
B.1.8. Informed Consent Form
Study
Participation Consent Form
The U.S. Department of Education’s National Center for Education Statistics (NCES) is conducting this study to understand how to make its research more useful and helpful. NCES has hired American Institutes for Research (AIR) to carry out this study.
We are asking you to participate in a 90-minute conversation where we will ask your opinion about a variety of topics, such as how you like to spend your time, how you go about day-to-day activities like sorting your mail, and how you feel about participating in research.
Your participation will contribute valuable insights that will help us improve NCES surveys; however, your participation is completely voluntary, and there are no penalties should you choose not to participate in this study. You may stop the conversation at any time without penalty.
The information you share with us will be combined with the responses of other participants in a summary report that does not identify you or anyone else individually. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
If you have any questions or concerns related to your participation in this interview, please contact Sarah Grady at the National Center for Education Statistics. You can reach her at 202-245-7063 or [email protected].
We would like to record this interview to make sure that we accurately capture the details that you provide. However, if you do not agree to be recorded, we will not record the interview. If you agree to be recorded, only AIR and NCES employees working on this study will be able to listen to the recording. All recordings will be destroyed at the end of the study. Do we have your permission to record the interview?
YES NO
By signing your name below, you agree to participate in this study.
Signed: ____________________________________________ Date: ___________________
Printed Name: ________________________________________
B.1.9. Interview Protocol
The goal of each interview is to conduct an efficient interview and observation with an ethnographic approach in order to glean an in-depth understanding of participants’ perspectives and experiences on a variety of topics related to surveys. The interaction should be a conversation where the participant does most of the talking and you, through active listening and engagement, probe them on particular objectives. Therefore, there is no set script. You must rely on your experience and training to emerge from the situation with the rich detail needed to address why this particular household did not respond to the National Household Education Survey.
Previous research has elucidated the role of certain factors, such as civic engagement and privacy concerns, in why people do not respond to surveys. However, an important component of this qualitative research project is to explore the context (physical, social, and emotional) in which these nonrespondents are embedded, so that themes can arise from the data that help explain why people did not respond to the screener. We are interested in why they did not participate, which is likely to be different for different people. For some, they may not be interested in taking surveys in general. For others, they may have wanted to participate but other factors in their lives intervened. Additionally, some households may not have received the initial screener. To accomplish that goal, the interviewer and the interview guide need to be flexible and able to explore and probe on issues the participant deems relevant.
Before you begin the interview, give the informed consent document to the participant, answer any questions they may have, and begin recording with their permission. Once they sign the informed consent, put it away in a folder.
Start with rapport building through basic questions about household make-up (Domain 1).
Next, proceed with the Experience of Mail Delivery and Experience of NHES Materials and Screener domains (Domains 2 and 3).
After this, you should rely on your training to decide what is the best course forward that will yield the most informative interview; however, you should at least touch on all of the domains listed in this document.
When there are about 5 minutes left in the interview, ask the questions in the demographics section.
Finally, hand the participant their incentive and make sure they sign the incentive receipt form.
Though we generally want to start with the Experience of Mail Delivery and Experience of NHES Letter and Screener domains, we expect that other domains, such as time use and privacy concerns, will arise during these discussions. The domains may flow into one another or may not be discrete categories to the participant. For example, you may begin talking about the participant checking her mail, which leads into a conversation about what activities she enjoys in her free time. Should that happen, you would want to interweave questions about time use and the related objectives. Although the conversation will be free-flowing, we want to get a chronological sense of how they handle their mail. For example, do they check their mailbox every evening around the same time? Do they review the mail immediately? Do they sort the mail while standing at the mailbox? Do they come inside and put it away to sort for later?
The remainder of this document lays out the key domain/objectives to be covered in the interview and proposes sample questions and probes you may want to ask in order to fully explore that objective. However, the key to this interview is your ability to listen to participants’ responses and devise follow-up questions in the moment that serve to deepen our understanding of their point of view. The sample questions are examples and may not be relevant or meaningful to every participant. Also, it is very important to let the participant use their own words to answer questions. Probe without putting words in their mouth with very open questions. For instance, ask “Why do you do that?” as opposed to “Do you do that so you can X?”
In addition to understanding the participants’ experiences around the given objectives, each interview will include two activities as part of the Experience of Mail Delivery and Experience of NHES Materials and Screener domains.
Mail Delivery: In the first, you will offer the participant a stack of mail and the participant will talk aloud about the process of sorting, organizing, and making determinations about how they would handle each piece if they received it, and will show you where and how that process occurs in the home.
NHES Materials and Screener: In the second, you will hand the participant NHES materials that were mailed to them earlier this year and the participant will talk aloud about his or her reactions to and opinions about the mailings. Finally, you will show the participant the NHES screener questionnaire and talk through their opinions and reactions.
Throughout the interview, please incorporate the participant’s surroundings into your questioning as appropriate. Ultimately, you want to pull in objects or observations to discuss if you believe they may be relevant to why this household did not respond to the screener. For example, if you see an American flag on the desk you may want to ask something like: “Can you tell me a little bit about the American flag on your desk?” or “I see you have an American flag on your desk. What, in particular, is important to you about displaying the flag?” The household observation is part of the interview process. You will want to take note of their surroundings: what do they hang on their walls? Do you see visible mail piles? Are there children’s toys present? What types of books or magazines are displayed? At the conclusion of the interview, in your field notes, you will need to write down your observations and describe (using thick description as discussed in the training) the exterior of the home and the rooms to which you had access during interview.
INTERVIEW DOMAINS
Domain 1: Rapport Building and Household Make up
Objective: To understand key aspects of the participant’s household composition and build rapport.
Sample Questions
Tell me a little about who lives here with you.
I see that you have three kids. What types of activities do they enjoy?
What drew you to the neighborhood?
I noticed X (such as a security sign, holiday décor, etc.) outside your home. Can you tell me a little about it?
Domain 2: Experience with Mail Delivery
Objective: To understand how the household interacts with their mail. For example, how mail enters the participant’s household, what happens to it after it arrives, how participants decide what mail to keep/open/throw away, and how easy or difficult it is for the household to check or send their mail. Have the participant walk you through their mail receipt routine, asking questions about timing and activities along the way. During this exchange you will incorporate the activity below.
Sample Questions
Where do you get your mail (home delivery mailbox, home delivery mail slot, P.O. Box)? Can you show me? (if nearby)
What kinds of mail do you typically get?
How do you know when mail is delivered?
How often do you check your mail?
Where do you send your mail (walk to P.O. box, down a building mail chute, mailbox, etc.)? Can you show me? (if different from delivery)
How do you decide which pieces of mail to open?
How would you describe the kinds of mail you keep when you sort through unopened mail?
How about the kinds of mail that you throw away?
What makes you throw away a piece of mail before opening it?
What makes you throw away a piece of mail before reading it?
What do you do with your mail once you’ve read it?
How do you keep track of mail you need to send/respond to?
How often does mail for other addresses get mixed up with your mail?
Mail Interaction Activity
Hand the participant the stack of mail prepared for this interview. Tell the participant to imagine that this was what they received in today’s mail and that their name is Jaime Smith and that they live at 123 Main St Anytown USA. Ask them to talk out loud as they walk through what they would do with the stack. Participants should verbalize what they are doing at each step. Make sure the participant covers sorting, organizing, distributing among household members, opening, storing, saving, discarding, and responding to mail. It is important to try to get the participant to verbalize which aspects of each mailings are leading them to make various decisions about what to do with them so that we can apply these findings to the potential redesign of future NHES mailings. The focus is on the envelope/exterior of the mailing and not its contents (many of the envelopes are empty). The prepared stack should include one NHES mailing. Observe what the participant does with this envelope during the activity, but wait to probe about it until the sorting activity is complete. In other words, let the participant treat it like the other pieces of mail. You will only call attention to it at the start of the next activity.
Domain 3: Experience with NHES Mailings and Screener
Objective: To gather participants’ opinions and reactions to the survey packages that were mailed to them earlier this year. To the participant, this should be a seamless transition from that activity above. In practice, this is one activity that covers multiple domains.
NHES Materials Activity
Locate the NHES screener material from the stack of mail and ask them to open it and “process” it as they normally would. Observe how they open it. Once it is opened, ask participants their opinions and reactions to it. Please cover all aspects of the materials – the design, layout, color, wording, the return address, the mailing address, the stamp, the envelope size, etc. Probe on what would make participants more likely to open, read, and respond to the mailing.
Sample Questions
Do you remember receiving this? (Hand the respondent the mailings (see below for the list) one at a time in the order they would have received them. Allow them a moment to look at each mailing so that they can answer if they remember receiving it. After they say if they remember, follow the general line of questioning below).
Advance letter + envelope
Initial screener invitation letter + $5 cash + envelope
Screener pressure-sealed mailer/postcard
Second screener reminder letter + envelope
Third screener reminder letter + questionnaire + envelope
Fourth screener reminder letter + questionnaire + envelope
If the participant remembers: (Note to interviewers: Repeat this line of questioning for each of the NHES mailings that the participant remembers.)
What do you remember about receiving this mailing?
What did you do with this envelope when you received it earlier this year?
(If opened the mailing): What do you remember about opening the letter? What stood out to you?
(If didn’t open the mailing): What changes could we make that would make it more likely that you would open this mailing?
Why did you throw it away (or keep it)?
(If remembers a letter that mentions the web screener): Why did you (or did you not) access the web page listed in the letter?
(If accessed the web page): What do you remember about accessing the web page listed in the letter?
(If accessed the web page): What stopped you from filling out the survey? (Confidentiality concerns, technical issues, not interested in the topic?)
(If remembers a mailing that includes a paper questionnaire): What do you remember about the paper screener that you received?
What stopped you from filling it out?
What changes could we make that would make it more likely that you would respond to this survey?
If the participant doesn’t remember any of the materials:
Based on what you see here, do you think you would complete this survey if you got it in the mail? Why or why not?
What stands out to you as you look at it right now?
If you received this in the mail today, how likely would you be to open it?
What would make you open (or not open) it?
What changes could we make that would make it more likely that you would open these mailings?
What changes could we make that would make it more likely that you would respond to this survey?
Domain 4: Understanding of and Attitudes Toward Surveys
Objective: To gauge the participant’s level of knowledge of and comfort with surveys
Sample Questions
What do you think of when I use the word ‘survey’? What do you understand about what a survey is?
Have you ever done a survey before?
What are the benefits, if any, to participating in a survey? What, in your opinion, are the costs/risks/burdens to participating?
Tell me about the last time you completed a survey.
Do you enjoy completing surveys? What do you like/not like about doing them?
How do you like to take surveys? (probe on paper, computer, tablet, phone, in-person)
What are your experiences with getting money or gift cards for completing surveys?
Domain 5: Privacy Concerns
Objective: To understand how the participant views privacy and how this may affect their willingness to do surveys.
Sample Questions
What does the word “privacy” mean to you? Tell me the first few words that pop into your head.
How do you feel about information that is collected about you and how it is being used? How do you think it is used? Does it matter to you who is collecting the information? (Note: If the participant is unclear about the meaning of the question, you can ask about different settings. “For example, how do you feel about filling out forms at the doctor’s office? How do you feel about web sites collecting cookies?”)
How do you feel about the amount of control you have over your personal information?
What are some things you do to protect your personal information?
How do you feel about providing information on a survey about yourself or your family? Does it matter who is doing the survey? Does it matter what the question is asking? (If so): What kinds of questions make you uncomfortable?
Thinking about your daily life, when you have private information you would like to share with another trusted person or organization, how secure do you feel sharing it through the mail? Online? Via email? Over the phone? What makes you feel secure / insecure about information sharing?
For respondents with children:
What rules, if any, do you have about your child/children using the Internet?
What discussions have you had with your child/children about giving out information when s/he is/are online?
Domain 6: Attitudes toward government
Objective: To understand the participant’s attitudes toward the federal government and how this may affect their willingness to participate in government-sponsored surveys. (Try to avoid extended discussion of presidential and electoral politics).
Sample Questions
What role do you believe federal government agencies should play in collecting information about people through surveys? What information do they need?
Why does the U.S. continue to do a Census every ten years? What is that information for? How do you feel about being asked to complete a Census form?
What are your thoughts on the Department of Education? What role does it play in your life?
How important is voting to you?
How much impact do you think the federal government has on your daily life?
Domain 7: Education
Objective: To understand the participant’s opinions about education.
Sample Questions
What role do you think parents should take when it comes to their child’s schooling?
In general, what are your thoughts on today’s education system?
Are you happy with the school your child attends? Why/why not?
How involved are you with your child’s school?
How important/valuable was your own education experience to your daily life?
Domain 8: Time Use
Objective: To understand how people use their time day-to-day to get a sense of their time commitments and priorities.
Sample Questions
Can you briefly describe a typical weekday/weekend day?
What activities do you make sure you do every day?
How much time do different people in your family typically spend at home?
What do you do to relax?
Domain 9: Civic and Community Engagement
Objective: To explore the participant’s connection to the community, which has been shown to influence survey participation.
Sample Questions
Tell me about any news sources you follow.
To what extent to do you feel like an important part of your neighborhood or community?
What makes you feel like an important part? What makes you feel like not an important part?
What types of groups are you involved in?
What about other members of your family?
Domain 10: Household Information and Demographics
Objective: To understand the types of households and people with whom we are speaking and to determine if different subsets of nonrespondents have different types of concerns or reasons for nonresponse.
Lastly, I’ll ask you a few questions about your home and the people who live here. These are basic demographic questions like the kinds you are asked on the Census. [Note to interviewers: Please read all the questions in this section and ask them exactly as worded.]
If applicable: We may have already spoken about some of these topics, but I would just like to confirm this information now. If you know the participant has already shared this information with you, add “Just to confirm” before the question; otherwise, questions should be read exactly as shown.
Including yourself, how many people live here? _______________
[] Don’t know
[] Refused
(if only 1 person lives here, skip to question 2; otherwise, ask question 1a.)
How many people who live here are younger than 21 years old? _______________
[] Don’t know
[] Refused
What is your current age? _______________
[] Refused
What is your gender? (Do not read response options.)
[] Male
[] Female
[] Other, specify: ______________________
[] Refused
Are you of Hispanic, Latino, or Spanish origin?
[] Yes
[] No
[] Other, specify: ______________________
[] Refused
What is your race? (Do not read response options. Select all that apply.)
[] American Indian or Alaska Native
[] Asian
[] Black or African American
[] Native Hawaiian or other Pacific Islander
[] White
[] Refused
Which language is spoken most often by the adults who live here? (Do not read response options.)
[] English
[] Spanish
[] Other, specify: ______________________
[] Refused
Do you have Internet access at home…?
|
[] Yes |
[] No |
[] Don’t know |
[] Refused |
|
[] Yes |
[] No |
[] Don’t know |
[] Refused |
|
[] Yes |
[] No |
[] Don’t know |
[] Refused |
What is the highest grade or level of school that you have completed? (Do not read response options.)
[] 8th grade or less
[] High school, but no diploma
[] High school diploma or equivalent (GED)
[] Vocational diploma after high school
[] Some college, but no degree
[] Associate’s degree (AA, AS)
[] Bachelor’s degree (BA, BS)
[] Some graduate or professional education, but no degree
[] Master’s degree (MA, MS)
[] Doctorate degree (PhD, EdD)
[] Professional degree (MD, DDS, JD, LLB)
[] Refused
Are you currently employed and working for pay?
[] Yes
[] No
[] Refused
Are you a full-time or part-time student?
[] Yes
[] No
[] Refused
Based on this sheet [Hand respondent the income sheet], which category best fits the total income of all persons in your home over the past 12 months? Include your own income. Include money from jobs or other earnings, pensions, interest, rent, Social Security payments, and so on. (Do not read response options. If respondent is struggling, let them know they can make their best guess.)
[] $0 to $10,000
[] $10,001 to $20,000
[] $20,001 to $30,000
[] $30,001 to $40,000
[] $40,001 to $50,000
[] $50,001 to $60,000
[] $60,001 to $75,000
[] $75,001 to $100,000
[] $100,001 to $150,000
[] $150,001 to $200,000
[] $200,001 to $250,000
[] $250,001 or more
[] Don’t know
[] Refused
B.1.10. Income Show Card
A |
$0 to $10,000 |
|
B |
$10,001 to $20,000 |
|
C |
$20,001 to $30,000 |
|
D |
$30,001 to $40,000 |
|
E |
$40,001 to $50,000 |
|
F |
$50,001 to $60,000 |
|
G |
$60,001 to $75,000 |
|
H |
$75,001 to $100,000 |
|
I |
$100,001 to $150,000 |
|
J |
$150,001 to $200,000 |
|
K |
$200,001 to $250,000 |
|
L |
$250,001 or more |
|
B.1.11. Incentive Receipt Acknowledgment Form
I, ________________________ (Participant Name), certify that I have received the following incentive from American Institutes for Research as a thank you for participating in an interview.
Type |
Amount |
Cash |
$120 |
Participant Signature __________________________ Date ________________________
Interviewer Signature __________________________ Date ________________________
B.2. Spanish Materials
Attachment B.2. will provide the Spanish versions of the materials that will be used to interact with households sampled for the qualitative interview. These materials will be provided to OMB as a change request in November 2018 and will include:
Invitation Letter
First Reminder Postcard
Second Reminder Postcard
Phone Follow-up Recruitment Script
“Sorry We Missed You!” Card
In-Person Recruitment Script
Commonly Asked Questions
Interview Consent Form
Interview Protocol
Income Show Card
Incentive Receipt Acknowledgment Form
B.2.1. Spanish Invitation Letter
B.2.2. Spanish First Reminder Postcard
B.2.3. Spanish Second Reminder Postcard
B.2.4. Spanish Phone Follow-up Recruitment Script
B.2.5. Spanish “Sorry We Missed You!” Card
B.2.6. Spanish In-Person Recruitment Script
B.2.7. Spanish Commonly Asked Questions
B.2.8. Spanish Informed Consent Form
B.2.9. Spanish Interview Protocol
B.2.10. Spanish Income Show Card
B.2.11. Spanish Incentive Receipt Acknowledgment Form
B.3. Observation Instrument
Please complete the observation protocol below. Record all observations as unobtrusively as possible. If necessary, you may explain the purpose of your observations if you are asked by any individual in the neighborhood, including anyone from the sampled household.
DEFINITIONS
What do we mean by the sample unit OR the building within which the sample unit resides? Here we are making a distinction between a sample unit that we might call a single occupancy dwelling unit (a single residential structure that stands alone) and a multi-unit structure where the sample unit resides in a building that houses other units.
The BLOCK FACE is both sides of the street along one side of the block (the side in which the sample unit is located as indicated in the illustration below). The block face can be much more difficult to determine in rural areas. If the boundaries of a block face are not obvious, do the following: 1) Face the sample unit/building, 2) Look left and then right, and 3) Turn 180 degrees, look left and then right – that is the block face.
GENERAL INFORMATION
Date of observation: _______________
Time of day: ______:_______ AM / PM
[Take at least one photo of the address you are going to observe. Photos should be taken from the front of the address, if possible. Photos should only be taken from public spaces such as the street, and will not be taken while on someone’s private property.]
Are you able to observe the sample unit?
Yes [CONTINUE TO QUESTION 1]
No, why not? ___________________________________________________________________
___________________________________________________________________
[IF NO, END OBSERVATION; OTHERWISE, CONTINUE]
STRUCTURE TYPE
Select the option that best describes the sample unit:
Single occupancy dwelling unit (one household in a single residential structure that stands alone)
Duplex (two-household structures attached together, but with separate entrances)
Townhouse/Rowhouse (3 or more household structures attached together, but with separate entrances)
Low-rise apartment or condominium buildings (1-3 floors)
Mid-rise apartment or condominium buildings (4-6 floors)
High-rise apartment or condominium buildings (7+ floors)
Housing unit over a commercial storefront
Commercial business
Vacant lot
Other non-residential
Other: _________________________________________________________________
Unsure
Justification: This item is meant to assess whether the mail survey response process (e.g., whether the mail is delivered successfully) or response decision (e.g., whether to respond to the survey invitation) varies by the type of housing unit. Frame information is available for NHES sample members that indicates the housing type. Hence, based on this item, if we find that certain structure types are particularly prone to non-contact or non-response, the data collection process can be modified to mitigate the issues of non-contact (e.g., use a different courier, use larger print on envelopes for multi-unit structures that might be more prone to sorting error) or non-response (e.g., tailor the language of contact materials, use a different incentive structure). Additionally, data from this item will help assess whether the available housing type information on the frame is accurate. Assessing the accuracy of the frame will help increase the efficiency of NHES:2022 operations because frame information is increasingly used for developing adaptive design models. For instance, the housing type variable on the frame was a significant predictor of the model used to assign cases to the modeled survey mode condition in the NHES:2019 design.
HOUSEHOLD RESIDENTIAL OCCUPANCY STATUS
Based on your observation, indicate whether the household is an/a:
Currently occupied address (e.g., car in the driveway, toys/chairs/yard tools outside the house, lights on inside the home)
Seasonal address, no current occupants (e.g., resort area, no evidence of current occupants)
Permanently vacant address (e.g., windows/doors boarded up, condemned notice/unlivable conditions, currently under construction)
Not a housing unit (e.g. commercial, vacant lot)
Other: _________________________________________________________________
Unsure
Describe what aspects of the address made you decide on the specific occupancy status above.
______________________________________________________________________________________________________________________________________________________________________________________________
Justification: This item indicates the occupancy status of the address. First, it will assess the accuracy of the vacancy and seasonal occupancy status flags on the NHES frame. Second, collecting these data will help confirm the accuracy of the postmaster’s applied status codes (which are currently used to identify ineligible addresses) – particularly, if there are addresses that are vacant that are not being caught by the postmaster. This item also will help to assess the quality of the frame’s vacancy status flag which is used in adaptive design models (e.g., the NHES:2019 model predicting cost of FedEx mailings) and to evaluate the current practice of determining ineligibility. However, it is important to note that the occupancy status of an address may change between the time the sample (with the frame) was drawn and the interviewer’s observation of the address. The open-ended sub-item will help identify specific characteristics associated with household occupancy.
HOUSEHOLD MEMBER ELIGIBILITY FOR NHES TOPICALS
Based on your observation of the sample unit or building within which the sample unit resides, is there any indication that children 15 or under (including babies) may live in the sample unit (e.g., visible toys, bikes, basketball hoop, sports equipment, car seats, strollers, chalk, outdoor swing/play set, bumper sticker or car window sticker, child finder sticker for firefighters, etc.)?
Yes What? _________________________________________________________
No
Unsure What? _________________________________________________________
Justification: This item is mean to assess whether the information available on the frame about whether children live in the household is accurate for nonrespondents. It also will indicate whether drivers of nonresponse to NHES vary depending on whether addresses have household members that are eligible for the NHES topical survey. For addresses where an interview is completed, the collection of household demographic data will help confirm whether any household members are eligible. However, collecting this information for all households sampled for this study will allow us to determine whether other characteristics on the observation form, such as mail access type and neighborhood characteristics, differ based on NHES topical survey eligibility. For example, if the observations show more evidence of privacy concerns for households that have children living in them than for those that do not, efforts could be made in NHES:2022 to ensure that all contact materials (including topical materials) address the privacy concerns of such households.
MAIL ACCESS
Based on your observation, how does this household receive mail?
Mail slot
Mailbox attached to their home
Mailbox at the end of a driveway
Mailbox across the street/at the end of the road
Mailbox, slot, or room in the multi-unit building
No mailbox in view
Other: _________________________________________________________________
Unsure
Justification: This item helps us to understand the context within which the NHES materials would have been received. Understanding specifically how nonrespondent households receive mail will help determine whether any design changes are needed for the contact materials. For instance, mail received via a mailbox in a common area could be more easily mixed up with a neighbor’s mail or be less conspicuous, suggesting a potential benefit of using bolder envelope designs. Also, if certain atypical methods of receiving mail are common among nonrespondents, for such households, the use of additional courier services (such as UPS/FedEx) may be considered to improve the likelihood of the NHES materials being received, given that they provide door-to-door service. Moreover, this item also allows for assessment of the accuracy of NHES sample frame variables, such as the “P.O. Box only way to get mail” flag, which has been used exclusively in the past NHES administrations to decide which households should receive a FedEx mailing.
HOUSEHOLD INCOME
Relative to the general population and based on your observations, would you judge this sample unit to have a household income in any of the following: Use visual cues of the conditions of the house/building; number, make, and age of cars at the residence; and knowledge of the neighborhood/environment.
In the bottom third of the population
In the middle third of the population
In the top third of the population
Don’t Know/Unsure
Justification: This item attempts to assess the general socio-economic composition of the observed unit. Collecting this information will be useful because it will help assess whether the interaction of the household with the NHES materials varies by general household income. Additionally, it will also allow us to assess the accuracy of the frame information available about the household’s income (which is a significant predictor in models predicting response propensity). The qualitative interview protocol includes questions that ask about household income, however, including this item in this tool is useful because it collects this data for all households sampled for this study.
Are there any unique attributes or decorations (e.g., flags, wreaths, bumper stickers, signs) on or around the household that highlight what is important to this household… If yes, write them down.
-
Pride in education (e.g., honor roll bumper stickers, university/college flags or decor)
[] Yes [] No What? _______________
Community involvement (e.g., political candidate signs, indicators of being involved with charities, kids sports/clubs, neighborhood associations)
[] Yes [] No What? _______________
Patriotism (e.g., American flag, state flag)
[] Yes [] No What? _______________
Privacy (e.g., security camera, no loitering sign, household alarm sign, locked gate?)
[] Yes [] No What? _______________
Anything else?
[] Yes [] No What? _______________
Justification: These items attempt to assess the prevalence of characteristics that have been shown to mediate the likelihood of survey response, such as salience of the survey topic, privacy concerns, or civic engagement. If such a pattern exists, then block-level Census planning database variables that are also indicators of such characteristics that indicate neighborhood civic engagement could be appended to the NHES sample frame to modify the data collection process.
Identifying these features could inform modification of NHES contact materials. For example, if signs of pride in education are evident in nonresponding households, efforts could be made to further emphasize the impact of NHES on the nation’s education system in the contact materials. The final item allows for identifying any common characteristics of nonresponding households of which we may not have been aware.
Provide a general description of the unit, commenting only on things that have not already been noted in the questions above (e.g., building entry requirements, condition of address/neighborhood, objects around the address, visible people, other building types in the neighborhood): ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
1 The final base weighted NHES:2016 screener response rate was 66.4 percent. The Early Childhood Program Participation survey yielded a 73.4 percent topical response rate. The Parent and Family Involvement in Education survey yielded a 74.3 percent topical response rate. The Adult Training and Education Survey yielded a 73.1 percent topical response rate. These rates resulted in overall response rates (the product of the screener response rate and topical response rate) of 48.7 percent, 49.3 percent, and 48.5 percent, respectively. The increased use of the web survey mode in 2019 is expected to increase topical response rates.
2 P.O. Boxes that are included in the NHES:2019 sampling frame will be excluded from the nonrespondent sampling frame due to not having a street address associated with the P.O. Box to visit.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Megra, Mahlet (Mahi) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |
© 2024 OMB.report | Privacy Policy