Nonresponse Bias Procedures
Appendix F PIAAC 2017-Nonresponse Bias Procedures.docx
Program for the International Assessment of Adult Competencies (PIAAC) 2017 National Supplement
Nonresponse Bias Procedures
OMB: 1850-0870
Appendix
F
PIAAC Reducing Nonresponse Bias and
Preliminary Nonresponse Bias Analysis
This document was provided by the OECD and the PIAAC Consortium for initial use for the PIAAC 2012 Main Study. The guidelines described within were followed when conducting the nonresponse bias analyses for both the PIAAC 2012 Main Study and the PIAAC 2014 National Supplement and will be followed again for the PIAAC 2017 National Supplement.
PIAAC
Reducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis
Table of Contents
1. INTRODUCTION AND OVERVIEW 4
2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES 4
3. PIAAC DATA QUALITY GOAL REDUCE NRB BEFORE, DURING, AND AFTER DATA COLLECTION 5
3.1 Reduce NRB Before Data Collection 6
3.2 Reduce NRB During Data Collection 7
3.3 Reduce NRB After Data Collection 7
3.3.1 Basic Nonresponse Bias Analysis 9
3.3.2 NRB Analysis if Response Rate is Less Than 70% 9
List of Figures
Figure 1. Example response rates in West region, USA, during data collection 8
Figure 2. Preliminary plans for PIAAC NRB analysis 8
List of Tables
Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics: 2003 13
Table 3. Weighted response rates for the ALL background questionnaire, by response cell: 2003 15
Table 4. Multivariate analysis of response indicators: 2003 16
Table 6. Correlations of demographic variable with the outcome statistic 17
Table 7. Distribution of BQ incompletes, by disposition code category 18
Table 8. Household race for screener incompletes with NIR data, by region 19
Table 9. Differences between the last 10% and first 90% of respondents, by domain 20
PIAAC
Reducing NONRESPONSE BIAS AND
PRELIMINARY NONRESPONSE BIAS
ANALYSIS
MARCH 2010
1. INTRODUCTION AND OVERVIEW
This document addresses the issue of bias in PIAAC outcome statistics and contains an overview of the survey goals to reduce nonresponse bias (NRB) and preliminary plans for conducting NRB analysis. The Consortium will produce detailed guidelines for the NRB analysis after the field test is completed and comments are collected from countries on their lessons learned from the field test.
Section 2 provides a brief background on why reducing NRB is critical in producing high-quality estimates for PIAAC. Section 3 contains a summary of plans and procedures that are critical to reducing bias in the outcome statistics and preliminary plans to conduct the NRB analysis, including examples using data from similar literacy surveys.
The PIAAC Technical Standards and Guidelines (TSG) includes plans and procedures that should be put in place before, during, and after main study data collection to improve response rates and help reduce the potential for NRB. More specifically, the TSG require each country to:
Identify sources of NRB,
Reduce potential for NRB before and during data collection,
Evaluate NRB throughout the design and data collection process,
Evaluate NRB after data collection, and
Adjust for biases through nonresponse adjustment weighting steps.
Sections 3.1 and 3.2 contain brief summaries of procedures included in the TSG that countries should use to make plans to reduce nonresponse bias to the extent possible before and during data collection. At the conclusion of the data collection, countries are required to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustment to the sampling weights. This will help reduce the NRB to the maximum extent possible. Countries with low response rates (see below) are then asked to evaluate the potential for remaining bias using final adjusted weights.
The NRB analyses conducted prior to weighting are referred to as the basic analysis and are described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for each stage of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and if the overall response rate is less than 70%. Section 3.3.2 provides preliminary plans for conducting the extended NRB analysis, including some examples using similar competency surveys.
The focus of this document is on unit response rate (not item response rate). Refer to the TSG for details on the item response rate goals and analyses.
2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES
Probability sampling is essential for PIAAC for two main reasons. First, probability sampling contains a set of designs that leads to a variety of unbiased sampling approaches that allow analysts to generalise the results to the target population. Second, measures of precision related to survey estimates can be computed only under a probability design. The production of unbiased estimates assumes that every eligible adult in the target population would have a nonzero chance of selection (exclusion rate is equal to zero) in a national sample, and would be located, agree to participate in the study, and respond to all survey items (100% response rate). In practice, these circumstances are not realised in any survey population. Missing data occurs when; 1) subsets of the target population are excluded from the sampling frame, 2) some of the adults selected in the sample are not contacted or refuse to participate (referred to as unit nonresponse), 3) they fail to respond to a particular survey item (referred to as item nonresponse), or 4) because data collected from the sampled adults is contaminated (and thus not useful) or lost during or after the data collection phase. The statistical theory behind unbiased survey estimates and associated precision levels are not true anymore when data is missing for the units selected into the probability sample.
NRB (due to missing data for selected adults) can be substantial when two conditions hold: 1) the response rate is relatively low and 2) the difference between the competency levels of respondents and those of nonrespondents is relatively large. This is reflected in the following deterministic NRB formula:
,
where WR is the proportion of respondents, YR is the mean outcome for respondents, and YNR is the mean outcome for nonrespondents.
Similarly, undercoverage bias (due to exclusions) can be substantial if 1) undercoverage rate is high, and 2) the difference in competency levels between adults included in the sample and those excluded from the frame is relatively large. PIAAC TSG sets the maximum allowable exclusion rate at 5% to guard against high undercoverage bias in PIAAC estimates.
Given the relationships between bias and undercoverage and response rates, countries must keep the exclusion rates low and implement procedures to reduce the potential for NRB and attain high response rates.
An alternative model of nonresponse assumes each sampled person has a certain propensity to respond, and NRB in a characteristic is a function of the covariance between the response propensity and the characteristic:
,
where σyp is the covariance between the outcome variable and response propensity, and p is the mean response propensity. Based on this model, NRB is present if missingness is related to competency, as measured PIAAC.
It is well known that NRB can be reduced to some unknown extent through sample weighting if competency is correlated with auxiliary variables, and auxiliary variables are correlated with response propensity. Weighting assumes response probabilities are constant within every group created for weight adjustment, the competency score has zero variance within each group, and response propensity is uncorrelated with competency. It is known that these assumptions are not correct, and the impact of weight adjustments is limited to the number of variables (correlated with competency) available for nonresponse adjustment. Also, it is not possible to measure the exact departure from these assumptions since competency levels of nonrespondents are not known. It is, therefore, necessary to evaluate NRB when response rates are low.
3. PIAAC DATA QUALITY GOAL
REDUCE NRB BEFORE, DURING, AND AFTER
DATA COLLECTION
There are several ways to reduce the potential for NRB. First and foremost is to plan and implement field procedures that obtain a high level of cooperation. Section 3.1 provides a brief overview of steps that countries should take to plan and implement effective survey operation procedures to reduce the potential for bias.
Second, it is also critical to monitor the field when data collection is in progress and monitor the distribution of the sample during data collection to ensure steps are taken to reduce the potential for bias as much as possible. Section 3.2 contains a brief summary of these steps.
Finally, countries should search for auxiliary data that can be used to evaluate the potential for bias and that could be used to reduce the impact of nonresponse on survey estimates. Countries should evaluate the potential for remaining bias after weighting adjustments are completed if response rate falls below 70%. Section 3.3 describes the procedures that need to be followed after the data collection is completed.
3.1 Reduce NRB Before Data Collection
No single factor leads to obtaining high response rates in any survey, especially large-scale national surveys. As mentioned above, countries need to focus on a number of critical factors that are key to obtaining reliable data and highest response rates possible.
One of the most critical steps in improving response rates is to hire the most qualified staff for operating the survey. First, countries should attempt to hire interviewers with experience, good people skills, enthusiasm, perseverance, and language skills necessary to conduct PIAAC interviews. Countries should hire some senior interviewers to travel to areas with high nonresponse if necessary. In addition, countries should hire supervisors with experience and knowledge to mentor, motivate, and monitor interviewers. There should be a sufficient number of interviewers and supervisors to conduct the field activities, taking into account the data collection period.
A related critical aspect is establishing effective plans for training the staff. The goal is to ensure that all field staff understand PIAAC so well that they can focus on obtaining quality data and high response rates, not just focusing on the mechanics of the work. Countries should provide thorough initial training, including importance of PIAAC, presentation of study to respondents, and techniques for convincing reluctant respondents to participate. In addition, countries should plan to provide ongoing training via telephone and email. Countries should train supervisors on initial case assignment and reassignment strategies for potential/actual nonresponse.
Community/respondent outreach is another critical component in increasing response rates. Countries should plan to develop an outreach plan, design attractive/cohesive set of materials (e.g., letters, brochures, etc.). Countries should make sure to produce sufficient amount of each material. They should obtain media coverage: newspaper/journal articles about PIAAC that interviewers can show/give to respondents. Various endorsements should be obtained from well-known and well-respected authorities and agencies.
Interviewers/supervisors should be trained on effective use of materials. In addition, countries should design a website that could be used by respondents for reference and to legitimise the survey. Countries should establish toll-free telephone number/hotline—to answer questions and concerns raised by survey participants.
The timing of various activities is also very critical. Introductory materials should be mailed 7-10 days prior to respondent contact. Nonresponse conversion letters should be used for reluctant or hard-to-contact/reach respondents in a timely manner.
Each country should develop a non-interview report (NIR) form to collect information about nonrespondents (see Standard 10.3.7 in the TSG). The form can be paper and pencil or automated, and it should contain, at minimum, demographic data on the non-interviewed person, the strength of the refusal (if applicable), problems encountered, comments, and the likelihood of conversion. The NIR can aid in future interview attempts and possibly provide information for a NRB analysis.
The National Sampling Managers (NSMs) should develop plans for monitoring the sample during data collection to ensure the resulting sample has the lowest NRB attainable. NSMs should monitor the sample and increase response rates for groups that have potential to increase NRB (i.e. identify pockets of nonresponse and use methods to bring in initial nonrespondents that are not like respondents [see Section 3.2, Figure 1]). NSMs are encouraged to use the field test process to develop and test their monitoring programs for the main survey.
A critical task that should be completed prior to the start of data collection is identifying and collecting correlated auxiliary variables that are known with a high degree of accuracy for both respondents and nonrespondents. Evaluating and reducing NRB requires auxiliary variables, available for all sampled units, that are correlated with competency and response propensity. The correlation with competency could be measured using past adult competency or similar studies; variables hypothesised to be correlated with literacy should also be included. A segmentation analysis (see section 3.2) could be used to determine correlation with response propensity. NSMs are encouraged to use the field test data to help identify a set of candidate auxiliary variables for evaluating and reducing NRB. More discussion can be found in the sampling Plan Part II forms, specifically form W-1.
The Cross-Cultural Survey Guidelines website (http://ccsg.isr.umich.edu/datacoll.cfm) has additional suggestions for improving response rates and reducing NRB.
3.2 Reduce NRB During Data Collection
National Project Managers (NPMs) should promote close and timely supervision/monitoring and conduct scheduled weekly telephone conference calls with interviewers. They should keep the pressure on performance and allow for timely feedback. Countries should use Consortium forms or country-specific automated reports that allow for regular/frequent monitoring of case completion. Reports must be based on daily interviewer transmission of interview/assessment data and current contact attempt and disposition code information.
It is important that countries seek and be able to implement new ideas or experiences used successfully by other survey organisations/countries prior to and during data collection. It is always helpful to encourage interviewers/supervisors to make suggestions for improving response rates throughout the data collection period.
Countries should use OTRS to ask for ad hoc advice and use quality control monitoring calls to discuss national issues, concerns, and solutions dealing with respondent cooperation. Countries are encouraged to share with the Consortium novel approaches for obtaining respondent cooperation that can be shared with other countries.
The NSMs should use the planned monitoring procedures (as described in Section 3.1) to monitor response rates and sample yield according to key demographics, known to be associated with competency. This will help countries identify potential shortfalls for specific subgroups that could result in biased estimates. The goal is to reach subgroups or areas with low response and use various approaches to improve response rates for these subgroups (areas). The first step is to identify pockets of potential NRB through modelling, for example, segmentation modelling with local area-level auxiliary variables (smallest geography for which reliable auxiliary data is available, or use registry data when available). A typical segmentation modelling approach involves a classification tree algorithm that uses a chi-square test of independence to identify auxiliary variables that best define subgroups with differential response rates. The subgroups with low response rates, or areas containing these subgroups, can then be targeted for followup efforts to address the potential for NRB.
Figure 1 shows an example of a classification tree used for the West region during a U.S. adult competency household survey data collection period.
From Figure 1, it is evident that during data collection in the West there was a low response rate (50%) in segments with a high minority concentration, a high percentage of the population not speaking English well, located in metropolitan statistical areas (MSAs), and having a low percentage of the population with less than a ninth grade education. To the extent that these variables were related to literacy, the low response rate in these areas indicated potential for NRB.
3.3 Reduce NRB After Data Collection
As mentioned in Section 1, all countries are required to conduct the basic NRB analysis described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for all stages of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and for the overall sample if the overall response rate is less than 70%. Figure 2 shows the overall plan for PIAAC NRB analysis. The remainder of this section describes the preliminary plans for various analyses mentioned in the figure.
Figure 1. Example response rates in West region, USA, during data collection
Figure 2. Preliminary plans for PIAAC NRB analysis
3.3.1 Basic NRB Analysis
The main goal is to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustments to the weights. The basic NRB analysis uses auxiliary variable estimates for survey respondents and nonrespondents. The auxiliary variables may be available on the sampling frame or from a previous data collection stage (e.g. screener data for the BQ/JRA analysis), can come from an external source that can be matched to each sampled unit, or could be observational data on respondents and nonrespondents collected during data collection, assuming the data is of sufficient quality. The auxiliary variables must be available for all eligible units and should be related to competency.
An advantage of the basic analysis is that they could be performed using data that are readily available for both respondents and nonrespondents. However, they are only beneficial if auxiliary variables are highly correlated with competency.
The following techniques can be used:
Use a chi-square test to compare the distribution of auxiliary variables (correlated with competency) for respondents and nonrespondents. Deviations between the auxiliary survey estimates and higher quality external source estimates may indicate a potential for bias in the survey estimates of competency levels, to the extent that the auxiliary variables are related to the key statistics;
Compare response rates for different subgroups;
Use a classification tree algorithm to identify subgroups with low response rates; and/or
Use logistic regression to model the relationship between response status and the auxiliary variables.
Sample Weighting Adjustments
At the conclusion of the basic NRB analysis, countries will be asked to put together a list of variables found to be significantly related to competency and response status. The goal is to use these variables in developing NRB adjustments. Such adjustments will reduce NRB to the extent that the auxiliary variables and the competency measures are correlated, and the auxiliary variables and response propensity are correlated.
3.3.2 NRB Analysis if Response Rate is Less Than 70%
The basic descriptive analysis is a good initial assessment of NRB and is essential in identifying effective weighting variables. However, it has its limitations. The analysis does not reflect the effect of weighting adjustments on NRB, and the extent of bias remaining after nonresponse adjustments are conducted.
Brief descriptions of some possible analyses are provided below. Note that multiple analyses to assess NRB are necessary because each analysis has its own limitations. Together, they will provide an insight into the patterns and potential for bias.
Comparison of estimates before and after weight adjustments are done - The basic analysis described in section 3.3.1 comparing the base-weighted estimates for respondents to the base-weighted full sample estimates can be extended to include a third stage – the respondent estimates using weights adjusted for nonresponse and calibrated to control totals. In other words, estimates from the full sample can be compared to estimates from the respondents before and after weighting adjustments. If the full sample estimates are closer to the final estimates than the base weighted estimates, this indicates bias in the auxiliary variables was reduced through the weighting process.
Comparison of weighted estimates to external totals - Another extended analysis is to compare estimates from PIAAC to estimates from external source. The PIAAC estimates should be produced using the final weights that have been adjusted for nonresponse and calibrated to control totals. Care should be taken to choose external source estimates that measure the same characteristic for a similar time period. The external source estimates will also be subject to error, and the variance of these estimates should be taken into account when making this comparison.
Correlations between weighting adjustment variables and competency measures - The analyses described thus far rely on auxiliary variables and do not directly measure bias in the competency estimates. As mentioned earlier, bias in the auxiliary variables is indicative of bias in the competency estimates to the extent that the auxiliary variables and competency estimates are correlated. Thus, correlations between the auxiliary variables and competency data can be computed to evaluate this relationship. For variables used in the weighting adjustments, a low correlation with literacy implies that using the variable in the weighting adjustments did little to reduce NRB. On the other hand, a high correlation with literacy implies a potentially high reduction in NRB. For variables not used in the weighting adjustments, a high correlation with literacy indicates potential bias in the literacy estimates. This can be used to inform weighting decisions for future surveys.
The disadvantage of using correlations to evaluate NRB is that the correlations are based on respondents only and the relationship between competency and the auxiliary variables might be different for nonrespondents. However, this is not as much a concern if the relationship can be confirmed using outside sources.
Comparison of estimates using alternative weighting adjustments - For this evaluation, an auxiliary variable is re-calibrated to known totals, and estimates of the key statistics are compared before and after the re-weighting. Re-weighting can be useful as an evaluation tool when:
The variable was not used in weighting (because it was of a low quality);
The variable is correlated with the outcome measure; and
The variable is correlated with response propensity.
Any differences between estimates using the official survey weights and the re-weighted weights reflect noncoverage as well as NRB, but if there is not a large change in the estimates, this is further confirmation that NRB may not be a concern.
Analysis of variables collected during data collection - BQ variables - The descriptive analyses described earlier assess the potential for bias in some statistics that may not necessarily be highly related to competency. Therefore, the next step of the analysis should be to evaluate bias on the key statistics or closely related variables. For PIAAC, the key statistics are scores measuring competency in the components of the assessments that will be available later. However, before literacy scores are available, an analysis can be performed with BQ/JRA variables believed to be closely related to literacy, such as questions on measuring habits (newspapers or magazines), for example. Analyses of such variables may show patterns of bias that cannot be effectively eliminated by the auxiliary variables used in weighting.
Analysis of variables collected during data collection - Disposition codes - Disposition codes contain information on reasons for nonresponse. For this analysis, distributions of sampled persons with known characteristics related to outcome (i.e. the language barrier cases) should be examined. For example, the demographic distribution of language barrier cases can be compared to other eligible persons using auxiliary data, and interview data.
Analysis of variables collected during data collection - NIR - The NIR forms identify observable demographic information and reasons for nonresponse that are not captured in the disposition codes. The NIR forms can potentially indicate whether the reasons for nonresponse are related to competency estimates and suggest ways to improve response rates for future surveys.
Comparison of late or “hard-to-contact” respondents to early respondents - This analysis is helpful for evaluating the potential for NRB for differences that cannot be captured through adjusting for known demographics. Significant differences between the competency levels of early and late respondents imply that the bias could have been potentially reduced through focused efforts to obtain the late respondents. However, to the extent that differences between early and late respondents reflect differences between respondents and nonrespondents, the findings indicate that some level of NRB might still be present (which depends on the magnitude of nonresponse rate). The validity of this assumption, that late respondents are similar to nonrespondents, depends largely on the strategies used in the late data collection effort. For many surveys this may not be a valid assumption, but it seems reasonable for a household survey such as PIAAC where multiple contacts are usually required to gain cooperation.
Calculation of the range of potential bias - How can we evaluate the potential for bias remaining after weighting if nonrespondents are very different from respondents within the weighting classes? The range of potential bias can be evaluated using a deterministic NRB formula that calculates how different respondent and nonrespondents competency estimates need to be to bias the overall competency estimate by varying degrees. For example, one can show that for a response rate of 70% (WR = 0.7), the mean of respondents and nonrespondents would have to differ by 50 points (YR -YNR = 50) to bias the overall estimate by 15 points (Bias( yR) = 15).
Other analysis under review - The Consortium is currently reviewing other NRB analysis procedures, such as the R indicator (Schouten, Cobben, and Bethlehem, 20091), for possible inclusion in the NRB plan for PIAAC.
The following provides a summary of the steps included in the NRB analysis when response rate falls below 70%.
Step 1 of 3: Identify Variables
1.1 Identify auxiliary variables from the frame or external sources.
1.2 Identify informative BQ variables correlated with competency.
1.3 Identify other informative variables collected during data collection.
Step 2 of 3: Basic NRB Analysis
2.1 Conduct basic descriptive and model-based analysis.
2.2 Finalise variables selected for weighting adjustments.
2.3 Conduct nonresponse adjustments and compute final sampling weights.
Step 3 of 3: Extended NRB Analysis
3.1 Conduct extended descriptive and model-based analysis using auxiliary data.
3.2 Conduct analysis of informative BQ variables.
3.3 Conduct analysis of disposition codes.
3.4 Conduct analysis of NIR (information not available in disposition codes).
Example: The following information and tables are summaries extracted from the NRB analysis conducted for the Adult Literacy and Lifeskills Survey (ALL) 2003: U.S. NRB Analysis http://nces.ed.gov/pubs2009/2009063.pdf, and other unpublished NRB analyses conducted for similar adult competency household surveys.
The steps carried out in the following NRB analysis are summarized below.
Step 1 of 3: Identify variables from the frame or external sources
Step 1.1 The following variables were extracted from Census 2000 county-level data, and Census 2000 block group level data.
Census variables available for local areas (where respondents and nonrespondents resided)
Urban/rural status
Majority own/rent
Average household size
Percentage of persons 25 years or older with less than a high school diploma
Percentage of persons speaking English not well or not at all
Percentage below 150% poverty
Step 1.2 The following variables were extracted from the BQ.
Percentage born in the United States
Percentage with high school diploma
Percentage never receiving remedial help in school
Percentage agreeing with the statement: I enjoyed math in school
Percentage speaking English most often at home
Percentage employed
Percentage reading letters, memos, or emails at least once a week as part of main job
Step 1.3 Disposition codes and NIR variables were collected after the end of data collection.
Step 2 of 3: Basic NRB analysis – Tables 1 through 4 provide examples of analysis conducted for ALL.
Table 1 shows the results of chi-square tests conducted for ALL. The chi-square tests indicated a significant relationship of response status to region, MSA status, the percentage below 150% of poverty, age, sex, and race/ethnicity. These variables were included in the nonresponse adjustments to reduce the potential NRB to the extent that these auxiliary variables were related to literacy.
Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by key characteristics: 2003
† Not applicable.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.
Table 2 contains estimates of bias and relative bias for the subgroup percentages presented in Table 1. Bias was calculated using the deterministic NRB formula in section 2. A t test was performed to determine whether the NRB was significantly different from zero. Another measure, the relative bias, was computed as the bias divided by the estimate from the eligible sample. The relative bias is a measure of the size of the bias compared to the eligible sample estimate.
The results of the analysis in Table 2 were consistent with the chi-square analysis in Table 1. It also showed that even though there were statistically significant results as described above, the relative magnitude of the differences was small. The relative bias estimates were all less than 10 percent, except in the Northeast, which was 15 percent.
Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics
2003 Analysis variable |
Respondentspercent |
Eligiblespercent |
Bias |
Percentrelative bias |
||
Estimate |
Standarderror |
p value |
||||
Majority own/rent |
|
|
|
|
|
|
Rent |
25.7 |
25.2 |
0.5 |
0.44 |
0.298 |
2.0 |
Own |
74.3 |
74.8 |
-0.5 |
0.44 |
0.298 |
-0.7 |
Average household size |
|
|
|
|
|
|
less than or equal to2.2 |
18.1 |
18.6 |
-0.5 |
0.39 |
0.215 |
-2.7 |
greater than 2.2 and less than or equal to 2.8 |
52.9 |
51.5 |
1.4 |
0.65 |
0.044 |
2.7 |
greater than 2.8 |
29.1 |
29.9 |
-0.9 |
0.6 |
0.155 |
-3.0 |
Percent less than high school |
|
|
|
|
|
|
less than or equal to10% |
33.1 |
33.4 |
-0.3 |
0.53 |
0.540 |
-0.9 |
greater than 10% and less than or equal to 20% |
30.9 |
31.4 |
-0.5 |
0.48 |
0.284 |
-1.6 |
greater than 20% |
36.1 |
35.2 |
0.9 |
0.52 |
0.112 |
2.6 |
Percent limited English proficient |
|
|
|
|
|
|
0% |
29.9 |
29.9 |
# |
0.56 |
0.977 |
# |
greater than 0% and less than 2% |
34.7 |
34.6 |
0.1 |
0.69 |
0.841 |
0.3 |
greater than or equal to 2% |
35.4 |
35.5 |
-0.1 |
0.55 |
0.826 |
-0.3 |
Percent below 150% of poverty |
|
|
|
|
|
|
less than or equal to10% |
30.7 |
31.5 |
-0.8 |
0.52 |
0.129 |
-2.5 |
greater than 10% and less than or equal to 30% |
44.9 |
45.4 |
-0.5 |
0.58 |
0.379 |
-1.1 |
greater than 30% |
24.4 |
23.1 |
1.3 |
0.38 |
0.001 |
5.6 |
Age |
|
|
|
|
|
|
16–25 |
23.7 |
22.4 |
1.3 |
0.35 |
0.001 |
5.8 |
26–35 |
21.2 |
20.8 |
0.4 |
0.25 |
0.106 |
1.9 |
36–45 |
22.2 |
22.9 |
-0.6 |
0.36 |
0.088 |
-2.6 |
46–65 |
32.9 |
33.9 |
-1 |
0.41 |
0.016 |
-2.9 |
Sex |
|
|
|
|
|
|
Male |
48.1 |
49.6 |
-1.5 |
0.31 |
0.000 |
-3.0 |
Female |
51.9 |
50.4 |
1.5 |
0.31 |
0.000 |
3.0 |
Race/ethnicity2 |
|
|
|
|
|
|
Hispanic |
14.3 |
13.3 |
0.9 |
0.29 |
0.003 |
6.8 |
Black, non-Hispanic |
11.1 |
10.2 |
0.9 |
0.31 |
0.009 |
8.8 |
Other, non-Hispanic |
74.6 |
76.4 |
-1.8 |
0.32 |
0.000 |
-2.4 |
# Rounds to zero
1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.
2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.
The analyses in Tables 1 and 2 were useful in explaining the relationship of response status to each auxiliary variable individually. A classification tree algorithm and a logistic regression model were used to evaluate the relationship between response status and multiple auxiliary variables. The results of the classification tree algorithm are shown in Table 3. The algorithm was similar to that described in section 2 and used chi-square tests to divide the sample into subgroups that best explain differential response rates. As Table 3 shows, 22 cells were formed with weighted response rates ranging from 58.7% to 97.4%. The subgroups defined by the classification tree algorithm could be used as nonresponse adjustment cells in the weighting adjustment.
Table 3. Weighted response rates for the ALL background questionnaire, by response cell: 2003
NOTE: The response cells were formed using the Chi-square Automated Interaction Detector.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.
Logistic regression models are also useful in identifying significant effects on response propensity. Response status was used as the binary dependent variable and the set of auxiliary variables in Table 1 as the predictors. An F test was performed for each auxiliary variable to determine if it was significantly related to response propensity, after accounting for the other auxiliary variables. Results are shown in Table 4. Average household size, age, race/ethnicity, sex, and region are shown to be potentially valuable variables to include in the nonresponse adjustments. While this analysis was limited to main effects, interaction terms can also be included.
Table 4. Multivariate analysis of response indicators: 2003
Test |
F statistic |
Degrees of freedom |
p value |
|
Numerator |
Denominator |
|||
Overall fit |
3.89 |
21 |
10 |
0.016 |
Average household size |
6.77 |
2 |
29 |
0.004 |
Majority own/rent |
0.32 |
1 |
30 |
0.578 |
Percent less than high school |
0.44 |
2 |
29 |
0.647 |
Percent limited English proficient |
2.08 |
2 |
29 |
0.143 |
Locale1 |
0.12 |
2 |
29 |
0.887 |
Age |
4.47 |
3 |
28 |
0.011 |
Race/ethnicity2 |
10.79 |
2 |
29 |
0.000 |
Sex |
19.43 |
1 |
30 |
0.000 |
Metropolitan Statistical Area status |
0.03 |
1 |
30 |
0.871 |
Percent below 150% of poverty |
3.22 |
2 |
29 |
0.055 |
Region |
6.65 |
3 |
28 |
0.002 |
1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.
2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.
Step 3 of 3: Extended analysis – Tables 5 through 10 provide examples of extended NRB analysis as conducted for ALL and other adult competency surveys
Table 5 shows an example of the comparison of estimates before and after weighting adjustments. The following t test comparisons were made:
Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with the BQ base weights for the BQ respondents only to check for differences due to nonresponse to the BQ
Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with that from the BQ nonresponse adjusted weights for respondents to check for differences even after the nonresponse adjustment process to the BQ
Comparison of percentage distributions from BQ nonresponse adjusted weights for respondents with that from the BQ raked weights for respondents to check for differences that may have been introduced through the initial raking procedure
This analysis showed that, through the nonresponse adjustment, the potential for bias was reduced (p-values moved from significant to nonsignificant) in all domains shown below, with the exception of two domains relating to average household size. These two domains show significant bias created by the nonresponse adjustments. While the differences in the percentage distributions of these domains remain statistically significant even after the raking adjustment, they are not large enough to be meaningful.
Table 5. Percentage distribution of sample cases at each weighting step for the ALL background questionnaire, by key characteristics: 2003
Total sample % |
BQ respondents |
Nonresponse adjustment |
Initial raking |
||||
% |
p value1 |
% |
p value1 |
% |
p value2 |
||
Total |
100.0 |
100.0 |
† |
100.0 |
† |
100.0 |
† |
Region |
|
|
|
|
|
|
|
Northeast |
18.4 |
15.6 |
0.000 |
18.6 |
0.355 |
19.0 |
0.866 |
Midwest |
22.1 |
22.9 |
0.180 |
21.9 |
0.355 |
23.0 |
0.728 |
South |
34.9 |
35.0 |
0.842 |
34.9 |
0.825 |
35.0 |
0.716 |
West |
24.7 |
26.6 |
0.001 |
24.7 |
0.895 |
23.0 |
0.413 |
Metropolitan Statistical Area status |
|
|
|
|
|
|
|
Non-Metropolitan Statistical Area |
21.8 |
22.7 |
0.012 |
21.9 |
0.558 |
18.0 |
0.100 |
Metropolitan Statistical Area |
78.2 |
77.3 |
0.012 |
78.1 |
0.558 |
82.0 |
0.100 |
Locale3 |
|
|
|
|
|
|
|
Urban |
65.3 |
64.6 |
0.080 |
65.1 |
0.448 |
68.0 |
0.097 |
Suburban |
12.4 |
13.0 |
0.103 |
12.4 |
0.849 |
11.0 |
0.198 |
Rural |
22.3 |
22.4 |
0.749 |
22.5 |
0.566 |
20.0 |
0.075 |
Majority own/rent |
|
|
|
|
|
|
|
Rent |
25.2 |
25.7 |
0.294 |
25.0 |
0.486 |
26.0 |
0.254 |
Own |
74.8 |
74.3 |
0.294 |
75.0 |
0.486 |
74.0 |
0.254 |
Average household size |
|
|
|
|
|
|
|
less than or equal to 2.2 |
18.6 |
18.1 |
0.225 |
18.1 |
0.213 |
18.1 |
0.946 |
greater than 2.2 and less than or equal to 2.8 |
51.5 |
52.9 |
0.051 |
53.3 |
0.003 |
52.5 |
0.257 |
greater than 2.8 |
30.0 |
29.1 |
0.157 |
28.7 |
0.005 |
29.4 |
0.381 |
Percent less than high school |
|
|
|
|
|
|
|
less than or equal to 10% |
33.4 |
33.1 |
0.542 |
33.1 |
0.569 |
32.7 |
0.431 |
greater than 10% and less than or equal to 20% |
31.4 |
30.9 |
0.292 |
31.4 |
0.887 |
31.3 |
0.937 |
greater than 20% |
35.2 |
36.1 |
0.123 |
35.5 |
0.349 |
36.0 |
0.524 |
Percent limited English proficient |
|
|
|
|
|
|
|
0% |
29.9 |
29.9 |
0.977 |
30.0 |
0.953 |
28.8 |
0.088 |
greater than 0% and less than 2% |
34.6 |
34.7 |
0.840 |
35.4 |
0.162 |
35.5 |
0.756 |
greater than or equal to 2% |
35.5 |
35.4 |
0.825 |
34.7 |
0.052 |
35.8 |
0.314 |
Percent below 150% of poverty |
|
|
|
|
|
|
|
less than or equal to 10% |
31.5 |
30.7 |
0.137 |
31.6 |
0.912 |
32.0 |
0.629 |
greater than 10% and less than or equal to 30% |
45.4 |
44.9 |
0.372 |
45.3 |
0.682 |
45.4 |
0.814 |
greater than 30% |
23.1 |
24.4 |
0.002 |
23.2 |
0.452 |
22.7 |
0.585 |
1 The p value and SE are for tests of difference between estimated percentage of current step and total sample (BQ base weights).
2 The p value and SE are for tests of difference between estimated percentage of current and previous weighting steps.
3 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.
SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.
Table 6 gives an example of correlations between auxiliary variables and competency score measures for an adult competency survey. The percentage aged 25+ with less than a high school education was found to be highly correlated with literacy score. If this variable was also related to response propensity, using it in nonresponse adjustments should reduce the potential NRB of literacy estimates.
Table 6. Correlations of demographic variable with the outcome statistic
Variable |
Absolute correlation with outcome statistic |
||||
<0.1 |
0.1-0.2 |
0.2-0.3 |
0.3-0.4 |
>0.4 |
|
Percent aged 25+ with less than high school education |
|
|
|
|
X |
Percent of adults speaking Spanish at home and English not well or not at all |
|
|
|
X |
|
Percent below 150% of poverty |
|
|
|
X |
|
Median Income |
|
|
X |
|
|
Percent who rent |
|
|
X |
|
|
Average household size |
|
X |
|
|
|
Age |
|
X |
|
|
|
Percent minority |
X |
|
|
|
|
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
Table 7 provides an example of the analysis of disposition codes to evaluate NRB in an adult competency survey. The distribution of cases with a BQ disposition code of language problem or mental disability was compared to the distribution of the other nonrespondents. Chi-square tests were processed to determine if there was a relationship between select demographic variables and the disposition code groups (literacy-related or other) for incompletes. Significant differences in the distribution by region, MSA status, and age were found between the two groups. These variables were used in the weighting adjustments, which are expected to reduce the potential bias from literacy-related nonresponse.
Table 7. Distribution of BQ incompletes, by disposition code category
Domain |
Literacy-related2 |
other |
|||
Total incompletes |
355 |
5,191 |
|||
Domain |
Incompletes1 |
Statistic |
p value |
||
Literacy- related2 percent |
OTHER percent |
||||
Total incompletes |
100 |
100 |
|
|
|
Region |
|
|
10 |
0.006 |
|
Northeast |
16 |
17 |
|
|
|
Midwest |
18 |
22 |
|
|
|
South |
21 |
38 |
|
|
|
West |
45 |
23 |
|
|
|
MSA status |
|
|
9 |
0.002 |
|
Non-MSA |
9 |
18 |
|
|
|
MSA |
91 |
82 |
|
|
|
Age |
|
|
23 |
0.000 |
|
16-29 |
16 |
20 |
|
|
|
30-49 |
37 |
43 |
|
|
|
50-69 |
23 |
27 |
|
|
|
70+ |
24 |
11 |
|
|
|
Gender |
|
|
3 |
0.067 |
|
Male |
47 |
53 |
|
|
|
Female |
53 |
47 |
|
|
|
Race/ethnicity |
|
|
2 |
0.244 |
|
Hispanic |
17 |
11 |
|
|
|
NH Black only |
6 |
9 |
|
|
|
Other |
77 |
79 |
|
|
|
† Not applicable
1 Nonrespondents
2 Literacy-related reasons for nonresponse at the BQ stage consist of language problems and mental disabilities only.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
An example of an analysis of NIR data for an adult competency survey is provided in Table 8. The table summarizes data collected on the NIR regarding the household race of screener incompletes, as observed or gathered from neighbour information. The base-weighted distribution is shown overall and by region. For households where race was determined, Hispanic households were less common among screener incompletes compared to the population proportion. This suggests a possible increase in response rates from accommodations made for Spanish speakers, such as bilingual interviewers.
Table 8. Household race for screener incompletes with NIR data, by region
SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
Table 9 shows an example of a comparison of literacy estimates for late versus early respondents. Calculations were done using marginal maximum likelihood (MML) means. The results indicate a small but significant bias in Statistic 2 if focused followup efforts (as describe in section 2) had not been successful in obtaining responses from respondents at the end of the data collection period (i.e. if data about respondents at the end of the data collection period had been missing). This suggests some small level of NRB may be present to the extent that nonrespondents are similar to respondents at the end of the data collection period.
Table 9. Differences between the last 10% and first 90% of respondents, by domain
† Not applicable
SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
An example of the calculation of the range of potential bias in literacy estimates is shown in Table 10. The table shows the difference between respondent and nonrespondents literacy scores that would bias the population estimate by b = 3, 5, 10, or 15 points. The required difference between respondents and nonrespondents was computed as described earlier in section 3.3.2. For the overall sample, literacy scores of respondents and nonrespondents would have to differ by about 8 points on average to bias the national estimate by 3 points and would need to differ by 39 points to bias the national estimate by 15 points.
Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics
Domain |
Percent of population1 represented by nonrespondents2 |
Average point difference between respondents and nonrespondents needed to bias estimated scores by |
|||
3 |
5 |
10 |
15 |
||
Total |
38 |
8 |
13 |
26 |
39 |
Region |
|
|
|
|
|
Northeast |
43 |
7 |
12 |
23 |
35 |
Midwest |
35 |
9 |
14 |
28 |
43 |
South |
38 |
8 |
13 |
26 |
40 |
West |
38 |
8 |
13 |
26 |
39 |
MSA status |
|
|
|
|
|
Non-MSA |
33 |
9 |
15 |
30 |
45 |
MSA |
40 |
8 |
13 |
25 |
38 |
Average household size |
|
|
|
|
|
≤ 2.42 |
39 |
8 |
13 |
26 |
39 |
(2.42, 2.8] |
39 |
8 |
13 |
26 |
39 |
> 2.8 |
38 |
8 |
13 |
27 |
40 |
Percent less than high school |
|
|
|
|
|
≤ 10.4% |
43 |
7 |
12 |
23 |
35 |
(10.4%, 20.3%] |
39 |
8 |
13 |
26 |
38 |
(20.3%, 32%] |
35 |
8 |
14 |
28 |
42 |
> 32% |
31 |
10 |
16 |
32 |
48 |
Percent speaking Spanish but not English |
|
|
|
|
|
0% |
39 |
8 |
13 |
26 |
38 |
(0%, 28%] |
40 |
8 |
13 |
25 |
38 |
> 28% |
34 |
9 |
15 |
29 |
44 |
Percent below 150% of poverty |
|
|
|
|
|
≤ 10.7% |
43 |
7 |
12 |
23 |
35 |
(10.7%, 20%] |
40 |
8 |
13 |
25 |
38 |
(20%, 33.3%] |
35 |
9 |
14 |
29 |
43 |
> 33.3% |
29 |
10 |
17 |
34 |
52 |
Median income |
|
|
|
|
|
≤ 28,400 |
28 |
11 |
18 |
36 |
53 |
(28,400, 37,850] |
34 |
9 |
15 |
29 |
44 |
(37,850, 52,100] |
40 |
8 |
13 |
25 |
38 |
> 52,100 |
44 |
7 |
11 |
23 |
34 |
Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics (Continued)
Domain |
Percent of population1 represented by nonrespondents2 |
Average point difference between respondents and nonrespondents needed to bias estimated scores by |
|||
3 |
5 |
10 |
15 |
||
Percent rent |
|
|
|
|
|
≤ 16% |
42 |
7 |
12 |
24 |
36 |
(16%, 31%] |
37 |
8 |
13 |
27 |
40 |
(31%, 59%] |
37 |
8 |
14 |
27 |
41 |
> 59% |
35 |
9 |
14 |
29 |
43 |
Low response rate CHAID cells |
|
|
|
|
|
Cell 13 |
59 |
5 |
8 |
17 |
25 |
Cell 24 |
51 |
6 |
10 |
20 |
30 |
Cell 35 |
50 |
6 |
10 |
20 |
30 |
1 The population excludes language problems and mental disabilities.
2 Nonrespondents to the Screener and BQ.
3 Median income greater than 52,100, average household size 2.42 or less, Northeast region
4 Median income 37,850 - 52,100, percent less than high school greater than 20.3, Northeast region
5 Median income greater than 52,100, average household size 2.42 - 2.8, South or West region
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
4. Summary
This document presented the main goals of PIAAC for reducing NRB throughout the survey process. A series of procedures for implementation before and during the data collection were described to help increase the response rates with the goal of reducing NRB to the extent possible. Preliminary plans are provided to conduct basic NRB analysis to allow an effective weighting adjustment plan that reduces the bias to the maximum extent possible. Then extended analyses are described with the goal of assessing the bias remaining in the estimates after nonresponse adjustment is completed. In addition to assessing the remaining bias in outcome statistics, the results of the NRB analysis can be used to inform analysts of data limitations, and can also help in future rounds of the survey in defining the target population and improving data collection methods, weighting, and estimation approaches.
The PIAAC Consortium is continuing its research and evaluation of various NRB analysis methods and welcomes comments and suggestions from participating countries on such analyses.
1 Schouten B., Cobben F., and Bethlehem J. Indicators for the representativeness of survey response, Survey Methodology, June 2009.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 8570.10.01.11: PIAACReducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis |
| Author | Leyla Mohadjer |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |
© 2024 OMB.report | Privacy Policy