Appendix F PIAAC National Supplement 2013-14 Nonresponse Bias Procedures

This document addresses the issue of bias in PIAAC outcome statistics and contains an overview of the survey goals to reduce nonresponse bias (NRB) and preliminary plans for conducting NRB analysis. The Consortium will produce detailed guidelines for the NRB analysis after the field test is completed and comments are collected from countries on their lessons learned from the field test.

Section 2 provides a brief background on why reducing NRB is critical in producing high-quality estimates for PIAAC. Section 3 contains a summary of plans and procedures that are critical to reducing bias in the outcome statistics and preliminary plans to conduct the NRB analysis, including examples using data from similar literacy surveys.

The PIAAC Technical Standards and Guidelines (TSG) includes plans and procedures that should be put in place before, during, and after main study data collection to improve response rates and help reduce the potential for NRB. More specifically, the TSG require each country to:

Identify sources of NRB,
Reduce potential for NRB before and during data collection,
Evaluate NRB throughout the design and data collection process,
Evaluate NRB after data collection, and
Adjust for biases through nonresponse adjustment weighting steps.

Sections 3.1 and 3.2 contain brief summaries of procedures included in the TSG that countries should use to make plans to reduce nonresponse bias to the extent possible before and during data collection. At the conclusion of the data collection, countries are required to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustment to the sampling weights. This will help reduce the NRB to the maximum extent possible. Countries with low response rates (see below) are then asked to evaluate the potential for remaining bias using final adjusted weights.

The NRB analyses conducted prior to weighting are referred to as the basic analysis and are described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for each stage of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and if the overall response rate is less than 70%. Section 3.3.2 provides preliminary plans for conducting the extended NRB analysis, including some examples using similar competency surveys.

The focus of this document is on unit response rate (not item response rate). Refer to the TSG for details on the item response rate goals and analyses.

2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES

Probability sampling is essential for PIAAC for two main reasons. First, probability sampling contains a set of designs that leads to a variety of unbiased sampling approaches that allow analysts to generalise the results to the target population. Second, measures of precision related to survey estimates can be computed only under a probability design. The production of unbiased estimates assumes that every eligible adult in the target population would have a nonzero chance of selection (exclusion rate is equal to zero) in a national sample, and would be located, agree to participate in the study, and respond to all survey items (100% response rate). In practice, these circumstances are not realised in any survey population. Missing data occurs when; 1) subsets of the target population are excluded from the sampling frame, 2) some of the adults selected in the sample are not contacted or refuse to participate (referred to as unit nonresponse), 3) they fail to respond to a particular survey item (referred to as item nonresponse), or 4) because data collected from the sampled adults is contaminated (and thus not useful) or lost during or after the data collection phase. The statistical theory behind unbiased survey estimates and associated precision levels are not true anymore when data is missing for the units selected into the probability sample.

NRB (due to missing data for selected adults) can be substantial when two conditions hold: 1) the response rate is relatively low and 2) the difference between the competency levels of respondents and those of nonrespondents is relatively large. This is reflected in the following deterministic NRB formula:

where W_R is the proportion of respondents, Y_R is the mean outcome for respondents, and Y_NRis the mean outcome for nonrespondents.

Similarly, undercoverage bias (due to exclusions) can be substantial if 1) undercoverage rate is high, and 2) the difference in competency levels between adults included in the sample and those excluded from the frame is relatively large. PIAAC TSG sets the maximum allowable exclusion rate at 5% to guard against high undercoverage bias in PIAAC estimates.

Given the relationships between bias and undercoverage and response rates, countries must keep the exclusion rates low and implement procedures to reduce the potential for NRB and attain high response rates.

An alternative model of nonresponse assumes each sampled person has a certain propensity to respond, and NRB in a characteristic is a function of the covariance between the response propensity and the characteristic:

where σ_yp is the covariance between the outcome variable and response propensity, and p is the mean response propensity. Based on this model, NRB is present if missingness is related to competency, as measured PIAAC.

It is well known that NRB can be reduced to some unknown extent through sample weighting if competency is correlated with auxiliary variables, and auxiliary variables are correlated with response propensity. Weighting assumes response probabilities are constant within every group created for weight adjustment, the competency score has zero variance within each group, and response propensity is uncorrelated with competency. It is known that these assumptions are not correct, and the impact of weight adjustments is limited to the number of variables (correlated with competency) available for nonresponse adjustment. Also, it is not possible to measure the exact departure from these assumptions since competency levels of nonrespondents are not known. It is, therefore, necessary to evaluate NRB when response rates are low.

3. PIAAC DATA QUALITY GOAL
REDUCE NRB BEFORE, DURING, AND AFTER DATA COLLECTION

There are several ways to reduce the potential for NRB. First and foremost is to plan and implement field procedures that obtain a high level of cooperation. Section 3.1 provides a brief overview of steps that countries should take to plan and implement effective survey operation procedures to reduce the potential for bias.

Second, it is also critical to monitor the field when data collection is in progress and monitor the distribution of the sample during data collection to ensure steps are taken to reduce the potential for bias as much as possible. Section 3.2 contains a brief summary of these steps.

Finally, countries should search for auxiliary data that can be used to evaluate the potential for bias and that could be used to reduce the impact of nonresponse on survey estimates. Countries should evaluate the potential for remaining bias after weighting adjustments are completed if response rate falls below 70%. Section 3.3 describes the procedures that need to be followed after the data collection is completed.

3.1 Reduce NRB Before Data Collection

No single factor leads to obtaining high response rates in any survey, especially large-scale national surveys. As mentioned above, countries need to focus on a number of critical factors that are key to obtaining reliable data and highest response rates possible.

One of the most critical steps in improving response rates is to hire the most qualified staff for operating the survey. First, countries should attempt to hire interviewers with experience, good people skills, enthusiasm, perseverance, and language skills necessary to conduct PIAAC interviews. Countries should hire some senior interviewers to travel to areas with high nonresponse if necessary. In addition, countries should hire supervisors with experience and knowledge to mentor, motivate, and monitor interviewers. There should be a sufficient number of interviewers and supervisors to conduct the field activities, taking into account the data collection period.

A related critical aspect is establishing effective plans for training the staff. The goal is to ensure that all field staff understand PIAAC so well that they can focus on obtaining quality data and high response rates, not just focusing on the mechanics of the work. Countries should provide thorough initial training, including importance of PIAAC, presentation of study to respondents, and techniques for convincing reluctant respondents to participate. In addition, countries should plan to provide ongoing training via telephone and email. Countries should train supervisors on initial case assignment and reassignment strategies for potential/actual nonresponse.

Community/respondent outreach is another critical component in increasing response rates. Countries should plan to develop an outreach plan, design attractive/cohesive set of materials (e.g., letters, brochures, etc.). Countries should make sure to produce sufficient amount of each material. They should obtain media coverage: newspaper/journal articles about PIAAC that interviewers can show/give to respondents. Various endorsements should be obtained from well-known and well-respected authorities and agencies.

Interviewers/supervisors should be trained on effective use of materials. In addition, countries should design a website that could be used by respondents for reference and to legitimise the survey. Countries should establish toll-free telephone number/hotline—to answer questions and concerns raised by survey participants.

The timing of various activities is also very critical. Introductory materials should be mailed 7-10 days prior to respondent contact. Nonresponse conversion letters should be used for reluctant or hard-to-contact/reach respondents in a timely manner.

Each country should develop a non-interview report (NIR) form to collect information about nonrespondents (see Standard 10.3.7 in the TSG). The form can be paper and pencil or automated, and it should contain, at minimum, demographic data on the non-interviewed person, the strength of the refusal (if applicable), problems encountered, comments, and the likelihood of conversion. The NIR can aid in future interview attempts and possibly provide information for a NRB analysis.

The National Sampling Managers (NSMs) should develop plans for monitoring the sample during data collection to ensure the resulting sample has the lowest NRB attainable. NSMs should monitor the sample and increase response rates for groups that have potential to increase NRB (i.e. identify pockets of nonresponse and use methods to bring in initial nonrespondents that are not like respondents [see Section 3.2, Figure 1]). NSMs are encouraged to use the field test process to develop and test their monitoring programs for the main survey.

A critical task that should be completed prior to the start of data collection is identifying and collecting correlated auxiliary variables that are known with a high degree of accuracy for both respondents and nonrespondents. Evaluating and reducing NRB requires auxiliary variables, available for all sampled units, that are correlated with competency and response propensity. The correlation with competency could be measured using past adult competency or similar studies; variables hypothesised to be correlated with literacy should also be included. A segmentation analysis (see section 3.2) could be used to determine correlation with response propensity. NSMs are encouraged to use the field test data to help identify a set of candidate auxiliary variables for evaluating and reducing NRB. More discussion can be found in the sampling Plan Part II forms, specifically form W-1.

The Cross-Cultural Survey Guidelines website (http://ccsg.isr.umich.edu/datacoll.cfm) has additional suggestions for improving response rates and reducing NRB.

3.2 Reduce NRB During Data Collection

National Project Managers (NPMs) should promote close and timely supervision/monitoring and conduct scheduled weekly telephone conference calls with interviewers. They should keep the pressure on performance and allow for timely feedback. Countries should use Consortium forms or country-specific automated reports that allow for regular/frequent monitoring of case completion. Reports must be based on daily interviewer transmission of interview/assessment data and current contact attempt and disposition code information.

It is important that countries seek and be able to implement new ideas or experiences used successfully by other survey organisations/countries prior to and during data collection. It is always helpful to encourage interviewers/supervisors to make suggestions for improving response rates throughout the data collection period.

Countries should use OTRS to ask for ad hoc advice and use quality control monitoring calls to discuss national issues, concerns, and solutions dealing with respondent cooperation. Countries are encouraged to share with the Consortium novel approaches for obtaining respondent cooperation that can be shared with other countries.

The NSMs should use the planned monitoring procedures (as described in Section 3.1) to monitor response rates and sample yield according to key demographics, known to be associated with competency. This will help countries identify potential shortfalls for specific subgroups that could result in biased estimates. The goal is to reach subgroups or areas with low response and use various approaches to improve response rates for these subgroups (areas). The first step is to identify pockets of potential NRB through modelling, for example, segmentation modelling with local area-level auxiliary variables (smallest geography for which reliable auxiliary data is available, or use registry data when available). A typical segmentation modelling approach involves a classification tree algorithm that uses a chi-square test of independence to identify auxiliary variables that best define subgroups with differential response rates. The subgroups with low response rates, or areas containing these subgroups, can then be targeted for followup efforts to address the potential for NRB.

Figure 1 shows an example of a classification tree used for the West region during a U.S. adult competency household survey data collection period.

Figure 1. Example response rates in West region, USA, during data collection

From Figure 1, it is evident that during data collection in the West there was a low response rate (50%) in segments with a high minority concentration, a high percentage of the population not speaking English well, located in metropolitan statistical areas (MSAs), and having a low percentage of the population with less than a ninth grade education. To the extent that these variables were related to literacy, the low response rate in these areas indicated potential for NRB.

3.3 Reduce NRB After Data Collection

As mentioned in Section 1, all countries are required to conduct the basic NRB analysis described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for all stages of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and for the overall sample if the overall response rate is less than 70%. Figure 2 shows the overall plan for PIAAC NRB analysis. The remainder of this section describes the preliminary plans for various analyses mentioned in the figure.

Figure 2. Preliminary plans for PIAAC NRB analysis

3.3.1 Basic NRB Analysis

The main goal is to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustments to the weights. The basic NRB analysis uses auxiliary variable estimates for survey respondents and nonrespondents. The auxiliary variables may be available on the sampling frame or from a previous data collection stage (e.g. screener data for the BQ/JRA analysis), can come from an external source that can be matched to each sampled unit, or could be observational data on respondents and nonrespondents collected during data collection, assuming the data is of sufficient quality. The auxiliary variables must be available for all eligible units and should be related to competency.

An advantage of the basic analysis is that they could be performed using data that are readily available for both respondents and nonrespondents. However, they are only beneficial if auxiliary variables are highly correlated with competency.

The following techniques can be used:

Use a chi-square test to compare the distribution of auxiliary variables (correlated with competency) for respondents and nonrespondents. Deviations between the auxiliary survey estimates and higher quality external source estimates may indicate a potential for bias in the survey estimates of competency levels, to the extent that the auxiliary variables are related to the key statistics;
Compare response rates for different subgroups;
Use a classification tree algorithm to identify subgroups with low response rates; and/or
Use logistic regression to model the relationship between response status and the auxiliary variables.

Sample Weighting Adjustments

At the conclusion of the basic NRB analysis, countries will be asked to put together a list of variables found to be significantly related to competency and response status. The goal is to use these variables in developing NRB adjustments. Such adjustments will reduce NRB to the extent that the auxiliary variables and the competency measures are correlated, and the auxiliary variables and response propensity are correlated.

3.3.2 NRB Analysis if Response Rate is Less Than 70%

The basic descriptive analysis is a good initial assessment of NRB and is essential in identifying effective weighting variables. However, it has its limitations. The analysis does not reflect the effect of weighting adjustments on NRB, and the extent of bias remaining after nonresponse adjustments are conducted.

Brief descriptions of some possible analyses are provided below. Note that multiple analyses to assess NRB are necessary because each analysis has its own limitations. Together, they will provide an insight into the patterns and potential for bias.

Comparison of estimates before and after weight adjustments are done - The basic analysis described in section 3.3.1 comparing the base-weighted estimates for respondents to the base-weighted full sample estimates can be extended to include a third stage – the respondent estimates using weights adjusted for nonresponse and calibrated to control totals. In other words, estimates from the full sample can be compared to estimates from the respondents before and after weighting adjustments. If the full sample estimates are closer to the final estimates than the base weighted estimates, this indicates bias in the auxiliary variables was reduced through the weighting process.

Comparison of weighted estimates to external totals - Another extended analysis is to compare estimates from PIAAC to estimates from external source. The PIAAC estimates should be produced using the final weights that have been adjusted for nonresponse and calibrated to control totals. Care should be taken to choose external source estimates that measure the same characteristic for a similar time period. The external source estimates will also be subject to error, and the variance of these estimates should be taken into account when making this comparison.

Correlations between weighting adjustment variables and competency measures - The analyses described thus far rely on auxiliary variables and do not directly measure bias in the competency estimates. As mentioned earlier, bias in the auxiliary variables is indicative of bias in the competency estimates to the extent that the auxiliary variables and competency estimates are correlated. Thus, correlations between the auxiliary variables and competency data can be computed to evaluate this relationship. For variables used in the weighting adjustments, a low correlation with literacy implies that using the variable in the weighting adjustments did little to reduce NRB. On the other hand, a high correlation with literacy implies a potentially high reduction in NRB. For variables not used in the weighting adjustments, a high correlation with literacy indicates potential bias in the literacy estimates. This can be used to inform weighting decisions for future surveys.

The disadvantage of using correlations to evaluate NRB is that the correlations are based on respondents only and the relationship between competency and the auxiliary variables might be different for nonrespondents. However, this is not as much a concern if the relationship can be confirmed using outside sources.

Comparison of estimates using alternative weighting adjustments - For this evaluation, an auxiliary variable is re-calibrated to known totals, and estimates of the key statistics are compared before and after the re-weighting. Re-weighting can be useful as an evaluation tool when:

The variable was not used in weighting (because it was of a low quality);
The variable is correlated with the outcome measure; and
The variable is correlated with response propensity.

Any differences between estimates using the official survey weights and the re-weighted weights reflect noncoverage as well as NRB, but if there is not a large change in the estimates, this is further confirmation that NRB may not be a concern.

Analysis of variables collected during data collection - BQ variables - The descriptive analyses described earlier assess the potential for bias in some statistics that may not necessarily be highly related to competency. Therefore, the next step of the analysis should be to evaluate bias on the key statistics or closely related variables. For PIAAC, the key statistics are scores measuring competency in the components of the assessments that will be available later. However, before literacy scores are available, an analysis can be performed with BQ/JRA variables believed to be closely related to literacy, such as questions on measuring habits (newspapers or magazines), for example. Analyses of such variables may show patterns of bias that cannot be effectively eliminated by the auxiliary variables used in weighting.

Analysis of variables collected during data collection - Disposition codes - Disposition codes contain information on reasons for nonresponse. For this analysis, distributions of sampled persons with known characteristics related to outcome (i.e. the language barrier cases) should be examined. For example, the demographic distribution of language barrier cases can be compared to other eligible persons using auxiliary data, and interview data.

Analysis of variables collected during data collection - NIR - The NIR forms identify observable demographic information and reasons for nonresponse that are not captured in the disposition codes. The NIR forms can potentially indicate whether the reasons for nonresponse are related to competency estimates and suggest ways to improve response rates for future surveys.

Comparison of late or “hard-to-contact” respondents to early respondents - This analysis is helpful for evaluating the potential for NRB for differences that cannot be captured through adjusting for known demographics. Significant differences between the competency levels of early and late respondents imply that the bias could have been potentially reduced through focused efforts to obtain the late respondents. However, to the extent that differences between early and late respondents reflect differences between respondents and nonrespondents, the findings indicate that some level of NRB might still be present (which depends on the magnitude of nonresponse rate). The validity of this assumption, that late respondents are similar to nonrespondents, depends largely on the strategies used in the late data collection effort. For many surveys this may not be a valid assumption, but it seems reasonable for a household survey such as PIAAC where multiple contacts are usually required to gain cooperation.

Calculation of the range of potential bias - How can we evaluate the potential for bias remaining after weighting if nonrespondents are very different from respondents within the weighting classes? The range of potential bias can be evaluated using a deterministic NRB formula that calculates how different respondent and nonrespondents competency estimates need to be to bias the overall competency estimate by varying degrees. For example, one can show that for a response rate of 70% (W_R = 0.7), the mean of respondents and nonrespondents would have to differ by 50 points (Y_R-Y_NR = 50) to bias the overall estimate by 15 points (Bias( y_R) = 15).

Other analysis under review - The Consortium is currently reviewing other NRB analysis procedures, such as the R indicator (Schouten, Cobben, and Bethlehem, 2009¹), for possible inclusion in the NRB plan for PIAAC.

The following provides a summary of the steps included in the NRB analysis when response rate falls below 70%.

Step 1 of 3: Identify Variables

1.1 Identify auxiliary variables from the frame or external sources.

1.2 Identify informative BQ variables correlated with competency.

1.3 Identify other informative variables collected during data collection.

Step 2 of 3: Basic NRB Analysis

2.1 Conduct basic descriptive and model-based analysis.

2.2 Finalise variables selected for weighting adjustments.

2.3 Conduct nonresponse adjustments and compute final sampling weights.

Step 3 of 3: Extended NRB Analysis

3.1 Conduct extended descriptive and model-based analysis using auxiliary data.

3.2 Conduct analysis of informative BQ variables.

3.3 Conduct analysis of disposition codes.

3.4 Conduct analysis of NIR (information not available in disposition codes).

Example: The following information and tables are summaries extracted from the NRB analysis conducted for the Adult Literacy and Lifeskills Survey (ALL) 2003: U.S. NRB Analysis http://nces.ed.gov/pubs2009/2009063.pdf, and other unpublished NRB analyses conducted for similar adult competency household surveys.

The steps carried out in the following NRB analysis are summarized below.

Step 1 of 3: Identify variables from the frame or external sources

Step 1.1 The following variables were extracted from Census 2000 county-level data, and Census 2000 block group level data.

Census variables available for local areas (where respondents and nonrespondents resided)
Urban/rural status
Majority own/rent
Average household size
Percentage of persons 25 years or older with less than a high school diploma
Percentage of persons speaking English not well or not at all
Percentage below 150% poverty

Step 1.2 The following variables were extracted from the BQ.

Percentage born in the United States
Percentage with high school diploma
Percentage never receiving remedial help in school
Percentage agreeing with the statement: I enjoyed math in school
Percentage speaking English most often at home
Percentage employed
Percentage reading letters, memos, or emails at least once a week as part of main job

Step 1.3 Disposition codes and NIR variables were collected after the end of data collection.

Step 2 of 3: Basic NRB analysis – Tables 1 through 4 provide examples of analysis conducted for ALL.

Table 1 shows the results of chi-square tests conducted for ALL. The chi-square tests indicated a significant relationship of response status to region, MSA status, the percentage below 150% of poverty, age, sex, and race/ethnicity. These variables were included in the nonresponse adjustments to reduce the potential NRB to the extent that these auxiliary variables were related to literacy.

Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by key characteristics: 2003

Analysis variable	Respondents percent	Standard error	Eligibles percent	Standard error	Chi-square statistic	p value
Region
Northeast	15.5	2.50	18.3	2.36	31.95	0.000
Midwest	22.9	2.54	22.1	2.24	†	†
South	35.0	1.66	34.9	1.50	†	†
West	26.6	2.47	24.7	2.21	†	†
Metropolitan Statistical Area status
Non-Metropolitan Statistical Area	22.7	2.63	21.8	2.40	7.26	0.007
Metropolitan Statistical Area	77.3	2.63	78.2	2.40	†	†
Locale¹
Urban	64.6	2.67	65.3	2.45	3.78	0.12
Suburban	13.0	2.00	12.4	1.83	†	†
Rural	22.4	2.11	22.3	2.02	†	†
Majority own/rent
Rent	25.7	2.48	25.2	2.40	1.14	0.287
Own	74.3	2.48	74.8	2.40	†	†
Average household size
less than or equal to 2.2	18.1	1.87	18.6	1.81	4.73	0.081
greater than 2.2 and less than or equal to 2.8	52.9	2.50	51.5	2.80	†	†
greater than 2.8	29.1	2.92	29.9	3.10	†	†
Percent less than high school
less than or equal to 10%	33.1	2.71	33.4	2.73	2.7	0.257
greater than 10% and less than or equal to 20%	30.9	1.81	31.4	1.75	†	†
greater than 20%	36.1	3.01	35.2	3.09	†	†
Percent limited English proficient
0%	29.9	2.68	29.9	2.51	0.06	0.963
greater than 0% and less than 2%	34.7	3.19	34.6	3.09	†	†
greater than or equal to 2%	35.4	3.78	35.5	3.64	†	†
Percent below 150% of poverty
less than or equal to 10%	30.7	2.65	31.5	2.79	8.59	0.011
greater than 10% and less than or equal to 30%	44.9	3.27	45.4	3.27	†	†
greater than 30%	24.4	2.95	23.1	2.76	†	†

Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by key characteristics: 2003 (Continued)

Analysis variable	Respondents percent	Standard error	Eligibles percent	Standard error	Chi-square statistic	p value
Age
16–25	23.7	1.38	22.4	1.25	17.92	0
26–35	21.2	0.80	20.8	0.77	†	†
36–45	22.2	0.84	22.9	0.79	†	†
46–65	32.9	1.29	33.9	1.16	†	†
Sex
Male	48.1	1.08	49.6	0.99	22.77	0
Female	51.9	1.08	50.4	0.99	†	†
Race/ethnicity²
Hispanic	14.3	1.79	13.3	1.61	19.11	0
Black, non-Hispanic	11.1	1.82	10.2	1.62	†	†
Other, non-Hispanic	74.6	2.08	76.4	1.94	†	†

† Not applicable.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Table 2 contains estimates of bias and relative bias for the subgroup percentages presented in Table 1. Bias was calculated using the deterministic NRB formula in section 2. A t test was performed to determine whether the NRB was significantly different from zero. Another measure, the relative bias, was computed as the bias divided by the estimate from the eligible sample. The relative bias is a measure of the size of the bias compared to the eligible sample estimate.

The results of the analysis in Table 2 were consistent with the chi-square analysis in Table 1. It also showed that even though there were statistically significant results as described above, the relative magnitude of the differences was small. The relative bias estimates were all less than 10 percent, except in the Northeast, which was 15 percent.

Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics

2003 Analysis variable	Respondents percent	Eligibles percent	Bias			Percent relative bias
2003 Analysis variable	Respondents percent	Eligibles percent	Estimate	Standard error	p value	Percent relative bias
Region
Northeast	15.5	18.3	-2.8	0.53	0.000	-15.3
Midwest	22.9	22.1	0.8	0.61	0.187	3.6
South	35.0	34.9	0.1	0.49	0.842	0.3
West	26.6	24.7	1.8	0.48	0.001	7.3
Metropolitan Statistical Area status
Non-Metropolitan Statistical Area	22.7	21.8	0.9	0.34	0.010	4.1
Metropolitan Statistical Area	77.3	78.2	-0.9	0.34	0.010	-1.2
Locale¹
Urban	64.6	65.3	-0.7	0.38	0.074	-1.1
Suburban	13.0	12.4	0.6	0.37	0.108	4.8
Rural	22.4	22.3	0.1	0.31	0.747	0.4

Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics (Continued)

2003 Analysis variable	Respondents percent	Eligibles percent	Bias			Percent relative bias
2003 Analysis variable	Respondents percent	Eligibles percent	Estimate	Standard error	p value	Percent relative bias
Majority own/rent
Rent	25.7	25.2	0.5	0.44	0.298	2.0
Own	74.3	74.8	-0.5	0.44	0.298	-0.7
Average household size
less than or equal to2.2	18.1	18.6	-0.5	0.39	0.215	-2.7
greater than 2.2 and less than or equal to 2.8	52.9	51.5	1.4	0.65	0.044	2.7
greater than 2.8	29.1	29.9	-0.9	0.6	0.155	-3.0
Percent less than high school
less than or equal to10%	33.1	33.4	-0.3	0.53	0.540	-0.9
greater than 10% and less than or equal to 20%	30.9	31.4	-0.5	0.48	0.284	-1.6
greater than 20%	36.1	35.2	0.9	0.52	0.112	2.6
Percent limited English proficient
0%	29.9	29.9	#	0.56	0.977	#
greater than 0% and less than 2%	34.7	34.6	0.1	0.69	0.841	0.3
greater than or equal to 2%	35.4	35.5	-0.1	0.55	0.826	-0.3
Percent below 150% of poverty
less than or equal to10%	30.7	31.5	-0.8	0.52	0.129	-2.5
greater than 10% and less than or equal to 30%	44.9	45.4	-0.5	0.58	0.379	-1.1
greater than 30%	24.4	23.1	1.3	0.38	0.001	5.6
Age
16–25	23.7	22.4	1.3	0.35	0.001	5.8
26–35	21.2	20.8	0.4	0.25	0.106	1.9
36–45	22.2	22.9	-0.6	0.36	0.088	-2.6
46–65	32.9	33.9	-1	0.41	0.016	-2.9
Sex
Male	48.1	49.6	-1.5	0.31	0.000	-3.0
Female	51.9	50.4	1.5	0.31	0.000	3.0
Race/ethnicity²
Hispanic	14.3	13.3	0.9	0.29	0.003	6.8
Black, non-Hispanic	11.1	10.2	0.9	0.31	0.009	8.8
Other, non-Hispanic	74.6	76.4	-1.8	0.32	0.000	-2.4

# Rounds to zero

¹ This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

² All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

NOTE: Detail may not sum to totals because of rounding.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

The analyses in Tables 1 and 2 were useful in explaining the relationship of response status to each auxiliary variable individually. A classification tree algorithm and a logistic regression model were used to evaluate the relationship between response status and multiple auxiliary variables. The results of the classification tree algorithm are shown in Table 3. The algorithm was similar to that described in section 2 and used chi-square tests to divide the sample into subgroups that best explain differential response rates. As Table 3 shows, 22 cells were formed with weighted response rates ranging from 58.7% to 97.4%. The subgroups defined by the classification tree algorithm could be used as nonresponse adjustment cells in the weighting adjustment.

Table 3. Weighted response rates for the ALL background questionnaire, by response cell: 2003

Response cell	Description	Weighted response rate (%)
Overall		79.7
1	Northeast, Non–Metropolitan Statistical Area (MSA)	88.9
2	Northeast, MSA, Average household size less than or equal to 2.8	68.1
3	Northeast, MSA, Average household size greater than 2.8	58.7
4	Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size less than or equal to 2.2	79.8
5	Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.2 and less than or equal to 2.8	91.1
6	Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.8	78.0
7	Midwest and South, Hispanic or non-Hispanic Black, Female, Age 16-45	94.4
8	Midwest and South, Hispanic or non-Hispanic Black, Female, Age 46-65	82.0
9	Midwest and South; Other, non-Hispanic; Age 16-35; Male; Rent	70.0
10	Midwest and South; Other, non-Hispanic; Age 16-35; Male; Own	82.1
11	Midwest; Other, non-Hispanic; Female; Age 16-25	97.4
12	Midwest; Other, non-Hispanic; Female; Age 26-35	90.2
13	South; Other, non-Hispanic; Female; Percent limited English proficient less than 2%	86.0
14	South; Other, non-Hispanic; Female; Percent limited English proficient greater than or equal to 2%	70.3
15	Midwest; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or equal to 30%	77.3
16	South; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or equal to 30%	71.5
17	Midwest and South; Other, non-Hispanic; Age 36-65; Percent below 150% of poverty greater than 30%	84.7
18	West, Rent	88.6
19	West, Own, Age 16-25	88.7
20	West, Own, Age 26-45	79.6
21	West, Own, Age 46-65, Urban or Suburban	90.5
22	West, Own, Age 46-65, Rural	78.5

NOTE: The response cells were formed using the Chi-square Automated Interaction Detector.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Logistic regression models are also useful in identifying significant effects on response propensity. Response status was used as the binary dependent variable and the set of auxiliary variables in Table 1 as the predictors. An F test was performed for each auxiliary variable to determine if it was significantly related to response propensity, after accounting for the other auxiliary variables. Results are shown in Table 4. Average household size, age, race/ethnicity, sex, and region are shown to be potentially valuable variables to include in the nonresponse adjustments. While this analysis was limited to main effects, interaction terms can also be included.

Table 4. Multivariate analysis of response indicators: 2003

Test	F statistic	Degrees of freedom		p value
Test	F statistic	Numerator	Denominator	p value
Overall fit	3.89	21	10	0.016
Average household size	6.77	2	29	0.004
Majority own/rent	0.32	1	30	0.578
Percent less than high school	0.44	2	29	0.647
Percent limited English proficient	2.08	2	29	0.143
Locale¹	0.12	2	29	0.887
Age	4.47	3	28	0.011
Race/ethnicity²	10.79	2	29	0.000
Sex	19.43	1	30	0.000
Metropolitan Statistical Area status	0.03	1	30	0.871
Percent below 150% of poverty	3.22	2	29	0.055
Region	6.65	3	28	0.002

¹This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

²All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Step 3 of 3: Extended analysis – Tables 5 through 10 provide examples of extended NRB analysis as conducted for ALL and other adult competency surveys

Table 5 shows an example of the comparison of estimates before and after weighting adjustments. The following t test comparisons were made:

Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with the BQ base weights for the BQ respondents only to check for differences due to nonresponse to the BQ
Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with that from the BQ nonresponse adjusted weights for respondents to check for differences even after the nonresponse adjustment process to the BQ
Comparison of percentage distributions from BQ nonresponse adjusted weights for respondents with that from the BQ raked weights for respondents to check for differences that may have been introduced through the initial raking procedure
This analysis showed that, through the nonresponse adjustment, the potential for bias was reduced (p-values moved from significant to nonsignificant) in all domains shown below, with the exception of two domains relating to average household size. These two domains show significant bias created by the nonresponse adjustments. While the differences in the percentage distributions of these domains remain statistically significant even after the raking adjustment, they are not large enough to be meaningful.

Table 5. Percentage distribution of sample cases at each weighting step for the ALL background questionnaire, by key characteristics: 2003

	Total sample %	BQ respondents		Nonresponse adjustment		Initial raking
	Total sample %	%	p value¹	%	p value¹	%	p value²
Total	100.0	100.0	†	100.0	†	100.0	†
Region
Northeast	18.4	15.6	0.000	18.6	0.355	19.0	0.866
Midwest	22.1	22.9	0.180	21.9	0.355	23.0	0.728
South	34.9	35.0	0.842	34.9	0.825	35.0	0.716
West	24.7	26.6	0.001	24.7	0.895	23.0	0.413
Metropolitan Statistical Area status
Non-Metropolitan Statistical Area	21.8	22.7	0.012	21.9	0.558	18.0	0.100
Metropolitan Statistical Area	78.2	77.3	0.012	78.1	0.558	82.0	0.100
Locale³
Urban	65.3	64.6	0.080	65.1	0.448	68.0	0.097
Suburban	12.4	13.0	0.103	12.4	0.849	11.0	0.198
Rural	22.3	22.4	0.749	22.5	0.566	20.0	0.075
Majority own/rent
Rent	25.2	25.7	0.294	25.0	0.486	26.0	0.254
Own	74.8	74.3	0.294	75.0	0.486	74.0	0.254
Average household size
less than or equal to 2.2	18.6	18.1	0.225	18.1	0.213	18.1	0.946
greater than 2.2 and less than or equal to 2.8	51.5	52.9	0.051	53.3	0.003	52.5	0.257
greater than 2.8	30.0	29.1	0.157	28.7	0.005	29.4	0.381
Percent less than high school
less than or equal to 10%	33.4	33.1	0.542	33.1	0.569	32.7	0.431
greater than 10% and less than or equal to 20%	31.4	30.9	0.292	31.4	0.887	31.3	0.937
greater than 20%	35.2	36.1	0.123	35.5	0.349	36.0	0.524
Percent limited English proficient
0%	29.9	29.9	0.977	30.0	0.953	28.8	0.088
greater than 0% and less than 2%	34.6	34.7	0.840	35.4	0.162	35.5	0.756
greater than or equal to 2%	35.5	35.4	0.825	34.7	0.052	35.8	0.314

Table 5. Percentage distribution of sample cases at each weighting step for the ALL background questionnaire, by key characteristics: 2003 (Continued)

	Total sample %	BQ respondents		Nonresponse adjustment		Initial raking
	Total sample %	%	p value¹	%	p value¹	%	p value²
Percent below 150% of poverty
less than or equal to 10%	31.5	30.7	0.137	31.6	0.912	32.0	0.629
greater than 10% and less than or equal to 30%	45.4	44.9	0.372	45.3	0.682	45.4	0.814
greater than 30%	23.1	24.4	0.002	23.2	0.452	22.7	0.585

¹The p value and SE are for tests of difference between estimated percentage of current step and total sample (BQ base weights).

² The p value and SE are for tests of difference between estimated percentage of current and previous weighting steps.

³This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Table 6 gives an example of correlations between auxiliary variables and competency score measures for an adult competency survey. The percentage aged 25+ with less than a high school education was found to be highly correlated with literacy score. If this variable was also related to response propensity, using it in nonresponse adjustments should reduce the potential NRB of literacy estimates.

Table 6. Correlations of demographic variable with the outcome statistic

Variable	Absolute correlation with outcome statistic
Variable	<0.1	0.1-0.2	0.2-0.3	0.3-0.4	>0.4
Percent aged 25+ with less than high school education					X
Percent of adults speaking Spanish at home and English not well or not at all				X
Percent below 150% of poverty				X
Median Income			X
Percent who rent			X
Average household size		X
Age		X
Percent minority	X

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

Table 7 provides an example of the analysis of disposition codes to evaluate NRB in an adult competency survey. The distribution of cases with a BQ disposition code of language problem or mental disability was compared to the distribution of the other nonrespondents. Chi-square tests were processed to determine if there was a relationship between select demographic variables and the disposition code groups (literacy-related or other) for incompletes. Significant differences in the distribution by region, MSA status, and age were found between the two groups. These variables were used in the weighting adjustments, which are expected to reduce the potential bias from literacy-related nonresponse.

Table 7. Distribution of BQ incompletes, by disposition code category

Domain		Literacy-related²		other
Total incompletes		355		5,191
Domain	Incompletes¹			Statistic	p value
Domain	Literacy- related²percent		OTHER percent	Statistic	p value
Total incompletes	100		100
Region				10	0.006
Northeast	16		17
Midwest	18		22
South	21		38
West	45		23
MSA status				9	0.002
Non-MSA	9		18
MSA	91		82
Age				23	0.000
16-29	16		20
30-49	37		43
50-69	23		27
70+	24		11
Gender				3	0.067
Male	47		53
Female	53		47
Race/ethnicity				2	0.244
Hispanic	17		11
NH Black only	6		9
Other	77		79

† Not applicable

¹Nonrespondents

² Literacy-related reasons for nonresponse at the BQ stage consist of language problems and mental disabilities only.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

An example of an analysis of NIR data for an adult competency survey is provided in Table 8. The table summarizes data collected on the NIR regarding the household race of screener incompletes, as observed or gathered from neighbour information. The base-weighted distribution is shown overall and by region. For households where race was determined, Hispanic households were less common among screener incompletes compared to the population proportion. This suggests a possible increase in response rates from accommodations made for Spanish speakers, such as bilingual interviewers.

Table 8. Household race for screener incompletes with NIR data, by region

Household race	Overall	Region
Household race	Overall	Northeast	Midwest	South	West
Total	4,276	1,327	819	1,571	559
Not ascertained	2,197	548	535	804	310
	Percent	Percent	Percent	Percent	Percent
Total	100	100	100	100	100
White	73	73	84	70	67
Black	10	6	7	15	7
Hispanic	4	3	4	3	8
Asian	4	4	1	2	9
NH/PI¹	0	#	#	#	1
AI/AN²	0	#	#	0	#
Other	0	1	#	0	1
Cannot determine	10	14	5	10	8

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

Table 9 shows an example of a comparison of literacy estimates for late versus early respondents. Calculations were done using marginal maximum likelihood (MML) means. The results indicate a small but significant bias in Statistic 2 if focused followup efforts (as describe in section 2) had not been successful in obtaining responses from respondents at the end of the data collection period (i.e. if data about respondents at the end of the data collection period had been missing). This suggests some small level of NRB may be present to the extent that nonrespondents are similar to respondents at the end of the data collection period.

Table 9. Differences between the last 10% and first 90% of respondents, by domain

Domain	Estimate of population proportion			Statistic 1		Statistic 2		Statistic 3
Domain	Proportion among last 10%	Proportion among first 90%	p value	Relative difference	p value	Relative difference	p value	Relative difference	p value
Overall	1.00	1.00	†	0.00	0.92	-0.03	0.01*	0.00	0.80
Region
Northeast	0.20	0.19	0.82	0.00	0.89	-0.03	0.12	0.01	0.79
Midwest	0.18	0.23	0.03	0.03	0.22	0.01	0.58	0.03	0.08
South	0.36	0.36	0.95	0.01	0.64	-0.02	0.36	0.00	0.96
West	0.27	0.21	0.02	-0.03	0.13	-0.06	0.01*	-0.01	0.61
MSA status
Non-MSA	0.13	0.19	0.04	0.00	0.95	-0.02	0.38	0.02	0.38
MSA	0.87	0.81	0.04	0.00	0.97	-0.03	0.01*	0.00	0.88
Gender
Male	0.53	0.48	0.01*	0.01	0.50	-0.02	0.22	0.00	0.94
Female	0.47	0.52	0.01*	-0.01	0.70	-0.04	0.01*	0.00	0.84
Race/ethnicity
Hispanic	0.17	0.12	0.00*	-0.02	0.35	-0.07	0.00*	-0.01	0.55
NH Black only	0.11	0.11	0.51	0.01	0.57	0.02	0.41	0.06	0.03
Other	0.73	0.77	0.04	0.01	0.40	-0.01	0.20	0.01	0.51
Educational attainment
Less than high school	0.18	0.19	0.33	-0.05	0.01*	-0.07	0.00*	-0.02	0.33
High school or equivalent	0.28	0.31	0.10	-0.02	0.50	-0.04	0.02	0.00	0.82
More than high school	0.54	0.50	0.06	0.00	0.99	-0.03	0.01*	-0.01	0.25
Age category
16-29	0.25	0.25	0.68	-0.01	0.75	-0.02	0.22	0.01	0.79
30-49	0.42	0.39	0.06	-0.01	0.35	-0.04	0.0*	0.00	0.87
50-69	0.25	0.25	0.92	0.02	0.25	0.00	0.85	0.00	0.96
70+	0.09	0.11	0.07	-0.03	0.53	-0.10	0.01*	-0.07	0.11
Country of birth
US	0.80	0.86	0.00	0.01	0.44	-0.01	0.32	0.01	0.29
Other	0.20	0.14	0.00	0.00	0.97	-0.05	0.04	0.00	0.97

† Not applicable

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

An example of the calculation of the range of potential bias in literacy estimates is shown in Table 10. The table shows the difference between respondent and nonrespondents literacy scores that would bias the population estimate by b = 3, 5, 10, or 15 points. The required difference between respondents and nonrespondents was computed as described earlier in section 3.3.2. For the overall sample, literacy scores of respondents and nonrespondents would have to differ by about 8 points on average to bias the national estimate by 3 points and would need to differ by 39 points to bias the national estimate by 15 points.

Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics

Domain	Percent of population¹ represented by nonrespondents²	Average point difference between respondents and nonrespondents needed to bias estimated scores by
Domain	Percent of population¹ represented by nonrespondents²	3	5	10	15
Total	38	8	13	26	39
Region
Northeast	43	7	12	23	35
Midwest	35	9	14	28	43
South	38	8	13	26	40
West	38	8	13	26	39
MSA status
Non-MSA	33	9	15	30	45
MSA	40	8	13	25	38
Average household size
≤ 2.42	39	8	13	26	39
(2.42, 2.8]	39	8	13	26	39
> 2.8	38	8	13	27	40
Percent less than high school
≤ 10.4%	43	7	12	23	35
(10.4%, 20.3%]	39	8	13	26	38
(20.3%, 32%]	35	8	14	28	42
> 32%	31	10	16	32	48
Percent speaking Spanish but not English
0%	39	8	13	26	38
(0%, 28%]	40	8	13	25	38
> 28%	34	9	15	29	44
Percent below 150% of poverty
≤ 10.7%	43	7	12	23	35
(10.7%, 20%]	40	8	13	25	38
(20%, 33.3%]	35	9	14	29	43
> 33.3%	29	10	17	34	52
Median income
≤ 28,400	28	11	18	36	53
(28,400, 37,850]	34	9	15	29	44
(37,850, 52,100]	40	8	13	25	38
> 52,100	44	7	11	23	34

Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics (Continued)

Domain	Percent of population¹ represented by nonrespondents²	Average point difference between respondents and nonrespondents needed to bias estimated scores by
Domain	Percent of population¹ represented by nonrespondents²	3	5	10	15
Percent rent
≤ 16%	42	7	12	24	36
(16%, 31%]	37	8	13	27	40
(31%, 59%]	37	8	14	27	41
> 59%	35	9	14	29	43
Low response rate CHAID cells
Cell 1³	59	5	8	17	25
Cell 2⁴	51	6	10	20	30
Cell 3⁵	50	6	10	20	30

¹The population excludes language problems and mental disabilities.

²Nonrespondents to the Screener and BQ.

³Median income greater than 52,100, average household size 2.42 or less, Northeast region

⁴ Median income 37,850 - 52,100, percent less than high school greater than 20.3, Northeast region

⁵ Median income greater than 52,100, average household size 2.42 - 2.8, South or West region

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

4. Summary

This document presented the main goals of PIAAC for reducing NRB throughout the survey process. A series of procedures for implementation before and during the data collection were described to help increase the response rates with the goal of reducing NRB to the extent possible. Preliminary plans are provided to conduct basic NRB analysis to allow an effective weighting adjustment plan that reduces the bias to the maximum extent possible. Then extended analyses are described with the goal of assessing the bias remaining in the estimates after nonresponse adjustment is completed. In addition to assessing the remaining bias in outcome statistics, the results of the NRB analysis can be used to inform analysts of data limitations, and can also help in future rounds of the survey in defining the target population and improving data collection methods, weighting, and estimation approaches.

The PIAAC Consortium is continuing its research and evaluation of various NRB analysis methods and welcomes comments and suggestions from participating countries on such analyses.

1 Schouten B., Cobben F., and Bethlehem J. Indicators for the representativeness of survey response, Survey Methodology, June 2009.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	8570.10.01.11: PIAACReducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis
Author	Leyla Mohadjer
File Modified	0000-00-00
File Created	2021-01-29

Appendix F PIAAC National Supplement 2013-14 Nonresponse Bias Procedures