Appendix F PIAAC National Supplement 2013-14 Nonresponse Bias Procedures

Appendix F PIAAC National Supplement 2013-14 Nonresponse Bias Procedures.docx

Program for the International Assessment of Adult Competencies (PIAAC) National Supplement Data Collection 2013-2014

Appendix F PIAAC National Supplement 2013-14 Nonresponse Bias Procedures

OMB: 1850-0870

Document [docx]
Download: docx | pdf















Appendix F

PIAAC Reducing Nonresponse Bias and

Preliminary Nonresponse Bias Analysis











PIAAC

Reducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis























P IAAC_NRB Paper_10March10


Table of Contents






List of Figures










PIAAC

Reducing NONRESPONSE BIAS AND
PRELIMINARY NONRESPONSE BIAS ANALYSIS

MARCH 2010



1. INTRODUCTION AND OVERVIEW

This document addresses the issue of bias in PIAAC outcome statistics and contains an overview of the survey goals to reduce nonresponse bias (NRB) and preliminary plans for conducting NRB analysis. The Consortium will produce detailed guidelines for the NRB analysis after the field test is completed and comments are collected from countries on their lessons learned from the field test.


Section 2 provides a brief background on why reducing NRB is critical in producing high-quality estimates for PIAAC. Section 3 contains a summary of plans and procedures that are critical to reducing bias in the outcome statistics and preliminary plans to conduct the NRB analysis, including examples using data from similar literacy surveys.


The PIAAC Technical Standards and Guidelines (TSG) includes plans and procedures that should be put in place before, during, and after main study data collection to improve response rates and help reduce the potential for NRB. More specifically, the TSG require each country to:


  • Identify sources of NRB,

  • Reduce potential for NRB before and during data collection,

  • Evaluate NRB throughout the design and data collection process,

  • Evaluate NRB after data collection, and

  • Adjust for biases through nonresponse adjustment weighting steps.


Sections 3.1 and 3.2 contain brief summaries of procedures included in the TSG that countries should use to make plans to reduce nonresponse bias to the extent possible before and during data collection. At the conclusion of the data collection, countries are required to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustment to the sampling weights. This will help reduce the NRB to the maximum extent possible. Countries with low response rates (see below) are then asked to evaluate the potential for remaining bias using final adjusted weights.


The NRB analyses conducted prior to weighting are referred to as the basic analysis and are described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for each stage of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and if the overall response rate is less than 70%. Section 3.3.2 provides preliminary plans for conducting the extended NRB analysis, including some examples using similar competency surveys.


The focus of this document is on unit response rate (not item response rate). Refer to the TSG for details on the item response rate goals and analyses.






2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES

Probability sampling is essential for PIAAC for two main reasons. First, probability sampling contains a set of designs that leads to a variety of unbiased sampling approaches that allow analysts to generalise the results to the target population. Second, measures of precision related to survey estimates can be computed only under a probability design. The production of unbiased estimates assumes that every eligible adult in the target population would have a nonzero chance of selection (exclusion rate is equal to zero) in a national sample, and would be located, agree to participate in the study, and respond to all survey items (100% response rate). In practice, these circumstances are not realised in any survey population. Missing data occurs when; 1) subsets of the target population are excluded from the sampling frame, 2) some of the adults selected in the sample are not contacted or refuse to participate (referred to as unit nonresponse), 3) they fail to respond to a particular survey item (referred to as item nonresponse), or 4) because data collected from the sampled adults is contaminated (and thus not useful) or lost during or after the data collection phase. The statistical theory behind unbiased survey estimates and associated precision levels are not true anymore when data is missing for the units selected into the probability sample.


NRB (due to missing data for selected adults) can be substantial when two conditions hold: 1) the response rate is relatively low and 2) the difference between the competency levels of respondents and those of nonrespondents is relatively large. This is reflected in the following deterministic NRB formula:


,


where WR is the proportion of respondents, YR is the mean outcome for respondents, and YNR is the mean outcome for nonrespondents.


Similarly, undercoverage bias (due to exclusions) can be substantial if 1) undercoverage rate is high, and 2) the difference in competency levels between adults included in the sample and those excluded from the frame is relatively large. PIAAC TSG sets the maximum allowable exclusion rate at 5% to guard against high undercoverage bias in PIAAC estimates.


Given the relationships between bias and undercoverage and response rates, countries must keep the exclusion rates low and implement procedures to reduce the potential for NRB and attain high response rates.


An alternative model of nonresponse assumes each sampled person has a certain propensity to respond, and NRB in a characteristic is a function of the covariance between the response propensity and the characteristic:

,

where σyp is the covariance between the outcome variable and response propensity, and p is the mean response propensity. Based on this model, NRB is present if missingness is related to competency, as measured PIAAC.


It is well known that NRB can be reduced to some unknown extent through sample weighting if competency is correlated with auxiliary variables, and auxiliary variables are correlated with response propensity. Weighting assumes response probabilities are constant within every group created for weight adjustment, the competency score has zero variance within each group, and response propensity is uncorrelated with competency. It is known that these assumptions are not correct, and the impact of weight adjustments is limited to the number of variables (correlated with competency) available for nonresponse adjustment. Also, it is not possible to measure the exact departure from these assumptions since competency levels of nonrespondents are not known. It is, therefore, necessary to evaluate NRB when response rates are low.




3. PIAAC DATA QUALITY GOAL
REDUCE NRB BEFORE, DURING, AND AFTER DATA COLLECTION

There are several ways to reduce the potential for NRB. First and foremost is to plan and implement field procedures that obtain a high level of cooperation. Section 3.1 provides a brief overview of steps that countries should take to plan and implement effective survey operation procedures to reduce the potential for bias.


Second, it is also critical to monitor the field when data collection is in progress and monitor the distribution of the sample during data collection to ensure steps are taken to reduce the potential for bias as much as possible. Section 3.2 contains a brief summary of these steps.


Finally, countries should search for auxiliary data that can be used to evaluate the potential for bias and that could be used to reduce the impact of nonresponse on survey estimates. Countries should evaluate the potential for remaining bias after weighting adjustments are completed if response rate falls below 70%. Section 3.3 describes the procedures that need to be followed after the data collection is completed.




3.1 Reduce NRB Before Data Collection

No single factor leads to obtaining high response rates in any survey, especially large-scale national surveys. As mentioned above, countries need to focus on a number of critical factors that are key to obtaining reliable data and highest response rates possible.


One of the most critical steps in improving response rates is to hire the most qualified staff for operating the survey. First, countries should attempt to hire interviewers with experience, good people skills, enthusiasm, perseverance, and language skills necessary to conduct PIAAC interviews. Countries should hire some senior interviewers to travel to areas with high nonresponse if necessary. In addition, countries should hire supervisors with experience and knowledge to mentor, motivate, and monitor interviewers. There should be a sufficient number of interviewers and supervisors to conduct the field activities, taking into account the data collection period.


A related critical aspect is establishing effective plans for training the staff. The goal is to ensure that all field staff understand PIAAC so well that they can focus on obtaining quality data and high response rates, not just focusing on the mechanics of the work. Countries should provide thorough initial training, including importance of PIAAC, presentation of study to respondents, and techniques for convincing reluctant respondents to participate. In addition, countries should plan to provide ongoing training via telephone and email. Countries should train supervisors on initial case assignment and reassignment strategies for potential/actual nonresponse.


Community/respondent outreach is another critical component in increasing response rates. Countries should plan to develop an outreach plan, design attractive/cohesive set of materials (e.g., letters, brochures, etc.). Countries should make sure to produce sufficient amount of each material. They should obtain media coverage: newspaper/journal articles about PIAAC that interviewers can show/give to respondents. Various endorsements should be obtained from well-known and well-respected authorities and agencies.


Interviewers/supervisors should be trained on effective use of materials. In addition, countries should design a website that could be used by respondents for reference and to legitimise the survey. Countries should establish toll-free telephone number/hotline—to answer questions and concerns raised by survey participants.


The timing of various activities is also very critical. Introductory materials should be mailed 7-10 days prior to respondent contact. Nonresponse conversion letters should be used for reluctant or hard-to-contact/reach respondents in a timely manner.


Each country should develop a non-interview report (NIR) form to collect information about nonrespondents (see Standard 10.3.7 in the TSG). The form can be paper and pencil or automated, and it should contain, at minimum, demographic data on the non-interviewed person, the strength of the refusal (if applicable), problems encountered, comments, and the likelihood of conversion. The NIR can aid in future interview attempts and possibly provide information for a NRB analysis.


The National Sampling Managers (NSMs) should develop plans for monitoring the sample during data collection to ensure the resulting sample has the lowest NRB attainable. NSMs should monitor the sample and increase response rates for groups that have potential to increase NRB (i.e. identify pockets of nonresponse and use methods to bring in initial nonrespondents that are not like respondents [see Section 3.2, Figure 1]). NSMs are encouraged to use the field test process to develop and test their monitoring programs for the main survey.


A critical task that should be completed prior to the start of data collection is identifying and collecting correlated auxiliary variables that are known with a high degree of accuracy for both respondents and nonrespondents. Evaluating and reducing NRB requires auxiliary variables, available for all sampled units, that are correlated with competency and response propensity. The correlation with competency could be measured using past adult competency or similar studies; variables hypothesised to be correlated with literacy should also be included. A segmentation analysis (see section 3.2) could be used to determine correlation with response propensity. NSMs are encouraged to use the field test data to help identify a set of candidate auxiliary variables for evaluating and reducing NRB. More discussion can be found in the sampling Plan Part II forms, specifically form W-1.


The Cross-Cultural Survey Guidelines website (http://ccsg.isr.umich.edu/datacoll.cfm) has additional suggestions for improving response rates and reducing NRB.




3.2 Reduce NRB During Data Collection

National Project Managers (NPMs) should promote close and timely supervision/monitoring and conduct scheduled weekly telephone conference calls with interviewers. They should keep the pressure on performance and allow for timely feedback. Countries should use Consortium forms or country-specific automated reports that allow for regular/frequent monitoring of case completion. Reports must be based on daily interviewer transmission of interview/assessment data and current contact attempt and disposition code information.


It is important that countries seek and be able to implement new ideas or experiences used successfully by other survey organisations/countries prior to and during data collection. It is always helpful to encourage interviewers/supervisors to make suggestions for improving response rates throughout the data collection period.


Countries should use OTRS to ask for ad hoc advice and use quality control monitoring calls to discuss national issues, concerns, and solutions dealing with respondent cooperation. Countries are encouraged to share with the Consortium novel approaches for obtaining respondent cooperation that can be shared with other countries.


The NSMs should use the planned monitoring procedures (as described in Section 3.1) to monitor response rates and sample yield according to key demographics, known to be associated with competency. This will help countries identify potential shortfalls for specific subgroups that could result in biased estimates. The goal is to reach subgroups or areas with low response and use various approaches to improve response rates for these subgroups (areas). The first step is to identify pockets of potential NRB through modelling, for example, segmentation modelling with local area-level auxiliary variables (smallest geography for which reliable auxiliary data is available, or use registry data when available). A typical segmentation modelling approach involves a classification tree algorithm that uses a chi-square test of independence to identify auxiliary variables that best define subgroups with differential response rates. The subgroups with low response rates, or areas containing these subgroups, can then be targeted for followup efforts to address the potential for NRB.


Figure 1 shows an example of a classification tree used for the West region during a U.S. adult competency household survey data collection period.


Figure 1. Example response rates in West region, USA, during data collection


From Figure 1, it is evident that during data collection in the West there was a low response rate (50%) in segments with a high minority concentration, a high percentage of the population not speaking English well, located in metropolitan statistical areas (MSAs), and having a low percentage of the population with less than a ninth grade education. To the extent that these variables were related to literacy, the low response rate in these areas indicated potential for NRB.




3.3 Reduce NRB After Data Collection

As mentioned in Section 1, all countries are required to conduct the basic NRB analysis described in Section 3.3.1. Countries will be required to conduct a more extensive NRB analysis for all stages of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and for the overall sample if the overall response rate is less than 70%. Figure 2 shows the overall plan for PIAAC NRB analysis. The remainder of this section describes the preliminary plans for various analyses mentioned in the figure.



Figure 2. Preliminary plans for PIAAC NRB analysis


3.3.1 Basic NRB Analysis

The main goal is to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustments to the weights. The basic NRB analysis uses auxiliary variable estimates for survey respondents and nonrespondents. The auxiliary variables may be available on the sampling frame or from a previous data collection stage (e.g. screener data for the BQ/JRA analysis), can come from an external source that can be matched to each sampled unit, or could be observational data on respondents and nonrespondents collected during data collection, assuming the data is of sufficient quality. The auxiliary variables must be available for all eligible units and should be related to competency.


An advantage of the basic analysis is that they could be performed using data that are readily available for both respondents and nonrespondents. However, they are only beneficial if auxiliary variables are highly correlated with competency.


The following techniques can be used:


  • Use a chi-square test to compare the distribution of auxiliary variables (correlated with competency) for respondents and nonrespondents. Deviations between the auxiliary survey estimates and higher quality external source estimates may indicate a potential for bias in the survey estimates of competency levels, to the extent that the auxiliary variables are related to the key statistics;

  • Compare response rates for different subgroups;

  • Use a classification tree algorithm to identify subgroups with low response rates; and/or

  • Use logistic regression to model the relationship between response status and the auxiliary variables.


Sample Weighting Adjustments

At the conclusion of the basic NRB analysis, countries will be asked to put together a list of variables found to be significantly related to competency and response status. The goal is to use these variables in developing NRB adjustments. Such adjustments will reduce NRB to the extent that the auxiliary variables and the competency measures are correlated, and the auxiliary variables and response propensity are correlated.


3.3.2 NRB Analysis if Response Rate is Less Than 70%

The basic descriptive analysis is a good initial assessment of NRB and is essential in identifying effective weighting variables. However, it has its limitations. The analysis does not reflect the effect of weighting adjustments on NRB, and the extent of bias remaining after nonresponse adjustments are conducted.


Brief descriptions of some possible analyses are provided below. Note that multiple analyses to assess NRB are necessary because each analysis has its own limitations. Together, they will provide an insight into the patterns and potential for bias.


Comparison of estimates before and after weight adjustments are done - The basic analysis described in section 3.3.1 comparing the base-weighted estimates for respondents to the base-weighted full sample estimates can be extended to include a third stage – the respondent estimates using weights adjusted for nonresponse and calibrated to control totals. In other words, estimates from the full sample can be compared to estimates from the respondents before and after weighting adjustments. If the full sample estimates are closer to the final estimates than the base weighted estimates, this indicates bias in the auxiliary variables was reduced through the weighting process.


Comparison of weighted estimates to external totals - Another extended analysis is to compare estimates from PIAAC to estimates from external source. The PIAAC estimates should be produced using the final weights that have been adjusted for nonresponse and calibrated to control totals. Care should be taken to choose external source estimates that measure the same characteristic for a similar time period. The external source estimates will also be subject to error, and the variance of these estimates should be taken into account when making this comparison.


Correlations between weighting adjustment variables and competency measures - The analyses described thus far rely on auxiliary variables and do not directly measure bias in the competency estimates. As mentioned earlier, bias in the auxiliary variables is indicative of bias in the competency estimates to the extent that the auxiliary variables and competency estimates are correlated. Thus, correlations between the auxiliary variables and competency data can be computed to evaluate this relationship. For variables used in the weighting adjustments, a low correlation with literacy implies that using the variable in the weighting adjustments did little to reduce NRB. On the other hand, a high correlation with literacy implies a potentially high reduction in NRB. For variables not used in the weighting adjustments, a high correlation with literacy indicates potential bias in the literacy estimates. This can be used to inform weighting decisions for future surveys.


The disadvantage of using correlations to evaluate NRB is that the correlations are based on respondents only and the relationship between competency and the auxiliary variables might be different for nonrespondents. However, this is not as much a concern if the relationship can be confirmed using outside sources.


Comparison of estimates using alternative weighting adjustments - For this evaluation, an auxiliary variable is re-calibrated to known totals, and estimates of the key statistics are compared before and after the re-weighting. Re-weighting can be useful as an evaluation tool when:


  • The variable was not used in weighting (because it was of a low quality);

  • The variable is correlated with the outcome measure; and

  • The variable is correlated with response propensity.


Any differences between estimates using the official survey weights and the re-weighted weights reflect noncoverage as well as NRB, but if there is not a large change in the estimates, this is further confirmation that NRB may not be a concern.


Analysis of variables collected during data collection - BQ variables - The descriptive analyses described earlier assess the potential for bias in some statistics that may not necessarily be highly related to competency. Therefore, the next step of the analysis should be to evaluate bias on the key statistics or closely related variables. For PIAAC, the key statistics are scores measuring competency in the components of the assessments that will be available later. However, before literacy scores are available, an analysis can be performed with BQ/JRA variables believed to be closely related to literacy, such as questions on measuring habits (newspapers or magazines), for example. Analyses of such variables may show patterns of bias that cannot be effectively eliminated by the auxiliary variables used in weighting.


Analysis of variables collected during data collection - Disposition codes - Disposition codes contain information on reasons for nonresponse. For this analysis, distributions of sampled persons with known characteristics related to outcome (i.e. the language barrier cases) should be examined. For example, the demographic distribution of language barrier cases can be compared to other eligible persons using auxiliary data, and interview data.


Analysis of variables collected during data collection - NIR - The NIR forms identify observable demographic information and reasons for nonresponse that are not captured in the disposition codes. The NIR forms can potentially indicate whether the reasons for nonresponse are related to competency estimates and suggest ways to improve response rates for future surveys.


Comparison of late or “hard-to-contact” respondents to early respondents - This analysis is helpful for evaluating the potential for NRB for differences that cannot be captured through adjusting for known demographics. Significant differences between the competency levels of early and late respondents imply that the bias could have been potentially reduced through focused efforts to obtain the late respondents. However, to the extent that differences between early and late respondents reflect differences between respondents and nonrespondents, the findings indicate that some level of NRB might still be present (which depends on the magnitude of nonresponse rate). The validity of this assumption, that late respondents are similar to nonrespondents, depends largely on the strategies used in the late data collection effort. For many surveys this may not be a valid assumption, but it seems reasonable for a household survey such as PIAAC where multiple contacts are usually required to gain cooperation.


Calculation of the range of potential bias - How can we evaluate the potential for bias remaining after weighting if nonrespondents are very different from respondents within the weighting classes? The range of potential bias can be evaluated using a deterministic NRB formula that calculates how different respondent and nonrespondents competency estimates need to be to bias the overall competency estimate by varying degrees. For example, one can show that for a response rate of 70% (WR = 0.7), the mean of respondents and nonrespondents would have to differ by 50 points (YR -YNR = 50) to bias the overall estimate by 15 points (Bias( yR) = 15).


Other analysis under review - The Consortium is currently reviewing other NRB analysis procedures, such as the R indicator (Schouten, Cobben, and Bethlehem, 20091), for possible inclusion in the NRB plan for PIAAC.


The following provides a summary of the steps included in the NRB analysis when response rate falls below 70%.

Step 1 of 3: Identify Variables

1.1 Identify auxiliary variables from the frame or external sources.

1.2 Identify informative BQ variables correlated with competency.

1.3 Identify other informative variables collected during data collection.


Step 2 of 3: Basic NRB Analysis

2.1 Conduct basic descriptive and model-based analysis.

2.2 Finalise variables selected for weighting adjustments.

2.3 Conduct nonresponse adjustments and compute final sampling weights.


Step 3 of 3: Extended NRB Analysis

3.1 Conduct extended descriptive and model-based analysis using auxiliary data.

3.2 Conduct analysis of informative BQ variables.

3.3 Conduct analysis of disposition codes.

3.4 Conduct analysis of NIR (information not available in disposition codes).


Example: The following information and tables are summaries extracted from the NRB analysis conducted for the Adult Literacy and Lifeskills Survey (ALL) 2003: U.S. NRB Analysis http://nces.ed.gov/pubs2009/2009063.pdf, and other unpublished NRB analyses conducted for similar adult competency household surveys.


The steps carried out in the following NRB analysis are summarized below.


Step 1 of 3: Identify variables from the frame or external sources

Step 1.1 The following variables were extracted from Census 2000 county-level data, and Census 2000 block group level data.


  • Census variables available for local areas (where respondents and nonrespondents resided)

  • Urban/rural status

  • Majority own/rent

  • Average household size

  • Percentage of persons 25 years or older with less than a high school diploma

  • Percentage of persons speaking English not well or not at all

  • Percentage below 150% poverty


Step 1.2 The following variables were extracted from the BQ.


  • Percentage born in the United States

  • Percentage with high school diploma

  • Percentage never receiving remedial help in school

  • Percentage agreeing with the statement: I enjoyed math in school

  • Percentage speaking English most often at home

  • Percentage employed

  • Percentage reading letters, memos, or emails at least once a week as part of main job

Step 1.3 Disposition codes and NIR variables were collected after the end of data collection.


Step 2 of 3: Basic NRB analysis – Tables 1 through 4 provide examples of analysis conducted for ALL.

Table 1 shows the results of chi-square tests conducted for ALL. The chi-square tests indicated a significant relationship of response status to region, MSA status, the percentage below 150% of poverty, age, sex, and race/ethnicity. These variables were included in the nonresponse adjustments to reduce the potential NRB to the extent that these auxiliary variables were related to literacy.




Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by key characteristics: 2003

Analysis variable

Respondents

percent

Standard

error

Eligibles

percent

Standard

error

Chi-square statistic

p value

Region

 

 

 

 

 

 

Northeast

15.5

2.50

18.3

2.36

31.95

0.000

Midwest

22.9

2.54

22.1

2.24

South

35.0

1.66

34.9

1.50

West

26.6

2.47

24.7

2.21

Metropolitan Statistical Area status







Non-Metropolitan Statistical Area

22.7

2.63

21.8

2.40

7.26

0.007

Metropolitan Statistical Area

77.3

2.63

78.2

2.40

Locale1







Urban

64.6

2.67

65.3

2.45

3.78

0.12

Suburban

13.0

2.00

12.4

1.83

Rural

22.4

2.11

22.3

2.02

Majority own/rent

 

 

 

 

 

 

Rent

25.7

2.48

25.2

2.40

1.14

0.287

Own

74.3

2.48

74.8

2.40

Average household size







less than or equal to 2.2

18.1

1.87

18.6

1.81

4.73

0.081

greater than 2.2 and less than or equal to 2.8

52.9

2.50

51.5

2.80

greater than 2.8

29.1

2.92

29.9

3.10

Percent less than high school






 

less than or equal to 10%

33.1

2.71

33.4

2.73

2.7

0.257

greater than 10% and less than or equal to 20%

30.9

1.81

31.4

1.75

greater than 20%

36.1

3.01

35.2

3.09

Percent limited English proficient







0%

29.9

2.68

29.9

2.51

0.06

0.963

greater than 0% and less than 2%

34.7

3.19

34.6

3.09

greater than or equal to 2%

35.4

3.78

35.5

3.64

Percent below 150% of poverty







less than or equal to 10%

30.7

2.65

31.5

2.79

8.59

0.011

greater than 10% and less than or equal to 30%

44.9

3.27

45.4

3.27

greater than 30%

24.4

2.95

23.1

2.76



Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by key characteristics: 2003 (Continued)

Analysis variable

Respondents

percent

Standard

error

Eligibles

percent

Standard

error

Chi-square statistic

p value

Age






 

16–25

23.7

1.38

22.4

1.25

17.92

0

26–35

21.2

0.80

20.8

0.77

36–45

22.2

0.84

22.9

0.79

46–65

32.9

1.29

33.9

1.16

Sex







Male

48.1

1.08

49.6

0.99

22.77

0

Female

51.9

1.08

50.4

0.99

Race/ethnicity2







Hispanic

14.3

1.79

13.3

1.61

19.11

0

Black, non-Hispanic

11.1

1.82

10.2

1.62

Other, non-Hispanic

74.6

2.08

76.4

1.94

† Not applicable.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.


Table 2 contains estimates of bias and relative bias for the subgroup percentages presented in Table 1. Bias was calculated using the deterministic NRB formula in section 2. A t test was performed to determine whether the NRB was significantly different from zero. Another measure, the relative bias, was computed as the bias divided by the estimate from the eligible sample. The relative bias is a measure of the size of the bias compared to the eligible sample estimate.


The results of the analysis in Table 2 were consistent with the chi-square analysis in Table 1. It also showed that even though there were statistically significant results as described above, the relative magnitude of the differences was small. The relative bias estimates were all less than 10 percent, except in the Northeast, which was 15 percent.


Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics

2003 Analysis variable

Respondents

percent

Eligibles

percent

Bias

Percent

relative bias

Estimate

Standard

error

p value

Region







Northeast

15.5

18.3

-2.8

0.53

0.000

-15.3

Midwest

22.9

22.1

0.8

0.61

0.187

3.6

South

35.0

34.9

0.1

0.49

0.842

0.3

West

26.6

24.7

1.8

0.48

0.001

7.3

Metropolitan Statistical Area status







Non-Metropolitan Statistical Area

22.7

21.8

0.9

0.34

0.010

4.1

Metropolitan Statistical Area

77.3

78.2

-0.9

0.34

0.010

-1.2

Locale1







Urban

64.6

65.3

-0.7

0.38

0.074

-1.1

Suburban

13.0

12.4

0.6

0.37

0.108

4.8

Rural

22.4

22.3

0.1

0.31

0.747

0.4





Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics (Continued)

2003 Analysis variable

Respondents

percent

Eligibles

percent

Bias

Percent

relative bias

Estimate

Standard

error

p value

Majority own/rent

 

 

 

 

 

 

Rent

25.7

25.2

0.5

0.44

0.298

2.0

Own

74.3

74.8

-0.5

0.44

0.298

-0.7

Average household size







less than or equal to2.2

18.1

18.6

-0.5

0.39

0.215

-2.7

greater than 2.2 and less than or equal to 2.8

52.9

51.5

1.4

0.65

0.044

2.7

greater than 2.8

29.1

29.9

-0.9

0.6

0.155

-3.0

Percent less than high school







less than or equal to10%

33.1

33.4

-0.3

0.53

0.540

-0.9

greater than 10% and less than or equal to 20%

30.9

31.4

-0.5

0.48

0.284

-1.6

greater than 20%

36.1

35.2

0.9

0.52

0.112

2.6

Percent limited English proficient







0%

29.9

29.9

#

0.56

0.977

#

greater than 0% and less than 2%

34.7

34.6

0.1

0.69

0.841

0.3

greater than or equal to 2%

35.4

35.5

-0.1

0.55

0.826

-0.3

Percent below 150% of poverty







less than or equal to10%

30.7

31.5

-0.8

0.52

0.129

-2.5

greater than 10% and less than or equal to 30%

44.9

45.4

-0.5

0.58

0.379

-1.1

greater than 30%

24.4

23.1

1.3

0.38

0.001

5.6

Age

 

 

 

 


 

16–25

23.7

22.4

1.3

0.35

0.001

5.8

26–35

21.2

20.8

0.4

0.25

0.106

1.9

36–45

22.2

22.9

-0.6

0.36

0.088

-2.6

46–65

32.9

33.9

-1

0.41

0.016

-2.9

Sex







Male

48.1

49.6

-1.5

0.31

0.000

-3.0

Female

51.9

50.4

1.5

0.31

0.000

3.0

Race/ethnicity2







Hispanic

14.3

13.3

0.9

0.29

0.003

6.8

Black, non-Hispanic

11.1

10.2

0.9

0.31

0.009

8.8

Other, non-Hispanic

74.6

76.4

-1.8

0.32

0.000

-2.4

# Rounds to zero

1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

NOTE: Detail may not sum to totals because of rounding.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.


The analyses in Tables 1 and 2 were useful in explaining the relationship of response status to each auxiliary variable individually. A classification tree algorithm and a logistic regression model were used to evaluate the relationship between response status and multiple auxiliary variables. The results of the classification tree algorithm are shown in Table 3. The algorithm was similar to that described in section 2 and used chi-square tests to divide the sample into subgroups that best explain differential response rates. As Table 3 shows, 22 cells were formed with weighted response rates ranging from 58.7% to 97.4%. The subgroups defined by the classification tree algorithm could be used as nonresponse adjustment cells in the weighting adjustment.



Table 3. Weighted response rates for the ALL background questionnaire, by response cell: 2003

Response

cell

Description

Weighted

response rate (%)

Overall

 

79.7

1

Northeast, Non–Metropolitan Statistical Area (MSA)

88.9

2

Northeast, MSA, Average household size less than or equal to 2.8

68.1

3

Northeast, MSA, Average household size greater than 2.8

58.7

4

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size less than or equal to 2.2

79.8

5

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.2 and less than or equal to 2.8

91.1

6

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.8

78.0

7

Midwest and South, Hispanic or non-Hispanic Black, Female, Age 16-45

94.4

8

Midwest and South, Hispanic or non-Hispanic Black, Female, Age 46-65

82.0

9

Midwest and South; Other, non-Hispanic; Age 16-35; Male; Rent

70.0

10

Midwest and South; Other, non-Hispanic; Age 16-35; Male; Own

82.1

11

Midwest; Other, non-Hispanic; Female; Age 16-25

97.4

12

Midwest; Other, non-Hispanic; Female; Age 26-35

90.2

13

South; Other, non-Hispanic; Female; Percent limited English proficient less than 2%

86.0

14

South; Other, non-Hispanic; Female; Percent limited English proficient greater than or equal to 2%

70.3

15

Midwest; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or equal to 30%

77.3

16

South; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or equal to 30%

71.5

17

Midwest and South; Other, non-Hispanic; Age 36-65; Percent below 150% of poverty greater than 30%

84.7

18

West, Rent

88.6

19

West, Own, Age 16-25

88.7

20

West, Own, Age 26-45

79.6

21

West, Own, Age 46-65, Urban or Suburban

90.5

22

West, Own, Age 46-65, Rural

78.5

NOTE: The response cells were formed using the Chi-square Automated Interaction Detector.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.



Logistic regression models are also useful in identifying significant effects on response propensity. Response status was used as the binary dependent variable and the set of auxiliary variables in Table 1 as the predictors. An F test was performed for each auxiliary variable to determine if it was significantly related to response propensity, after accounting for the other auxiliary variables. Results are shown in Table 4. Average household size, age, race/ethnicity, sex, and region are shown to be potentially valuable variables to include in the nonresponse adjustments. While this analysis was limited to main effects, interaction terms can also be included.




Table 4. Multivariate analysis of response indicators: 2003

Test

F statistic

Degrees of freedom

p value

Numerator

Denominator

Overall fit

3.89

21

10

0.016

Average household size

6.77

2

29

0.004

Majority own/rent

0.32

1

30

0.578

Percent less than high school

0.44

2

29

0.647

Percent limited English proficient

2.08

2

29

0.143

Locale1

0.12

2

29

0.887

Age

4.47

3

28

0.011

Race/ethnicity2

10.79

2

29

0.000

Sex

19.43

1

30

0.000

Metropolitan Statistical Area status

0.03

1

30

0.871

Percent below 150% of poverty

3.22

2

29

0.055

Region

6.65

3

28

0.002

1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.


Step 3 of 3: Extended analysis – Tables 5 through 10 provide examples of extended NRB analysis as conducted for ALL and other adult competency surveys

Table 5 shows an example of the comparison of estimates before and after weighting adjustments. The following t test comparisons were made:

  • Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with the BQ base weights for the BQ respondents only to check for differences due to nonresponse to the BQ

  • Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with that from the BQ nonresponse adjusted weights for respondents to check for differences even after the nonresponse adjustment process to the BQ

  • Comparison of percentage distributions from BQ nonresponse adjusted weights for respondents with that from the BQ raked weights for respondents to check for differences that may have been introduced through the initial raking procedure

  • This analysis showed that, through the nonresponse adjustment, the potential for bias was reduced (p-values moved from significant to nonsignificant) in all domains shown below, with the exception of two domains relating to average household size. These two domains show significant bias created by the nonresponse adjustments. While the differences in the percentage distributions of these domains remain statistically significant even after the raking adjustment, they are not large enough to be meaningful.




Table 5. Percentage distribution of sample cases at each weighting step for the ALL background questionnaire, by key characteristics: 2003

 

Total sample %

BQ respondents

Nonresponse adjustment

Initial raking

%

p value1

%

p value1

%

p value2

Total

100.0

100.0

100.0

100.0

Region








Northeast

18.4

15.6

0.000

18.6

0.355

19.0

0.866

Midwest

22.1

22.9

0.180

21.9

0.355

23.0

0.728

South

34.9

35.0

0.842

34.9

0.825

35.0

0.716

West

24.7

26.6

0.001

24.7

0.895

23.0

0.413

Metropolitan Statistical Area status








Non-Metropolitan Statistical Area

21.8

22.7

0.012

21.9

0.558

18.0

0.100

Metropolitan Statistical Area

78.2

77.3

0.012

78.1

0.558

82.0

0.100

Locale3








Urban

65.3

64.6

0.080

65.1

0.448

68.0

0.097

Suburban

12.4

13.0

0.103

12.4

0.849

11.0

0.198

Rural

22.3

22.4

0.749

22.5

0.566

20.0

0.075

Majority own/rent








Rent

25.2

25.7

0.294

25.0

0.486

26.0

0.254

Own

74.8

74.3

0.294

75.0

0.486

74.0

0.254

Average household size








less than or equal to 2.2

18.6

18.1

0.225

18.1

0.213

18.1

0.946

greater than 2.2 and less than or equal to 2.8

51.5

52.9

0.051

53.3

0.003

52.5

0.257

greater than 2.8

30.0

29.1

0.157

28.7

0.005

29.4

0.381

Percent less than high school








less than or equal to 10%

33.4

33.1

0.542

33.1

0.569

32.7

0.431

greater than 10% and less than or equal to 20%

31.4

30.9

0.292

31.4

0.887

31.3

0.937

greater than 20%

35.2

36.1

0.123

35.5

0.349

36.0

0.524

Percent limited English proficient








0%

29.9

29.9

0.977

30.0

0.953

28.8

0.088

greater than 0% and less than 2%

34.6

34.7

0.840

35.4

0.162

35.5

0.756

greater than or equal to 2%

35.5

35.4

0.825

34.7

0.052

35.8

0.314







Table 5. Percentage distribution of sample cases at each weighting step for the ALL background questionnaire, by key characteristics: 2003 (Continued)

 

Total sample %

BQ respondents

Nonresponse adjustment

Initial raking

%

p value1

%

p value1

%

p value2

Percent below 150% of poverty








less than or equal to 10%

31.5

30.7

0.137

31.6

0.912

32.0

0.629

greater than 10% and less than or equal to 30%

45.4

44.9

0.372

45.3

0.682

45.4

0.814

greater than 30%

23.1

24.4

0.002

23.2

0.452

22.7

0.585

1 The p value and SE are for tests of difference between estimated percentage of current step and total sample (BQ base weights).

2 The p value and SE are for tests of difference between estimated percentage of current and previous weighting steps.

3 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.



Table 6 gives an example of correlations between auxiliary variables and competency score measures for an adult competency survey. The percentage aged 25+ with less than a high school education was found to be highly correlated with literacy score. If this variable was also related to response propensity, using it in nonresponse adjustments should reduce the potential NRB of literacy estimates.





Table 6. Correlations of demographic variable with the outcome statistic

Variable

Absolute correlation with outcome statistic

<0.1

0.1-0.2

0.2-0.3

0.3-0.4

>0.4

Percent aged 25+ with less than high school education





X

Percent of adults speaking Spanish at home and English not well or not at all




X


Percent below 150% of poverty




X


Median Income



X



Percent who rent



X



Average household size


X




Age


X




Percent minority

X





SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.



Table 7 provides an example of the analysis of disposition codes to evaluate NRB in an adult competency survey. The distribution of cases with a BQ disposition code of language problem or mental disability was compared to the distribution of the other nonrespondents. Chi-square tests were processed to determine if there was a relationship between select demographic variables and the disposition code groups (literacy-related or other) for incompletes. Significant differences in the distribution by region, MSA status, and age were found between the two groups. These variables were used in the weighting adjustments, which are expected to reduce the potential bias from literacy-related nonresponse.




Table 7. Distribution of BQ incompletes, by disposition code category

Domain

Literacy-related2

other

Total incompletes

355

5,191

Domain

Incompletes1

Statistic

p value

Literacy- related2 percent

OTHER percent

Total incompletes

100

100


 

Region



10

0.006

Northeast

16

17



Midwest

18

22



South

21

38



West

45

23



MSA status



9

0.002

Non-MSA

9

18



MSA

91

82



Age



23

0.000

16-29

16

20



30-49

37

43



50-69

23

27



70+

24

11



Gender



3

0.067

Male

47

53



Female

53

47



Race/ethnicity



2

0.244

Hispanic

17

11



NH Black only

6

9



Other

77

79


 

Not applicable

1 Nonrespondents

2 Literacy-related reasons for nonresponse at the BQ stage consist of language problems and mental disabilities only.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.


An example of an analysis of NIR data for an adult competency survey is provided in Table 8. The table summarizes data collected on the NIR regarding the household race of screener incompletes, as observed or gathered from neighbour information. The base-weighted distribution is shown overall and by region. For households where race was determined, Hispanic households were less common among screener incompletes compared to the population proportion. This suggests a possible increase in response rates from accommodations made for Spanish speakers, such as bilingual interviewers.






Table 8. Household race for screener incompletes with NIR data, by region

 

Household race

 

Overall

Region

Northeast

Midwest

South

West

Total

4,276

1,327

819

1,571

559

Not ascertained

2,197

548

535

804

310

 

Percent

Percent

Percent

Percent

Percent

Total

100

100

100

100

100

White

73

73

84

70

67

Black

10

6

7

15

7

Hispanic

4

3

4

3

8

Asian

4

4

1

2

9

NH/PI1

0

#

#

#

1

AI/AN2

0

#

#

0

#

Other

0

1

#

0

1

Cannot determine

10

14

5

10

8

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.


Table 9 shows an example of a comparison of literacy estimates for late versus early respondents. Calculations were done using marginal maximum likelihood (MML) means. The results indicate a small but significant bias in Statistic 2 if focused followup efforts (as describe in section 2) had not been successful in obtaining responses from respondents at the end of the data collection period (i.e. if data about respondents at the end of the data collection period had been missing). This suggests some small level of NRB may be present to the extent that nonrespondents are similar to respondents at the end of the data collection period.

Table 9. Differences between the last 10% and first 90% of respondents, by domain

Domain

Estimate of population proportion

Statistic 1

Statistic 2

Statistic 3

Proportion

among last 10%

Proportion

among first 90%

p value

Relative difference

p value

Relative difference

p value

Relative difference

p value

Overall

1.00

1.00

0.00

0.92

-0.03

0.01*

0.00

0.80

Region










Northeast

0.20

0.19

0.82

0.00

0.89

-0.03

0.12

0.01

0.79

Midwest

0.18

0.23

0.03

0.03

0.22

0.01

0.58

0.03

0.08

South

0.36

0.36

0.95

0.01

0.64

-0.02

0.36

0.00

0.96

West

0.27

0.21

0.02

-0.03

0.13

-0.06

0.01*

-0.01

0.61

MSA status










Non-MSA

0.13

0.19

0.04

0.00

0.95

-0.02

0.38

0.02

0.38

MSA

0.87

0.81

0.04

0.00

0.97

-0.03

0.01*

0.00

0.88

Gender










Male

0.53

0.48

0.01*

0.01

0.50

-0.02

0.22

0.00

0.94

Female

0.47

0.52

0.01*

-0.01

0.70

-0.04

0.01*

0.00

0.84

Race/ethnicity










Hispanic

0.17

0.12

0.00*

-0.02

0.35

-0.07

0.00*

-0.01

0.55

NH Black only

0.11

0.11

0.51

0.01

0.57

0.02

0.41

0.06

0.03

Other

0.73

0.77

0.04

0.01

0.40

-0.01

0.20

0.01

0.51

Educational attainment










Less than high school

0.18

0.19

0.33

-0.05

0.01*

-0.07

0.00*

-0.02

0.33

High school or equivalent

0.28

0.31

0.10

-0.02

0.50

-0.04

0.02

0.00

0.82

More than high school

0.54

0.50

0.06

0.00

0.99

-0.03

0.01*

-0.01

0.25

Age category










16-29

0.25

0.25

0.68

-0.01

0.75

-0.02

0.22

0.01

0.79

30-49

0.42

0.39

0.06

-0.01

0.35

-0.04

0.0*

0.00

0.87

50-69

0.25

0.25

0.92

0.02

0.25

0.00

0.85

0.00

0.96

70+

0.09

0.11

0.07

-0.03

0.53

-0.10

0.01*

-0.07

0.11

Country of birth










US

0.80

0.86

0.00

0.01

0.44

-0.01

0.32

0.01

0.29

Other

0.20

0.14

0.00

0.00

0.97

-0.05

0.04

0.00

0.97

Not applicable

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

An example of the calculation of the range of potential bias in literacy estimates is shown in Table 10. The table shows the difference between respondent and nonrespondents literacy scores that would bias the population estimate by b = 3, 5, 10, or 15 points. The required difference between respondents and nonrespondents was computed as described earlier in section 3.3.2. For the overall sample, literacy scores of respondents and nonrespondents would have to differ by about 8 points on average to bias the national estimate by 3 points and would need to differ by 39 points to bias the national estimate by 15 points.


Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics

Domain

Percent of population1 represented by nonrespondents2

Average point difference between respondents and nonrespondents needed to bias estimated scores by

3

5

10

15

Total

38

8

13

26

39

Region






Northeast

43

7

12

23

35

Midwest

35

9

14

28

43

South

38

8

13

26

40

West

38

8

13

26

39

MSA status






Non-MSA

33

9

15

30

45

MSA

40

8

13

25

38

Average household size






2.42

39

8

13

26

39

(2.42, 2.8]

39

8

13

26

39

> 2.8

38

8

13

27

40

Percent less than high school






10.4%

43

7

12

23

35

(10.4%, 20.3%]

39

8

13

26

38

(20.3%, 32%]

35

8

14

28

42

> 32%

31

10

16

32

48

Percent speaking Spanish but not English






0%

39

8

13

26

38

(0%, 28%]

40

8

13

25

38

> 28%

34

9

15

29

44

Percent below 150% of poverty






10.7%

43

7

12

23

35

(10.7%, 20%]

40

8

13

25

38

(20%, 33.3%]

35

9

14

29

43

> 33.3%

29

10

17

34

52

Median income






28,400

28

11

18

36

53

(28,400, 37,850]

34

9

15

29

44

(37,850, 52,100]

40

8

13

25

38

> 52,100

44

7

11

23

34





Table 10. Average differences between respondents and nonrespondents needed to achieve various amounts of bias in outcome statistics (Continued)

Domain

Percent of population1 represented by nonrespondents2

Average point difference between respondents and nonrespondents needed to bias estimated scores by

3

5

10

15

Percent rent






16%

42

7

12

24

36

(16%, 31%]

37

8

13

27

40

(31%, 59%]

37

8

14

27

41

> 59%

35

9

14

29

43

Low response rate CHAID cells






Cell 13

59

5

8

17

25

Cell 24

51

6

10

20

30

Cell 35

50

6

10

20

30

1 The population excludes language problems and mental disabilities.

2 Nonrespondents to the Screener and BQ.

3 Median income greater than 52,100, average household size 2.42 or less, Northeast region

4 Median income 37,850 - 52,100, percent less than high school greater than 20.3, Northeast region

5 Median income greater than 52,100, average household size 2.42 - 2.8, South or West region

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.





4. Summary

This document presented the main goals of PIAAC for reducing NRB throughout the survey process. A series of procedures for implementation before and during the data collection were described to help increase the response rates with the goal of reducing NRB to the extent possible. Preliminary plans are provided to conduct basic NRB analysis to allow an effective weighting adjustment plan that reduces the bias to the maximum extent possible. Then extended analyses are described with the goal of assessing the bias remaining in the estimates after nonresponse adjustment is completed. In addition to assessing the remaining bias in outcome statistics, the results of the NRB analysis can be used to inform analysts of data limitations, and can also help in future rounds of the survey in defining the target population and improving data collection methods, weighting, and estimation approaches.


The PIAAC Consortium is continuing its research and evaluation of various NRB analysis methods and welcomes comments and suggestions from participating countries on such analyses.


1 Schouten B., Cobben F., and Bethlehem J. Indicators for the representativeness of survey response, Survey Methodology, June 2009.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title8570.10.01.11: PIAACReducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis
AuthorLeyla Mohadjer
File Modified0000-00-00
File Created2021-01-29

© 2024 OMB.report | Privacy Policy