PIACC Nonresponse Bias Analysis Summary

PIAAC
Nonresponse Bias Analysis (NRBA) Plan Summary

Country	United States
Date	20 June 2011

Analysis Variables

What variables are available for both respondents and nonrespondents and can be used in analysing nonresponse bias?		Data Source (e.g. registry, Census, screener)	Potential NRB analysis variables (e.g. age, % below poverty in PSU)
	1	Census	Region, metropolitan status
	2	American Community Survey 2005-2009 tract-level data	% Hispanic, % Black, % foreign born, % at least high school (HS) education, % HS, % some college, % below poverty, average household size, % linguistically isolated, % speaking English not well or not at all, % employed, % unemployed, % own housing unit
	3	Screener (for BQ NRBA)	Age, gender, race/ethnicity, language of screener, number of persons age 16-65 in household. Also possibly total household size and indicator for whether there are children in the household.
	4
	5
How will you evaluate the relationship of the analysis variables to proficiency (e.g. variable selection toolkit, past experience)?	We will use the PIAAC Consortium Variable Selection Toolkit to run a regression tree and/or regression model with field test data, using standardized logit scores (similar to proficiency scores) as the dependent variable and variables listed above as the predictors. We will also review 2003 National Assessment of Adult Literacy (NAAL) and 2003 Adult Literacy and Lifeskills (ALL) survey for characteristics related to literacy.

Basic NRBA*

Describe the process to select variables for nonresponse weighting adjustments.	We will use a classification tree to perform a multivariate analysis of the relationship between response status and proficiency-related variables, as determined above. Variables found to be related to both proficiency and response status will be used to form the nonresponse adjustment cells.
B1: Response rates B2: Chi-square test B3: Classification tree B4: Logistic regression Do you foresee any difficulties with producing the 4 required basic analyses for unit nonresponse bias (listed above), which should incorporate base weights and replicates (or variance strata/variance units) to properly account for the sample design? If so, please explain.	No

Extended NRBA*

Do you anticipate an overall response rate of less than 70% or a response rate under 80% for any stage of data collection? If Yes, answer the remaining “Extended NRBA” questions. If No, skip to “Other Unit NRBA”.	It is possible the overall response rate will be below 70% or the response rate for a particular stage below 80%. Therefore, we will respond to the questions on the extended NRBA. We will conduct our procedures to meet both US NCES and PIAAC standards.
E1: Comparison of estimates before and after weighting adjustments How will you evaluate the effect of weighting adjustments on estimates of your analysis variables?	We will compute estimates at each stage of weighting and flag ones that are over 2 standard errors away from the estimate at the previous weighting stage. Flagged estimates will be reviewed as to the reason for the difference.
E2: Comparison to external totals	Possibilities include:
What external estimates are available for comparison with final weighted PIAAC estimates? Note that these should be variables that were not used in calibration or that have different categories.		Data Source	Estimates
	1	2011 or 2012 American Community Survey	Employment status and occupation category. Also, we will compare totals by education and age, with different categories than were used in calibration.
	2	2010 Census	Census division
	3	Other	Also considering: ever used a computer, language spoken at home, income category
b.) How will you perform this comparison?	T-tests will be used to compare estimates from PIAAC and the external source.
E3: Correlations Do you foresee any difficulties with computing correlations of your analysis variables and proficiency scores? If so, please explain.	No
E4: Alternative weights Do you foresee any difficulties with comparing proficiency estimates under alternative weighting adjustments?	No
E5: Analysis of variables collected during data collection
a.) Do you foresee any difficulties with conducting a chi-square test to compare characteristics of literacy-related cases to other nonrespondents?	No. This analysis will be similar to the 2003 NAAL Analysis of ESL Nonparticipants. Weighted estimates of the analysis variables will be computed for the comparison groups. For sampled persons, base weights will be used. For nonresponding households, the base weights will be adjusted for unknown eligibility to represent only the eligible population, and then a person-level weight will be created by applying an imputed within-household selection probability. We will use a Rao-Scott (RS3 in WesVar) chi-squared test of independence to test whether the distribution of the characteristics of literacy-related cases is significantly different from the distribution for the other nonrespondents.
b.) What additional information (not listed under “Analysis Variables”) was collected during data collection that can be used to evaluate nonresponse bias (e.g. Non-Interview Report data, disposition codes)? What analysis will be performed with this data?	The Non-Interview Report (NIR) Form contains a question on language spoken for cases that were unable to complete the survey because of a language barrier and a question on the reason for refusals (e.g. too busy). Simple frequencies of these variables will be produced to learn more about the reasons for nonresponse and could possibly indicate improvements to data collection strategies for the next round of PIAAC. The NIR also includes a question on the socio-economic level (e.g. affluent) of the household based on the interviewer’s observation of the external appearance of the housing unit. This will be compared against the similar ZZ question for BQ respondents.
E6: Level-of-effort How will you define level-of-effort (e.g. number of contacts)?	Number of contacts until completion
Describe your procedure for analysing proficiency estimates by level-of-effort.	We will calculate and plot the average proficiency score by the number of contact attempts until completion. Assuming nonrespondents are similar to those with a large number of contact attempts, this could give an indication of possible nonresponse bias.
E7: Range of potential bias Do you foresee any difficulties with completing the workbook in the Range of Potential Bias toolkit? If so, please explain.	No

Other Unit NRBA

Describe any additional unit nonresponse bias analyses that will be performed (e.g. R indicator, nonresponse follow-up).

N/A

Item NRBA*

Do you have any questions on how to compute item response rates? If so, please indicate them here.	No
Do you know how many BQ items had a response rate below 85% in the field test? If so, how many? Do you expect a similar number in the main study?	We have not computed this for the field test, but in the 2003 Adult Literacy and Lifeskills (ALL) survey, there were 8 BQ items with response rates below 85% (of which 7 were related to income). We expect similar results for PIAAC.
Do you foresee any difficulties performing the 4 required item NRBA analyses (listed below)? If so, please explain. I1: Response rates I2: Chi-square test I3: Classification tree I4: Logistic regression	No

Within your country, are you required to perform a nonresponse bias analysis? If so, what is the timeline for that analysis?	Yes, the national nonresponse bias analysis is due 30 April 2013. We plan to follow the timeline below: 30 June 2011 Provide the Consortium with a draft summary of plans for the NRBA April 2012 Begin Basic NRBA 31 May 2012 Deliver weighting variables in the SDIF Dec 2012 Begin item NRBA and preparations for extensive NRBA Jan 2013 Receive preliminary scores from the Consortium 31 Mar 2013 Provide NCES with draft NRBA report for review 30 April 2013 Deliver NRBA report to the Consortium
Your nonresponse bias analysis report should be provided to the Consortium by 30 April 2013. Do you foresee any problems with meeting this deadline? If so, please explain.	No, but the completion of the nonresponse bias analysis is dependent upon receipt of the respondent proficiency scores in January 2013.
Are there any planned deviations from the nonresponse bias analysis standards and guidelines? If so, please explain.	No
Are there any other anticipated problems or questions? If so, please describe.	No

* For an explanation of the basic, extended, and item nonresponse bias analyses, refer to the slides from Sessions 12 and 14 of the December 2010 Sampling Workshop, located at https://piaac.ets.org/sites/piaac/npms/sampling/December2010_Sampling_Workshop/Forms/AllItems.aspx.