Non-response Bias Analysis and Evaluation Reporting

Non-response Bias Analysis and Evaluation Reporting
Passport Demand Forecasting Study

11/14/2013

 

Agenda

• •Current Reporting Status 

• •Overall Non-response Rates 

• •Factors Associated with Non-response 

• •Actions Taken to Address Non-response Bias 

• •Non-response Rates by Passport Agency/Center 

• •Current Options for Improving Case Response Rate 

• •Potential Actions 

1

 

Current Reporting Status

• •Every month, the LMI Team provides the “Technical Report for Weighting for Month Passport Study”. 

• • –The report covers the procedures by which the LMI Team constructed the sample design, generated initial sampling weights, generated a nonresponse adjustment, imputed values for missing responses for design-critical variables, and generated a post-stratification adjustment. 

1

 

Overall Non-response Rate

• •The overall non-response rate for the Passport Demand Forecast Study is 88%. 

• •This is a good non-response rate. 

• • –Non-response rates for telephone surveys are typically on the order of 91% (Kohut, Keeter, Doherty, Dimock, and Christian 2012). 

Month	Sample	Non-response Count	Non-response rate
Total	180,000	158,489	88.0%
June	40,000	35,432	88.6%
July	35,000	30,828	88.1%
August	35,000	30,991	88.5%
September	35,000	30,637	87.5%
October	35,000	30,601	87.4%

<number>

 

Factors Associated with Non-response

• •Several consistent “bias” features appear in the reports: 

• • –Addresses that cannot be matched to a phone listing (57% of the samples) are less likely to respond, which is in part a consequence of the survey design to follow-up non-responses with up to 10 phone calls to get participation by persons at sampled addresses with a phone match; 

  • –Addresses occupied by renters or persons of unknown ownership status are less likely to respond, particularly if the address cannot be matched to a phone listing; 

  • –Addresses with only 1 adult or an unknown number of adults in the household are less likely to respond; 

  • –Addresses where there is no age information on the head of household are less likely to respond; 

  • –Addresses in certain U.S. Census divisions are less likely to respond, particularly in the East-South Central, West-South Central, South Atlantic divisions. 

<number>

 

Takeaway

• •The behaviors that lead people to have their phone numbers and personal information in publicly available databases are associated with their likelihood of responding to the Passport Demand Forecast Study. 

• • –The more information available about persons residing at an address, the more likely the persons at that address are to respond to the Passport Demand Forecast Study. 

<number>

 

Actions Taken to Address Non-response Bias

• •The LMI Team has weighted information from the responding addresses as a function of the degree of non-response from addresses that share similar known characteristics (e.g., presence/absence of phone match, age of head of household, Census Division, etc.). 

• • –The more non-responses for a particular characteristics set, the greater the weight assigned to each response for that particular characteristics set (i.e. non-response adjustment class approach) 

• •After the non-response weights are assigned, a further calibration aligns the projected counts with known population totals for U.S. residents over the age of 18. 

  • –The percentage of addresses with listed phone numbers is included in the final calibration models for household level weights to smooth out the impact of the larger non-response adjustments for households with no phone matched phone number. 

<number>

 

Non-Response by Passport Agency/Center

Month	3 Highest Non-response Agencies/Centers	3 Lowest Non-response Agencies/Centers
July	Atlanta, Dallas, Los Angeles	Minneapolis, Colorado, National Processing Center
August	Atlanta, Houston, Los Angeles	Minneapolis, Buffalo, Vermont
September	Detroit, San Francisco, Vermont	Houston, Dallas, Seattle
October	TBA	TBA

• •There does not appear to be any consistent pattern of high or low levels of non-response for any passport agency/center. 

• • –The geographic non-response pattern seen is at the Census Division level, and moderated by the address having a phone match 

<number>

 

Current Options for Improving Case Response Rate

• •Per the original OMB application, the LMI Team can address case non-response by adjusting the:  

• • –Invitation  

• • –Follow-up calls 

<number>

•envelope  •letterhead   •content of letter

•salutation  •timing of mailing

 

Potential Actions

• •Item (question) non-response rate analysis 

• •Recommended question adjustments to improve item response rates 

• •Summation of case non-response analysis and item non-response analysis as report to CA/PPT by 2/21/2014 

• •Potential requirement by OMB for a non-response follow-up survey to determine additional characteristics of non-respondents and whether responses from non-respondents are different than respondents  

• • –When survey response rate is below 80% OMB often requires a follow-on survey (Harris-Kojetin 2004, Graham 2006) 

  • –This generally requires contacting a sample of non-respondents, incentivizing (paying) non-respondents to respond to the survey, and analyzing the responses for differences with the main (non-incentivized) survey 

• • • •Current OMB paperwork states study is not incentivized for any of the survey participants 

<number>

 

File Type	application/vnd.openxmlformats-officedocument.presentationml.presentation
File Modified	0000-00-00
File Created	0000-00-00