Terms of Clearance_Response note from Michael Kashner_VHACO

Terms of Clearance: Learner’s Perception Survey, 2900-0691

Michael Kashner Notes

This document is in reference to the non-response bias analysis requested by OMB. OMB made a second request for the lead Statistician to expound on their previous response, by providing more detail. 

Response note from Michael Kashner (VHA 14)

Response bias is difficult to compute, because the bias equals the difference between results that would be obtained with a 100% response rate from results actually obtained from the observed sample.   The estimate that we computed below compared adjusted scores for facilities with high response rates against adjusted scores among facilities with low response rates.  These scores are adjusted for the respondent’s gender, year graduated, discipline and specialty, academic level or rank (master, doctoral) mix of patients seen based on seven patient categories, and an instrument that is designed to measure response bias.  The response instrument is called “grumpiness” because it is designed to measure the extent to which a responder reports above or below the average among all responders whose collective response would represent a “zero bias.”   The response instrument is computed by subtracting the respondent’s answers to three important questions: their satisfaction with parking, with computers, and with the facility location.  For a given facility, responses should not vary because all respondents at a given facility are perceiving the same thing.  However, they will vary because some responders will over-estimate their responses, and others will under-estimate their responses.  We then compare each respondent’s mean response to the instrument questions and compare that with the facility average over all responders.  The instrument is used as a predictor to predict respondent answers to critical domains, including teaching, personal, and clinical training experiences.  The instrument is also used to compute scores that are adjusted to reflect differences in responders.  This is described in the attached papers.  We then compare the “adjusted scores” that were adjusted to account for the instrument bias plus other factors, with actual scores that were adjusted for other factors, but not the instrument.  The difference is an approximation of a response bias, which we estimated at about 4%.  The instrument was in fact highly predictive of responses, with a responder who reports one level higher on a 5-point Likert scale over the facility-level mean for instrument questions was found to be 2.5x (p<.001) more likely to report a higher level of satisfaction for their training experience and clinical environment domains than their counterparts.