Note on Sample Size Determination for Veterans Complaint Resolution Experience

Note on Sample Size Determination for Veterans Complaint Resolution Experience Survey

Question: Veterans are encouraged to issue any complaints they may have with the VA’s many valuable services. On average, there are 750 entrees filed per week within the Complaint Management System. How many of these veterans should be contacted to get a precision estimate of the proportion of complaints that were resolved?

Response: We recommend a quarterly survey where the data is continuously collected each week over a 3-month period. Presuming adequate numbers of persons with a valid email address, the weekly sample target adjusted for non-response is 530 veterans.

Details: Sample size determination is subjectively based on the amount of precision that is required. This is a function of the confidence level and margin of error that are set by the investigator. The industry standard among survey practitioners is to use 95% as the confidence level and 3% as the margin of error when trying to measure a proportion. In practice, there are two phenomena that must also be taken in to account when estimating sample size. Firstly, not all persons who filed a complaint will have an email address (under-coverage). Secondly, only a fraction of persons that are contacted will respond to the email message and complete the survey (non-response). Based on prior email surveys on the veteran population conducted by the VEO office, we expect the response rate to be approximately 15%.

The table below depicts the amount of sample required to establish precise estimates on complaint resolution (see column Precision Sample Size). Another column adjusts this amount to account for non-response (see column NR Adjusted Sample Size). This information is provided for 3 different reporting levels: Weekly, Monthly, or Quarterly. Note that only quarterly-level reporting is feasible (Population > Adjusted Sample Size), due to the expected amount of non-response (15%). We have not included any adjustments for veterans within the population that do not have a valid email address. It is implicitly assumed that the resolution rate is independent of veteran email status (no non-coverage bias).

T able 1. Veterans Filing Complaints: Average Population and Sample Sizes by Reporting Period

We recommend sampling at least 530 = 6,367/12 veterans each week over 3 months to guarantee a precise estimate at the quarterly level. This presumes that at least 70% of the veterans filing complaints will have a valid email address on record with the VA. If the amount of weekly emails is below 530, then all available email addresses should be sampled so that a precise estimate can be attained after a sufficient number weeks of data collection have elapsed. The following link provides context on the population size and the corresponding sample size required for a precise estimate at the 95% confidence level w/ 3% margin of error: https://www.surveymonkey.com/mp/sample-size-calculator/.

Finally, transactional surveys are generally more effective when offered to respondents shortly after the transaction (in this case, filing a complaint) to maintain cognitive recall. On the other hand, respondents should not be surveyed until ample time has passed to allow for the complaint to be addressed. The survey administrator will need to be conscious of these factors when determining the timing of the contact with the veterans.

William R. Waldron

Data and Analytics Team (VEO)

[email protected]

Source: Veteran’s Experience Office

September 7, 2017