VE BVA Survey Sample Design

Board of Veterans Appeals survey

methodology brief: sample design

Introduction

The Board of Veterans Appeals (BVA) administered by the Department of Veterans Affairs (VA) determines whether Veterans are entitled to claimed benefits which have initially been denied or reduced. Its mission is to conduct hearings and pass proper judgement on appeals in a timely manner. The vast majority of appeals involve claims for disability compensation, in addition to other veteran benefits. BVA allows Veterans to submit medial and lay evidence at any point during the appeals process. Veterans Law Judges will eventually review benefit claims determinations made by local VA offices and issue decisions on appeals.

The Veteran Experience Office (VEO) has been commissioned to measure the satisfaction of Veterans or their appellants regarding the VA’s decision on their benefit claims. VEO proposed to conduct a brief transactional survey on veterans who recently received an appeals decision or attended an appeals hearing. Some appellants may receive more than one decision, and their satisfaction with the most recent appeals decision will be inquired within the survey. The survey is completed online using email recruitment. The purpose of this document is to outline the sample design and provide a description of the data collection and reporting.

Table 1. Measurement Goals and Survey Mode

Survey Type	Preferred Mode of Data Collection	Recruitment Method	Time After Transaction	Recruitment
Appeals Decision	Online Survey	Email Recruitment	Within a Month after Receiving an Appeals Decision	2 Emails over 2 Weeks
Appeals Hearing	Online Survey	Email Recruitment	Within a Week after Completing an Appeals Hearing	2 Emails over 2 Weeks

Target Population and Sample Size Determination

The target population of the BVA survey is all veterans who have received an appeals decision within the past month or completed an appeals hearing within the past week. Selected respondents will have 14 days to complete the survey, with an email reminder after 7 days if the survey has not been completed. The goal of the VEO survey operation is to continuously provide highly reliable monthly overall estimates.

Table 2A depicts the approximate BVA monthly decision population, as well as the sample size determination with Confidence Level (CL) and Margin of Error (MOE) at various levels. The sample size numbers listed indicate the number of responses from veterans required to attain the stated level of precision. Based on the current decision population estimate (6,000), the email availability (30%), and the expected survey response rate (20%), the ideal responses received from veterans would be around 360, if all veterans qualified are surveyed. However, this number is still much smaller (360 vs. 907) than the standard level of precision corresponding to a 95% CL and 3% MOE (Lohr, 1999). Therefore, it is necessary to sample all available veterans to produce quality results at the 90% CL and 5% MOE.

In the BVA hearing population, the expected respondents with 100% sampling rate are still below any of the sample sizes at proposed CL and MOE, due to the small hearing population size. Thereby all veterans in this population will be sampled as well.

Table 2B provides the weekly and monthly sample targets for the decision and hearing populations, presuming a return rate of 20%. The current method obviates the need for any stratification or formal probability sampling.

Table 2A. Target Population Figures

Population Type	Approximate Monthly Population	Approximate Monthly Contact	Expected Yield at 100% Sampling Rate	Precision Sample: 95% CL, 3% MOE	Precision Sample: 90% CL, 5% MOE	Precision Sample: 90% CL, 8% MOE
Decision Population (Received an Appeals Decision)	6,000	1,800	360	907	258	−
Hearing Population (Completed an Appeals Hearing)	1,400	420	84	606	226	98

Table 2B. Proposed Sample Targets by Time Period

Population Type	Weekly Target	Monthly Target
Decision Population	63	250
Hearing Population	25	100
Total	88	350

Weighting Adjustments

Many survey practitioners recommend the use of sample weighting to improve inference on the population. Under this process, the respondent sample is artificially made to more closely resemble the true population, with respect to age. It is reported earlier that email population comprises 30% of the overall veteran population and 20% veterans usually respond to the survey, weighting will be used to adjust for non-coverage (for the non-email population) and non-response (occurring when certain subpopulations are less prone to participate). The weighting variable will be the respondent’s Age.

Reporting and Quality Control

The sample sizes reported above adhere to monthly reporting. To ensure the prevention of errors and inconsistencies in the data and the analysis, quality control procedures will be instituted in several steps of the survey process. Records will undergo a cleaning during the population file creation. The quality control steps are as follows.

1. Records will be reviewed for missing sampling and weighting variable data. When records with missing data are discovered, they will be either excluded from the population file or put into separate strata upon discussion with subject matter experts.

2. Any duplicate records will be removed from the population file.

3. Invalid emails will be removed.

The survey sample loading and administration processes will have quality control measures built into them.

1. The survey load process will be rigorously tested prior to the induction of the BVA Survey to ensure that customers are not inadvertently dropped or sent multiple emails.

2. The email delivery process is monitored to ensure that bounce-back records will not hold up the email delivery process.

The weighting and data management quality control checks are as follows:

1. The sum of the weighted respondents will be compared to the overall population count to confirm that the records are being properly weighted. When the sum does not match the population count, weighting classes will be collapsed to correct this issue.

2. The unequal weighting effect will be used to identify potential issues in the weighting process. Large unequal weighting effects indicate a problem with the weighting classes, such as a record receiving a large weight to compensate for nonresponse or coverage bias.

Quarantine Rules

VEO seeks to limit contact with Veterans as much as possible, and only as necessary to achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts upon survey participants. VEO also monitors veteran participation within other surveys, to ensure veterans do not experience survey fatigue. All VEO surveys offer options for respondents to opt out, and ensure they are no longer contacted for a specific survey.

Table 3. Proposed Quarantine Protocol

Quarantine Rule	Description	Elapsed Time
Repeated Sampling for BVA Survey	Number of days between completing online survey, and receiving another online survey related to another complaint.	3 Months or 90 Days
Other Surveys	Veterans engaged that have recently completed other VEO surveys will not be selected for 30 days.	1 Month or 30 Days
Anonymous	Callers explicitly wishing to remain anonymous will not be contacted.	N/A
Opt Outs	Persons indicating their wish to opt out of either phone or online survey will no longer be contacted.	N/A

3