BVA_SamplingPlan

Board of Veterans’ Appeals survey

methodology brief: sample design

Introduction

The Board of Veterans’ Appeals is a tribunal within the Department of Veterans Affairs (VA) which determines whether Veterans are entitled to claimed benefits which have initially been denied or reduced. Its mission is to pass proper judgement on appeals in a timely manner, conducting hearings as needed. The vast majority of appeals involve claims for disability compensation, in addition to other Veteran benefits. The Board allows Veterans to file an appeal of their case following a decision from the Veteran Benefits Association (VBA). If the file to appeal is approved, the Veteran will present their case information to a Veterans Law Judge (VLJ).The VLJ will review the case and determine if the claim is granted or remitted back to BVA.

Beginning in February of 2019, the Board will begin the rollout of the Appeals Modernization (AMA) system. The goal of the AMA program is to provide a more streamlined system in which Veterans can file an appeal and to reduce the overall time to completion. Veterans filing an appeal after February of 2019 will be automatically entered into the AMA system. Those currently in the legacy system will have the option to elect AMA instead. In the fall of 2018, a select group of Veterans will enter a trial program for the AMA system called Rapid Appeals Modernization Program (RAMP).

The Veteran Experience Office (VEO) has been commissioned to measure the satisfaction of Veterans regarding the appeals process. VEO proposed to conduct three brief transactional surveys on Veterans who have recently filed an appeal, attended a hearing, or received a Board decision. The purpose of the filing survey aims to understand if the appeals enrollment process is clear and easy to understand for Veterans. The hearing survey measures Veterans’ interaction with the VLJ, the helpfulness of staff, and the transparency of the appeals process. Finally, the decision survey is focused on whether Veterans perceive their supporting evidence was thoroughly reviewed, the decision language was easy to comprehend, and the final decision was justly determined. Each of the three survey types has been written to apply to both AMA and Legacy participants.

The surveys will be completed online using email recruitment. The purpose of this document is to outline the sample design and provide a description of the data collection and reporting.

Table 1. Measurement Goals and Survey Mode

Survey Type	Preferred Mode of Data Collection	Recruitment Method	Time After Transaction	Recruitment Period
Filing Survey	Online Survey	Email	Within a Week after a filing is processed	2 Week Period, with a reminder after 1 week
Hearing Survey			Within a Week after Completing an Appeals Hearing
Decision Survey			Within a Week after Receiving a Board Decision

Target Population and Sample Size Determination

The target population of the Board of Veterans’ Appeals survey (the Board survey) is all Veterans who have filed an appeal, attended an appeals hearing, or received a Board decision within the past week. Selected respondents will have 14 days to complete the survey, with an email reminder after 7 days if the survey has not been completed by that time. The goal of the VEO survey operation is to continuously provide reliable monthly overall estimates.

Table 2A depicts the approximate monthly Filing, Hearing, and Decision populations, as well as the sample size determination with Confidence Level (CL) and Margin of Error (MOE) at various levels. The sample sizes listed indicate the number of responses from Veterans required to attain the stated level of precision.

Table 2B provides the weekly and monthly sample targets for all three target populations, as well as the number of Veterans necessary to be contacted presuming an email population of 30% and return rate of 20% (these are the observed rates in other VEO online Veteran surveys).

The Filing population has been estimated at 6,000 per month, and an email population of 1,800 per month. Since the standard level of precision (95% CL, 3% MOE) is not attainable with the presumed 20% response rate, the precision target is set at 300, corresponding to a reliability of (90% CL, 5% MOE). The Hearing population is more sparse (1,700 total, and 510 with email information), and the highest possible level of reliability would correspond to getting responses from 100 Veterans (90% CL with 8% MOE). Thus, all available Veterans in the hearing population will be surveyed. In the Decision population, the target sample size is determined to be 400, corresponding to a reliability somewhat greater than (90% CL and 5% MOE) at the monthly level. This requires VEO to contact approximately 2,000 Veterans per month due to non-response.

Table 2A. Target Population Figures

Population Type	Approximate Monthly Population	Approximate Monthly Email Population	Precision Sample: 95% CL, 3% MOE	Precision Sample: 90% CL, 5% MOE	Precision Sample: 90% CL, 8% MOE
Filing	6,000	1,800	907	258
Hearing	1,700	510	656	233	100
Decision	7,400	2,220	933	260

Table 2B. Proposed Sample Targets by Time Period

Population Type	Weekly Target	Weekly Contact	Monthly Target	Monthly Contact
Filing	75	375	300	1,500
Hearing	25	125	100	500
Decision	100	500	400	2,000
Total	200	1,000	800	4,000

Weighting Adjustments

Many survey practitioners recommend the use of sample weighting to improve inference on the population. Under this process, the respondent sample is artificially made to more closely resemble the true population. It is reported earlier that email population comprises 30% of the overall Veteran population and 20% Veterans usually respond to the survey, weighting will be used to adjust for non-coverage (for the non-email population) and non-response (occurring when certain subpopulations are less prone to participate). The variables used in the weighting process will correspond to the respondent’s Age Group and Gender.

Quarantine Rules

VEO seeks to limit contact with Veterans as much as possible, and only as necessary to achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts upon survey participants. VEO also monitors Veteran participation within other surveys, to ensure Veterans do not experience survey fatigue. All VEO surveys offer options for respondents to opt out, and ensure they are no longer contacted for a specific survey.

Table 3. Proposed Quarantine Protocol

Quarantine Rule	Description	Elapsed Time
Other Surveys	Veterans engaged that have recently completed other VEO surveys (excluding Outpatient Services Survey) will not be selected for 30 days.	1 Month or 30 Days
Anonymous	Callers explicitly wishing to remain anonymous will not be contacted.	N/A
Opt Outs	Persons indicating their wish to opt out of either phone or online survey will no longer be contacted.	N/A

3