VBA Contact Center Sampling Plan

VBA Call center transactional Survey

methodology brief: sample design

The Veteran Benefits Administration (VBA) oversees government programs supporting Veterans, including those furthering their education or filing for pension benefits. These programs engage Veterans through the National Call Center (NCC) or other benefit-specific call centers. The Veterans Experience Office (VEO) was contacted by VBA to measure the customer satisfaction of callers contacting the following call centers: NCC, Pension, and Education. A random sample of recent callers to these hotlines will be contacted via email and invited to complete a brief transactional online survey. Selected respondents will have 14 days to complete the survey, with an email reminder after 7 days if the survey has not been completed. The goal of this document is to provide a high-level description of the sampling design, including the number of callers planned to be contacted for the survey.

Target Population: Veterans, their dependents, and their representatives (such as caregivers, those with power of attorney, funeral home directors) who contacted the National Call Center, Pension Call Center, or Education Call Center, within the past week. Note only callers sharing their demographic information and email address will be included in the survey frame.

Sample Size Determination: Over six million beneficiaries initiated calls to these centers in FY 2018. The chart below (Table 1) depicts the population figures (i.e., total incoming calls) based on call records from that period. Beginning in September 2018, VEO will conduct a semi-weekly random sample of eligible callers. To reduce the burden on the beneficiary population, only sufficient numbers of persons will be contacted to obtain reliable weekly estimates. The reliability standard will be consistent with the customary threshold utilized in the survey industry: 95% Confidence Level and 3% Margin of Error. In the case of the NCC, the overall target respondent size is increased beyond this threshold to accommodate reliability targets (90% CL & 5% MOE) within the eight individual call centers which comprise the NCC. The resulting respondent targets for each of the NCC facilities is 300, leading to an overall NCC respondent target of 2,400 (See Table 2).

Based on prior VEO online surveys with email recruitment, the expected response rate is 20%. Thus, the number of contacts to NCC callers necessary to yield 2,400 respondents is approximately 12,000. Younger Veterans have lower response rates while older Veterans generally have higher response rates. The populations of the Education and Pension call centers may skew younger and older, respectively; the response rates have been modified accordingly. This is reflected in the proposed weekly contacts in Table 1 below.

Table 1. Population Counts and Proposed Sample Sizes:

Call Center	FY 2018 Weekly Incoming Calls†	Proposed Weekly Respondent Size	Estimated Response Rate	Proposed Weekly Email Contacts	Weekly Sampling Rate
Education Call Center	47,954	1,100	10%	11,000	23%
Pension Call Center	7,355	919	25%	3,678	50%
National Call Center	122,745	2,400	20%	12,000	10%

^†Source: Office of Field Operations, VBAPHIL

Note: Sample sizes could be reduced for certain calendar weeks where the caller population is lower, especially for the Pension Call Center. These figures assume there are sufficient numbers of callers providing their email address to VBA. Thus, the numbers above represent upper limits to the amount that would be contacted; they would be reduced in the event that a large proportion of callers fail to provide email contact information within a call center.

Table 2. NCC Subsidiary Call Center Targets:

NCC Call Center	Average Weekly Calls†	Weekly Respondent Target	# Weekly Contacts
Cleveland, OH	15,290	300	1,500
Columbia, SC	12,770	300	1,500
Philadelphia, PA	7,660	300	1,500
St. Louis, MO	14,195	300	1,500
Nashville, TN	24,060	300	1,500
Salt Lake City, UT	16,825	300	1,500
Phoenix, AZ	28,435	300	1,500
San Juan, PR	3,510	300	1,500

^†Source: Office of Field Operations, VBAPHIL

Sampling Design: Data collection will occur twice a week to improve cognitive recall of customer experience by initiating contact within 2-3 days of their call. Caller information will be regularly extracted from the VBA database resources—the Enterprise Data Warehouse. The extraction process will be executed and validated by the Office of Performance Improvement and Assessment (PA&I), and the population extracts sent to VEO on a semi-weekly basis. These extracts will also be used to obtain universe figures for the sample weighting process (as data is available).

Beneficiaries will be randomly selected from the half-week population according to a stratified design. The general object is to obtain a representative sample with respect to the stratification variables. However, the NCC sample will have fixed targets for each subsidiary call centers to ensure accurate estimates for each center. Otherwise, callers will be selected with a balanced representation with respect to caller demographics, Veterans Status, and Call Characteristics (i.e., number of calls, call length). The allocation targets from each stratum will fluctuate with monthly changes in the population (with the exception of NCC subsidiaries). Finally, the demographic strata in level 3 below may require implicit selection via sequential sampling and would not have explicit targets for each grouping.

Sampling Stratification (Tentative)

• Level 1: Call Center Type, Subsidiary Call Centers, Caller and Veteran Status

• Level 2: Phone Characteristics (e.g., # Number of Calls, Total Call Length)

• Level 3: Demographics: Geographic Region

Many survey practitioners recommend the use of weighted survey estimates to improve inference on the population. The method attempts to correct for under-coverage and non-response bias. The weighting methodology will be applied separately to each encounter type – NCC, Pension, and Education will be considered as independent weighting models. VEO will also seek to incorporate Caller Characteristics and available demographic information into the algorithm.

Summary:

Sampling Design Feature	Description
Survey Reporting	Weekly
Data Collection	Semi-Weekly
Stratification (Explicit Sample Targets)	Call Center Type, Subsidiary Call Center, Call Characteristics (Multiple, Length), Veteran Status
Stratification (Implicit, No Sample Targets)	Geography
Quarantine	1 Month for VBA CC Survey, 2 Months for Other VEO surveys
Weighting Classes	Call Center Type
Weighting Variables (Tentative, based on availability of key demographic information)	Subsidiary Call Center (NCC only), Call Characteristics, Veteran Status, Geographic Region

1. Lohr, Sharon Sampling Design and Analysis 1999