NCA Memorial Affairs Sampling Plan - Caller Survey

The NCA Memorial Affairs - Caller Survey measures the experience of callers who called the National Cemetary Administration (NCA) Memorial Affairs – call center. The purpose of this report is to document the methodology of the NCA Memorial Affairs – Caller Survey and document data-based decisions. The survey is designed to measure the customer’s overall experience when contacting the NCA – Memorial Affairs call center.

The survey questions focus on the caller’s experience after speaking with a NCA – Memorial Affairs call center representative. The questions use a five-point Likert scale (Strongly Disagree to Strongly Agree) that asks the caller to rate his or her experience.

Callers are defined as anyone who has contacted the NCA – Memorial Affairs call center and spoke with a call center representative within the last 7 days.

The most recent call data for NCA – Memorial Affairs is sent from the NCA – Memorial Affairs telephony system, Cisco, daily, and will be compiled weekly by VEO. The compiled data will be reviewed and two files will be created. The first file contains all eligible callers and the second file contains all eligible callers, once business and quarantine rules have been applied.

The current sample design of the NCA – Memorial Affairs Caller Survey is to randomly select a call each sampled caller made to the NCA – Memorial Affairs call center in a week. The sampling unit is the Caller and the observational unit is the call made to NCA – Memorial Affairs that reached a call center representative.

Part I – Survey Background

To measure Callers experience, the Veterans Experience Office (VEO) uses a Human-Centered Design (HCD) approach to design a survey instrument aimed at measuring trust and experiences with VA. The HCD process includes speaking to Veterans and Contact Center employees to validate the usefulness and measurability of the survey questions. Regarding the NCA Memorial Affairs - Caller Survey, the scope is telephony data within the Cisco phone system for NCA Memorial Affairs.

Part II – Methodology

A. Survey Instrument

The NCA – Memorial Affairs Caller Survey consists of one brand and four experience questions. The NCA – Memorial Affairs Caller Survey questions were developed by the VEO team in 2017 following the Forrester Model of customer service and leveraging customer experience measurement best practices from the private sector.

The NCA – Memorial Affairs Caller Survey brand and experience questions consists of responding to the following statements on a Likert scale of 1-5 with 1 being Strongly Disagree, 2 being Disagree, 3 being Neither Agree nor Disagree, 4 being Agree and 5 being Strongly Agree:

Brand question:

I trust the VA call center will meet my needs.

Experience questions:

Effectiveness:

The call center representative was adequately prepared to help me.

Ease:

My issue was addressed in a reasonable amount of time.
The call center representative provided the information in a way that I understood.

Emotion:

The call center representative I spoke with was courteous and respectful.

Included is the Journey Map for the NCA – Memorial Affairs Caller Survey.

A.2. Pre-testing of the Instrument

As part of the HCD process, validation of proposed questions is a two-phase approach. The first phase focuses on testing question comprehension and usefulness to Veteran Callers, which took place in June 2017. The second phase focuses on evaluating the questions for actionable insights with Contact Center employees and Contact Center Leaders, which took place in July 2017.

Part III - Target Population and Frame Construction

The target population for the survey consists of people contacting and speaking with an NCA - Memorial Affairs call center representative in a given week. The primary sampling unit is the caller (identified by their phone number) and the secondary sampling unit is the specific call. Thus, the data can be analyzed to represent either callers or calls as the unit of analysis. It is also important to note that a caller can speak with a NCA - Memorial Affairs call center representative more than once in a week. In other words, a given data entry will contain a unique record of each call a caller had with NCA - Memorial Affairs during that week as well as a unique identifier for that caller. Therefore, the population file may contain multiple records for each caller.

Abandoned calls are not flagged in the NCA Memorial Affairs databases, but each call that is connected to NCA - Memorial Affairs is logged. This allows for the exclusion of calls by length of time, assuming that length of call time is indicative of abandonment. The initial NCA - Memorial Affairs recorded greeting is 37 seconds and then the caller is placed in the queue¹. Therefore, all calls under one minute in length are retained in the population file but excluded from sampling under the assumption that they were abandoned or did not reach a NCA - Memorial Affairs call center representative.

All calls for the reference period, which is Monday-Friday² will be queried from the NCA databases every Saturday night, and the population file is created based on the query. Once the data are pulled by the VEO team, the following quality control checks will be executed:

Identification of missing data in variables required for later merging and sampling
Identification of multiple call records per caller

Constructing and cleaning of the frame, i.e. the population file, will require several steps as shown in Figure 2.

Figure 2. Flowchart Frame Construction

To reduce respondent burden, the frame will undergo a screening for eligibility process on a weekly basis. Sampled callers also have the option to opt-out of completing the current and all future surveys by requesting to be put on a “Do Not Contact” list during the automated phone survey. This opt-out is documented by the VEO data team.

Thus, Callers will be quarantined from sampling based on the following rules:

Rule 1: Callers will be excluded from sampling for any transactions if they were sampled for another VEO survey or the Enterprise Trust Survey in the past 30 days.
Rule 2: Callers will be excluded in a weekly sample if they were also sampled in the previous week.
Rule 3: Callers will be excluded from sampling within a transaction if they are sampled for another transaction in the current round of sampling.
Rule 4: Callers will be excluded from sampling for a transaction if they requested to be put on a “Do Not Contact” list in the VEO Survey for that transaction.

The quarantine rules will be applied during the creation of the sampling population frame updates on a weekly basis. The distribution of the quarantine records will be compared to the population distribution to identify if there are potential concerns of biasing factors that might impact the analysis and results. Callers are reintroduced to the sampling pool after their corresponding quarantine period is over unless they had opted out from all future VEO surveys.

The variables and information needed to support the sampling process include:

A unique caller identifier, which is maintained over all calls a caller makes
A unique encounter identifier must be created
The length in minutes for each call
The date and time the call was made
The time of day the call was made
A flag that identifies records ineligible for sampling (previously sampled, opted out, etc.)

Part IV - Population Estimation

Introduction

The goal of the NCA Memorial Affairs Contact Center Survey is to create representative estimates on the experiences of callers with the NCA Memorial Affairs Contact Center over time. This is accomplished either through surveying all callers (the “census” approach) or selecting a representative sample and calculating population adjustements for the respondents (the “sample” approach). While the census approach reaches out to all callers, it is not appropriate for this survey due to the quarantine rules listed in the previous section. The quarantine rules used within the VEO surveys would greatly reduce the number of records available for future sampling in the census approach. Therefore, sampling is needed for the NCA Memorial Affairs Contact Center survey.

Because the sample is only a subset of the target population, population-representative weights must be applied to the responses to obtain representative estimates. The weights adjust the final estimates to account for the sample size and to allowaccurate inference to the population. Furthermore, using weighted estimates prevents the distortion on the final estimates due to population undercoverage and nonresponse. Therefore, the weighted estimates are statistically valid for the population.

Sample Design

The sample design is a stratified simple random sample (SRS). This design addresses several (sometimes competing) objectives within the sample. First, the design maintains the characteristics of the NCA Memorial Affairs Contact Center caller population to create population-representative estimates. Second, the design accounts for factors specific to this survey. Third, this design prevents unnecessary oversampling of the eligible population, thereby permitting sampling for future surveys.

Prior to sampling, all calls will be screened by length of call. Calls less than one minute in length are excluded from sampling under the assumption that they did not reach an NCA – Memorial Affairs call center representative (See Limitations for justification). All inbound numbers will be stratified by number of eligible calls made to the NCA - Memorial Affairs Call Center in the reference week (one call only versus more than one call). Inbound numbers will be randomly selected within each stratum to create a representative sample. Within the sampled numbers, the most recent call will be selected for the survey. The sample flow chart can be seen in Figure 3.

Figure 3. Sample Flowchart.

Sample Size Calculations

The sample size was calculated to achieve two objectives:

Obtain estimates that possess a reasonable degree of precision.
Attenuate the impact of the quarantine rules on future VEO survey samples.

These objectives act against each other. Larger sample sizes ensure estimates with a high degree of precision. However, the VEO survey quarantine rules reduce the number of records eligible to be sampled in the current week based on the size of the previous week’s sample sizes. Larger sample sizes in one week will lead to fewer eligible records in later weeks. Therefore, the sample size needs to maintain a balance between the two objectives.

Table 1 shows several target sample sizes and the justification for the selected sample size. The optimal sample size is one that obtains a precise sample that does not overtax the population for future sampling. The sample size will be drawn weekly; however, the estimates will include all respondents in the previous 30 days. Therefore, the target sample size needs to be determined at the monthly level to ensure that the final estimate meets the target level of precision. The weekly sample size was then divided by an estimated response rate of 20% to adjust for nonresponse. The adjusted sample size was then divided by 0.9 to account for nonworking numbers. This is an arbitrary working rate yet it seems reasonable since these are inbound calls to the NCA - Memorial Affairs Contact Center.

Based on the known data, the recommended monthly sample size should be 300 because this will provide a precise estimate without overtaxing the population for future sampling. Therefore, the fielded sample will be 389 each week, which is roughly one-quarter to one-third of the weekly population, which is row 3 in Table 1. Table 1 illustrates the various samples using varying 30-day respondent size and margin of error.

Table 1. 30-day and Weekly Sample Sizes.

30-day respondent size	Margin of error	Weekly respondent size	Weekly fielded sample size	30-day fielded sample size
100	0.10	24	134	577
200	0.07	47	262	1127
300	0.06	70	389	1673
400	0.05	94	523	2249
500	0.04	117	650	2795

Finally, the sample is selected using PROC SURVEYSELECT in SAS.

Probabilities of Selection

The probabilities of selection were calculated for each sampled record. These probabilities represent the chance that a call was selected given this sample design. Probabilities of selection are the starting points for creating population-representative estimates. For the NCA - Memorial Affairs Contact Center survey, the probability of selection is calculated as the product of the probability of selecting a caller times the probability of the selected call being the most recent call independent of all other calls, or:

where:

= the probability of selection for the ith sampled caller in the hth stratum.

= the probability of selecting the ith sampled caller in the hth stratum.

= the number of calls made by the ith sampled caller in the hth stratum.

Sampling Weights

The sampling weights are calculated as the inverse of the probabilities of selection of each record:

Where = the sampling weight for the ith record in the hth stratum.

Typically, the sampling weights should allow for the sample to be representative of the total number of calls made within the reference week. This is true for callers who made one call to the NCA - Memorial Affairs Contact Center. For callers who made more than one call, the NCA - Memorial Affairs Contact Center Survey sample design does not allow for population-representative sampling weights because the probabilities of selection are not equal across all records. In other words, differs for all calls. This is because each caller has made a varying number of calls to the NCA - Memorial Affairs Call Center in the reference week. Consequently, the probabilities of selection are conditional on the number of calls a caller has made. This leads the weighted counts of calls not being equal to the true number of calls in the population.

To prevent the weighted call count from being not representative, an adjustment must be applied to be sampling weight for callers who made more than one call. The sampling fraction for all records in a stratum can be rewritten as:

Where

= the sampled number of calls in the hth stratum

= the total number of calls in the hth stratum email population

= the sampled number of callers in the hth stratum

= the total number of callers in the hth stratum email population

= the average number of calls a caller made in the hth statum

= and = .

With this new equation, all calls in the hth stratum have the same probability of selection. In other words, = for all records in the hth stratum. The adjustment now must make each sampled record’s probability of selecting a call match . The design adjusted weight is calculated as:

Where

= the number of calls made by the ith sampled record in the hth stratum

Nonresponse Adjustment

Because not all sampled callers respond to the survey, the adjusted sampling weight must be further adjusted to account for nonresponse. The nonresponse-adjusted weight is calculated as

Where

= the number of survey respondents in the hth stratum

Call Coverage Adjustment

The weights undergo one final adjustment to make the weighted number of calls representative of the population totals for call length. The final weight accounting for these factors is calculated as:

Where:

= the final weight for the ith caller in the hth stratum

= the number of calls in the jth call length for the ith caller in the hth stratum

= the number of callers for the hth stratum

= the total number of calls on the jth call length the ith caller had in the hth stratum

Because rare events may not be sampled, all events with less than 10 occurrences within the population will be collapsed into one group. After collapsing, the event coverage adjustment is calculated to allow the respondent weight to be representative of the population number of the collapsed group.

Unequal Weighting Effect

The unequal weighing effects (UWE) are calculated to examine the impact of the weights on the precision of the final estimates. Weighting adjustments affect the variance of the final estimate because of the variation added by the presence of sampling error, nonresponse error and coverage error. The UWE is a diagnostic statistics that estimates the percent increase in the variance of the final estimate due to the presence of weights and is calculated for the h^th stratum as (Kish, 1992; Liu et al., 2002)

where

= the unequal weighting effect of the hth stratum

= the square of the coefficient of variation of the final weights in the hth stratum

The UWE estimates the increase in the variance of the final estimate due to weighting. A UWE of 1.3 means that weighting increases the variance in the weighted estimate by 30%. UWEs can be calculated for the overall estimates or the estimates on population subsets. The lower bound for UWEs is 1.0, meaning that the weights do not increase the variance. While there is no distinct upper limit for UWEs, a UWE of 2.0 is considered the upper guideline for weights overinflating the variance of the final estimate.

Weighted Estimates

The final weights are calculated for each administered sample file upon completion of data collection. The final estimate is calculated as

Where:

= the weighted score for the hth stratum

= the number of respondents in the hth stratum

= the experience reported by the ith respondent in the hth stratum

Part V - Survey Administration

Survey Mode

The NCA – Memorial Affairs Caller Survey is administered as an automated phone survey using the Cisco Platform. The lack of available demographic information, including email address, preclude the survey from being sent via Medallia or another email based surveying tool.

Survey Invitation File

The sampled records will be compiled by the VEO Data Team into a file called a survey invitation file that is uploaded to the Cisco Unified Contact Center Enterprise (UCCE) system in use by NCA call center each non-holiday Wednesday of the month. The invitation file will contain the sampled phone numbers.

Response File

Upon all survey calls being completed, the Cisco Unified Contact Center Enterprise (UCCE) system in use by NCA call center exports a file containing the call dispositions for each sampled number. This file is merged with the population file to obtain the rest of the data on the sampled calls. This will allow for postsurvey adjustments and calculation of the response rates.

Part VI – Post-Survey Adjustments and Reporting

Postsurvey adjustments are calculated for each sample that has completed the data collection period when the response file is received. The postsurvey adjustments are added to the Medallia response file for the weighted analysis.

Data Analysis and Reporting

The weighted responses to each question will be displayed within the Medallia dashboard tool.

Part VII - Assumptions and Limitations

The following section addresses assumptions and limitations regarding NCA – Memorial Affairs data, callers, and other surveying considerations.

Assumptions

A.1. Abandoned calls

The assumption is that abandoned calls are calls that last less than one minute, because abandoned calls are not documented within the NCA telephony data. This assumption seems reasonable in that callers would have to go through an automated menu system, which includes an initial message of 37 seconds, prior to reaching an NCA – Memorial Affairs call center representative. While this assumption may not hold in cases where the caller spoke with the NCA – Memorial Affairs call center representative for a very short time, we cannot recognize these cases and we believe these cases to be very few in number.

A.2. NCA database variables

This assumption states that the variables provided to us by NCA adequately provide all the information needed to derive the sample. In other words, there are no other variables that can be included to improve the sample design. As NCA expands their databases, we will analyze including more of their data into the sampling plan.

A.3. Response rates and nonworking phone rates

The response rate and working phone number rate adjustments were assumed to be 0.2 and 0.9, respectively. These are based on previous work in telephone surveys (CHIS reference) and the fact that these numbers are inbound numbers. These adjustments may be altered based on future survey paradata.

A.4. Sampling design

The sampling strata are independent of the survey responses. In other words, the response from a person is not correlated to the call length, time of day or day called based on the data provided. This assumption will be revisited as survey responses come in.

Limitations

B.1. NCA - Burials Services

This sample is limited to only to call made to the NCA – Memorial Affairs call center. Calls for NCA - Burial Services are excluded from the population because they exist on a separate phone system. Therefore, the data and records are not available for sampling, and represent a sizable bias because there are more calls made to Burial Services than Memorial Affairs annually. With the availability of Burial Services call data, those calls can be included in the sample design. Additional work with NCA is needed to include Burial Services in the sample design.

B.2. Nonresponse

The high level of nonresponse from the use of a phone survey introduces two issues that could affect analysis. First, some strata may not contain respondents. The extent of this cannot be measured until the survey has been conducted for a quarter. Therefore, no analysis of these strata can be conducted as they are currently defined. Collapsing strata prior to sampling will mitigate this. However, only after the survey is conducted will any strata with no respondents be known.

Second, the desire to respond may be correlated with the demographics of the population. For example, in the Enterprise Trends Survey (previously called APG survey), older veterans tend to have a higher response rate as compared to younger veterans. Therefore, the final estimate would be biased towards the responses from the older population, if not for the post-survey adjustment that is calculated after the Enterprise Trends Survey is completed.

The recommended fixes for these issues can only be applied after the survey is conducted. The fix for the first issue is to collapse strata with no respondents into strata that contain respondents. The fix for the second issue is to calculate a nonresponse adjustment to reduce the nonresponse bias within the estimates. For the NCA - Memorial Affairs Call Center survey, the demographics of the callers are not available; thereby analyzing the potential nonresponse bias by demographics cannot be currently be done. This does not mean that such biases do not exist. VEO will incorporate demographic information about the callers as they become available, which will allow us to examine this potential source of nonresponse bias.

B.3. Coverage Bias

This survey only covers the most recent call callers made to the NCA-Memorial Affairs Contact Center that were one minute or more in length and reached a VA representative. Therefore, there are two sources of potential coverage bias. The first source are calls less than a minute that were answered by a VA representative. As stated above, we assume these to be few in number. However, these callers my report different experiences than the eligible callers if they were they included. This source of covereage bias cannot be currently analyzed. This is only possible if the NCA – Memorial Affairs Contact Center creates a method for identifying and flagging these records in their weekly reports.

The second source of coverage bias is that no calls to all other NCA Contact Centers or calls to other VA Contact Centers that handle NCA affairs were included in the sampling frame. This survey only focuses on the NCA – Memorial Affairs Contact Center. No inferences can be drawn from this survey to any other NCA contact centers or VA contact centers that handle NCA affairs.

B.4. Cisco Unified Contact Center Enterprise (UCCE) system

The Cisco Unified Contact Center Enterprise (UCCE) system is a telephony system deployed within the NCA – Memorial Affairs contact center. The automated phone survey is also being conducted via the Cisco Unified Contact Center Enterprise (UCCE) system and should be viewed as a proof of concept, due to the manual nature of sampling and launching the survey. The long-term feasibility of the manual process is unclear.

Part VIII – Quality Control Process

There are several quality control measures that are conducted throughout the survey process.

Population file

The file provided by the NCA – Memorial Affairs Contact Center contains the necessary variables.
The number of missing values in the necessary variables is very small.
Records are accurately flagged based on quarantine rules.
All calculated variables are present and accurate.

Sample file

The required sample size is met for each stratum.
The most recent call is selected for each sampled number.
The invitation file contains the sampled phone numbers.

Data collection

The calling rules are being followed.

Postsurvey Adjustments

The calculated post survey adjustments match the population totals in each step.
The UWEs for the final adjustments are less than 2.

Part IX - Appendices

Appendix 1: NCA Data Fields

Using Medallia, the following data fields are necessary for launching an email survey. Given the limited NCA – Memorial Affairs data, the only fields necessary to launch the automated phone survey are the Phone Number of the Caller and the Data of the call.

Data Field	R/P/N*	Availability
First Name of Caller	Required
Last Name of Caller	Preferred
Phone Number Caller	Preferred	
Email Address of Caller	Required
Gender of Caller	Nice to Have
DOB/Age of Caller	Nice to Have
Unique Identifier of Veteran (SSN, ICN, etc.)	Required
Reason for the call (topic/issue)	Preferred
Date of call	Required	
Contact Center Employee First Name	Required
Contact Center Employee Last Name	Required
Contact Center Employee Staff Type or Role	Required

*R = Required, P = Preferred, N = Nice to Have

Additional telephony data is available and will be utilized on the Dashboards.

Appendix 2: Acronyms

Acronym	Definition	Description
Cisco		Telephony switch used by NCA – Memorial Affairs
NCA	National Cemetary Administration	Administers death and burial benefits for eligible Veterans and their families.

Appendix 3: Contacts

Stakeholders

Name	Organization	Email Address	Role
Anil Tilbe	EM&PI	[email protected]	Director, EM&PI
Denise Kitts	VEO	[email protected]	Director of Operations
Eric Powell	NCA	[email protected]	Deputy Director, Memorial Programs Service, NCA

Data and Technical Points of Contact

Name	Organization	Email Address	Role
Pete Smith	VBA	[email protected]	VBA & NCA data SME

VEO Team

Name	Organization	Email Address	Role
Marcelle Saab	EM&PI (CTR)	[email protected]	Deputy Project Manager
Megan Iverson	EM&PI (CTR)	[email protected]	HCD Lead
David Lyle	EM&PI	[email protected]	VEO Data Team
Vesta Gueschkova	EM&PI	[email protected]	VEO Data Team
Jane Newman	VBA	[email protected]	Insights & Design, Content Strategist
Michael Jacobsen	EM&PI (CTR)	[email protected]	Statistician

1 Statistic provided by Eric Powell, Deputy Director, Memorial Programs Service, NCA

2 The NCA – Memorial Affairs contact center is operational Mon-Fri, 8:30am-5pm EDT, excluding federal holidays and observed federal holidays.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Amer, Safaa
File Modified	0000-00-00
File Created	2021-01-21