Supporting Statement B for Agencies_A 11 Section 280 Improving Customer Experience

SUPPORTING STATEMENT B

FOR PAPERWORK REDUCTION ACT SUBMISSION

Generic Clearance for Improving Customer Experience

(OMB Circular A-11, Section 280 Implementation)

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The activities under this clearance will cover a wide array of customers who interact with the Veterans Administration. This may include but not limited to Veterans, other beneficiaries, family members, caretakers, Veteran Advocates, organizations checking beneficiary status, VA employees, visitors to Veteran Cemeteries, and anyone who contacts the VA either directly or through VA websites or message lines. The sizes of the populations or number of covered entities and sample frames for these studies will vary depending on the study and will be communicated in each submission under this proposed generic clearance. Response rates will vary depending on the mode of interview and the population covered by the individual study. The expected response rate will also be provided as part of the detailed methodology included in each submission made under this clearance.

The activities under this clearance may involve probability-based and nonprobability-based samples, with respondents selected either to cover a broad range of customers or to include specific characteristics related to certain products or services. Unless otherwise stated, the samples associated with customer surveys are scientifically drawn samples intended to generate statistically rigorous estimates that are generalizable to a known and defined population. The samples associated with focus groups and customer testing in this collection are not subjected to the same scrutiny as scientifically drawn samples as they are not meant to be interpreted similarly.

Results from focus groups, customer journey maps and customer testing will not be used to make statements representative of the universe of study, to produce statistical descriptions (careful, repeatable measurements), or to generalize the data beyond the scope of the sample. The specific sample planned for each individual collection and the method for soliciting participation will be described fully in each information collection request.

Qualitative research interviews and surveys are tools used by program managers to change or improve programs, products, or services. The accuracy, reliability, and applicability of the results of these surveys are adequate for their purpose.

2. Describe the procedures for the collection of information, including:

   • Statistical methodology for stratification and sample selection.

   • Estimation procedure.

   • Degree of accuracy needed for the purpose described in the justification.

   • Unusual problems requiring specialized sampling procedures.

   • Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Data collection methods and procedures will vary and the specifics of these will be provided with each collection request. The Agency expects to use a variety of methodologies for these collections. For example, the Agency or its contractors may use commercial survey-specific software to automate its collection and analysis of feedback. In addition to physical copies, information collection instruments may be electronically disseminated and/or posted on target pages of the Agency’s web site. Telephone scripts, personal interviews, and focus groups with professional guidance and moderation may also be used. The specific method of data collection will be described in the methodology document attached with each submission under this generic clearance.

Quantitative surveys, will be drawn to distribute the burden evenly among the population for the study and reduce systematic bias in the implementation of quarantine rules mitigating survey burden. Proportionality and/or, when designed, disproportionality will be guaranteed through a stratified design to meet design and analytic needs for the study. Additionally, design weights (when applicable) and non-response weights will be employed for estimation. The degree of accuracy will depend on a number of factors including, the level in which tracking and/or comparisons are needed, the availability of sample at the specified level, and the relative importance of the study. All of these details will be provided in quantitative submissions under this proposed generic clearance.

Qualitative studies will be sampled using non-probabilistic methods that will provide the needed set of potential respondents to meet the study targets. The target populations for these studies will be defined by the availability of targeted respondents and the locations chosen by subject matter experts. Because qualitative studies are not representative of a population, no weights will be calculated for these projects.

3. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

VEO will utilize multiple data collection methods to obtain data for measuring customer experience. Nonresponse will impact the data collected from each method differently and will be handled in manners appropriate to each method.

VEO surveys are designed to limit respondent burden, minimize item and unit non-response, and maximize response validity. The questionnaires are designed with customer input and validated through cognitive testing. There are several aspects of VEO surveys that are designed, specifically, to help address survey non-response:

• The surveys are short with a set of seven core questions and no more than eight additional questions specific to the customer experience measured for an expected median completion time of two to six minutes;

• Survey invitations and reminders are designed to emphasize the benefits of responding and the low burden required to provide that response;

• Survey respondents receive a single reminder sent one week after the initial invitation to minimize the burden and perceived hassle of survey response;

• Recency of the experience is improved by the rapid fielding frequency. This frequency improves the respondent’s ability to recall the survey trigger and provide salient observations; and,

• VEO enterprise management of respondent burden across all VA surveys.

  • Strict exclusion criteria eliminate multiple, successive administrations of surveys to respondents.

  • Where practical, proportional sampling is used to mitigate the risk of survey fatigue within the sub-population.

For surveys administered from probability-based samples, VEO is aware of the potential for non-response and coverage bias and will continually seek ways to improve coverage and response rates. As this survey provides continuous tracking with real-time reporting, survey results will need to be monitored closely for non-response bias. Furthermore, any proposed improvements in the methodology will be thoroughly tested to understand the extent, if any, the changes would disrupt trend data.

The primary tool used to counter potential non-response bias in surveys from probability-based samples is through weighting. Weights are applied in real time through monthly population cell targets. Depending on data availability, we may use demographic data such as age, gender, and Veteran status for weighting the population strata. Additional variables, if available, may be analyzed if it is determined to improve unit non-response.

As part of our monitoring and approach of continual improvement, the VEO team will actively monitor and model non-response and non-coverage patterns in surveys based on probability-based samples to assure consistency and to inform improvements in sampling, weighting, and/or data collection methodology.

For surveys administered from nonprobability-based samples and focus group, methods to control nonresponse bias will be conducted exclusively at the time of data collection. Such methods will include, but are not limited to:

• Invitations tailored to the group being administered the data collection method.

• Questions that are designed to only obtain the information relevant to the group under study.

• For surveys, short survey instruments that follow the guidelines listed in Justification A.

• For focus groups, interviewer methods that encourage maximum participation among interviewees in the focus group.

Because the purpose of these data collections provide insight to change or improve programs, products, or services, the results obtained from the respondents are adequate to meet the requirements.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The VEO team will continually test the efficacy of scores obtained through surveys from probability-based samples. Some of this will be formalized such as the checking of sample and the effectiveness of weights. Other tests such as non-response analysis/modeling will be periodically conducted in accordance with OMB policy to improve sampling and weights. Changes in weights may be tested retroactively to determine the effect on key measures.

The survey’s results dashboards will allow two lines of analysis. Firstly, the VSignals dashboard responses provide line item responses, proportions and mean scores, trends, and filters by sample or time periods. Secondly, the VSignals platform can support open-ended responses that may be evaluated individually or coded through natural language processing and tabulated.

For surveys from nonprobability-based samples and focus groups, pretesting of the data collection instruments may be done with internal staff, and no more than nine external colleagues, and/or customers who are familiar with the programs and products. Information collected using these methods will not yield generalizable quantitative findings; however, they can provide useful customer input. Results from these methods will not be used to make statements representative of the universe of study, to produce statistical descriptions (careful, repeatable measurements), or to generalize the data beyond the scope of the sample.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractors, grantees, or other person(s) who will actually collect or analyze the information for the agency.

Contact information will be provided in the methodology documents for each data collection request submitted under this generic clearance.

3