VOV Tracking_OMB Supporting Statement A 022915

Supporting Statement A

For

Department of Veterans Affairs, Veterans Benefits Administration (VBA)

Continuous Measurement Satisfaction Research Survey

OMB Control Number: 2900-0782

A. JUSTIFICATION

1. Explain the circumstances that make the collection of information necessary. Identify legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

In 2008, VBA recognized a need to develop and design an integrated, comprehensive Voice of the Veteran (VOV) Continuous Measurement Study for its lines of business: Compensation (CS), Pension (PS), Education (EDU), Loan Guaranty (LGY) and Vocational Rehabilitation and Employment (VR&E). This continuous measurement program helps VBA understand what is important to Veterans relative to benefits and services provided by VBA and provides VA/VBA leadership with actionable and timely Veteran feedback on how VBA is performing against those metrics. Insights help identify opportunities for improvement and measure the impact of improvement initiatives.  

VBA conducted a benchmark study in Fiscal Year 2013 (October 2012 through January 2013) in order to validate the survey instruments, identify Key Performance Indicators, and establish performance benchmarks. Findings and recommendations were presented to VBA Leadership and stakeholders within each line of business in April 2013.

Based on interviews conducted, J.D. Power (JDP) has separated the Veterans experience with VBA into two categories –Access to a Benefit (Transaction-based) and Servicing of a Benefit (Relationship-based). Stakeholders expressed the need to understand the different elements that make up the overall experience with VBA for the Veteran and wanted to ask questions relevant to each specific population. The following outlines how that is approached with each of the lines of business.

Compensation and Pension Programs

During 2014 J. D. Power fielded three survey instruments for the Compensation and Pension programs. Discussions with stakeholders from both programs indicated that one survey instrument could be used for both compensation and pension Access category claimants. In FY2015, Compensation and Pension identified the need to segment the Access survey surveys to better serve the business needs of each program.

The Compensation Access survey pool for the VOV Continuous Measurement Study includes individuals who have received a decision on a compensation benefit claim within 30 days prior to the fielding period. This includes those who were found eligible on a new or subsequent claim and those who have been denied and lack a current appeal of the decision. The Pension Access survey pool includes individuals who have received a decision on a pension benefit claim within the past 30 days. The Compensation Servicing survey pool includes individuals who received a decision and are receiving benefit payments. The Pension Servicing survey pool includes individuals who established and completed a claim in the previous fiscal year.

Education Program

J.D. Power fielded two survey instruments for Education Service. The Education Access survey pool includes individuals who received a decision on their education benefit application within 90 days (i.e., the original end-product was cleared within the past 90 days) prior to the fielding period. The Education Servicing survey pool includes beneficiaries who are currently receiving benefits. The definition of those receiving benefits varies based on the educational program. Chapter 33 beneficiaries who have received at least 2 payments for “tuition” in the past 9 months are included in the survey pool. Chapter 30, Chapter 1606, and Chapter 1607 beneficiaries who have received 5 monthly payments during the past 9 months are included.

Loan Guaranty and Specially Adapted Housing Programs

J.D. Power fielded two survey instruments for Loan Guaranty Service. The LGY Access survey pool includes individuals who closed on a VA home loan in the 30 days prior to the fielding period. The sample is stratified as follows: (1) those that closed on purchase loans, (2) those that received loans for interest rate reductions, and (3) those that obtained cash out or other refinancing. The SAH Servicing survey pool includes individuals who are eligible for a Specially Adapted Housing grant and in the past 12 months have: (1) received an approval on their grant and are currently in post-approval process, (2) had all their funds dispersed and the final accounting is not completed, and (3) had all of their funds dispersed and the final accounting completed.

Vocational Rehabilitation and Employment Program

J.D. Power fielded three survey instruments for Vocational Rehabilitation & Employment Service (VR&E). The VR&E Access survey pool includes individuals who applied within the last 12 months, entered Evaluation and Planning and (1) entered any of the following case statuses: Extended Evaluation, Independent Living (IL), Rehabilitation to Employment (RTE), or Job Ready Status (JRS) (excludes re-applicants), or (2) were found not entitled. The VR&E Servicing survey pool includes individuals who in the last 30 days were in a plan of services for more than 60 days, all rehabilitated participants, and MRGs. Participants who interrupted their plan are excluded. The VR&E Non-Participant survey explores why eligible individuals chose not to pursue the benefit entitlement. The VR&E Non-Participant questionnaire survey pool includes individuals who dropped out of the program prior to completing a rehabilitation plan. The sample is stratified as follows: (1) applicants who never attended the initial meeting with a counselor, (2) applicants who were entitled to the program but did not pursue a plan of service, and (3) applicants who started, but did not complete a rehabilitation plan (i.e., negative closures).

Findings

Findings and recommendations from the survey fielding periods were presented to VBA leadership and stakeholders within each line of business in FY 2014 and FY 2015. These discussions identified a need to refine the survey instruments due to technological and process changes within the VBA programs. J.D. Power collaborated with the lines of business in order to edit the survey instruments. In addition VBA requested the addition of standardized Agency Performance Goal (APG) questions to the survey instruments.

The results of the VOV Continuous Measurement Study allow VBA to measure the effectiveness of new initiatives and changes in processes identified within the VOV Study. Further, since the benchmark study, the volumes for each survey-type have been increased in order to provide regional offices with a statistically valid sample sizes. These changes were reviewed by statisticians and VBA leadership.

These surveys are conducted per the legislative requirement set forth in the Executive Orders 12862 - Setting Customer Service Standards and 13571- Streamlining Service Delivery and Improving Customer Service. The Executive Orders mandate that government agencies set standards and seek to measure agency performance against such standards while streamlining and improving delivery of benefits and services. These surveys are also being conducted in concurrence with the Government Performance Results Act (GPRA) and its requirements for improving customer service.

Specifically, VBA requires customer satisfaction information from Veterans in order to track the effectiveness and usability of the benefits offered, as well as to assess the satisfaction level with VBA provided services. The data collected is used by VBA to make improvements in both the application process and access to benefits, which in turn, enables VBA to maximize the efficiency and quality of service provided to Veterans and their dependents.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from current collection.

The information to be collected with the planned surveys is used by VBA leadership and the lines of business: Compensation, Pension, Education (EDU) Service, Loan Guaranty (LGY) Service, and Vocational Rehabilitation and Employment (VR&E) Service to determine Veteran satisfaction with benefits and services. The major use of this data is three-fold: 1) to provide Veterans an opportunity to comment on their experience with benefits and services, 2) to formulate program and policy changes in the access and servicing process to ensure the most effective delivery of benefits and services, and 3) to identify the most influential elements of Veterans’ experience on their satisfaction and establish performance benchmarks to gauge the effectiveness of program and policy changes in improving Veterans’ experience with benefits and services. Currently, there are no other means of continuously evaluating Veteran satisfaction with VBA’s benefit programs.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

In conjunction with a contractor, VBA staff develops, administers, and analyzes the Compensation, Pension, Education, and VR&E Access surveys, Compensation, Pension, Education, and VR&E Servicing surveys, the LGY and SAH surveys, and the VR&E Non-Participant surveys.

The surveys are self-administered in both paper form and online using web-based data collection with the exception of Pension Access and Servicing, SAH Servicing, VR&E Servicing, and VR&E Non-Participant which are available in paper form only. This maximizes the timeliness, efficiency, and response rate of data collection from Veterans. Based on previous years fielding experience, it is expected that the completion rates will be composed of 25% online collection and 75% of respondents will be received by the mail option.

JDPA has strong capabilities in programming and hosting surveys on the worldwide web, and maintains effective security and privacy procedures (e.g., unique passwords for respondents, data encryption) when designing and programming web surveys. The web address (URL) on which the surveys are hosted and accessed by respondents are included in the Veterans’ notification materials regarding the surveys. Veterans can either respond online or complete the physical survey package.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

As noted, VBA has not recently conducted customer satisfaction surveys specifically geared toward Veteran satisfaction with Compensation, Pension, EDU, LGY or VR&E benefits and services on a continuous basis since 2005.

Our review of available data finds that no sources outside of VBA’s VOV survey program are available that provide a reliable, representative, and statistically valid sample of the targeted audience who have used Compensation, Pension , EDU, LGY or VR&E services and from which detailed customer satisfaction indicators can be derived.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

No small businesses or other small entities are impacted by this information collection.

6. Describe the consequences to Federal program or policy activities if the collection is not conducted or is conducted less frequently as well as any technical or legal obstacles to reducing burden.

If the Voice of the Veteran Continuous measurement surveys are not conducted, VBA will not be able to fulfill its performance measurement requirements as articulated by the requirements of Public Law 103-62, the Government Performance and Results Act of 1993 and Title 38 USC, §527, Evaluation and Data Collection. Furthermore, VBA would miss an opportunity to learn from America’s Veterans about how to improve the Compensation, Pension, EDU, LGY or VR&E benefit access and servicing processes to best serve their needs. VBA would also lose an opportunity to track and document improvements or declines in Veteran satisfaction and service delivery over time.

The design and administration of the suite of VOV surveys incorporates significant measures to minimize burden on respondents. (These specific measures are discussed in more detail in section 12). There are currently no technical or legal obstacles to reducing burden using the planned methods.

7. Explain any special circumstances that would cause an information collection to be conducted more often than quarterly or require respondents to prepare written responses to a collection of information in fewer than 30 days after receipt of it; submit more than an original and two copies of any document; retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years; in connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study and require the use of a statistical data classification that has not been reviewed and approved by OMB.

There are no special circumstances that would require respondents to prepare or submit the documents outlined above, or respond in fewer than 30 days. The surveys are designed and carried out with appropriate scientific rigor, and are intended to produce valid and reliable results that can be generalized to the universe of study.

8. Part A: If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the sponsor’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the sponsor in responses to these comments. Specifically address comments received on cost and hour burden.

On June 20, 2013 a 60 Day Federal Register Notice was published at 78 FR 37278. There were no comments received.

On September 27, 2013 a 30 Day Federal Register Notice was published at 78 FR 59769. There were no comments received.

8. Part B: Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, clarity of instructions and record keeping, disclosure or reporting format, and on the data elements to be recorded, disclosed or reported. Explain any circumstances which preclude consultation every three years with representatives of those from whom information is to be obtained.

VBA is not aware of any tracking surveys in private or government sectors.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payment or gift shall be provided to respondents.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statue, regulation, or agency policy.

An assurance that the information will be kept private to the extent of the law is made in the introductory letter/postcards as well as the reminder letter respondents receive with the survey, regardless of the mode of administration. Respondents are assured that answers given will be kept private to the extent of the law and will not affect their current or future entitlement to benefits and services. The information that respondents supply is protected by law (the Privacy Act of 1974, 5 U.S.C. 522a and section 5701 of Title 38 of the United States Code).

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private; include specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

None of the survey instruments contain any questions of a sensitive nature.

12. Estimate of the hour burden of the collection of information.

In order to understand the different elements that make up the overall experience with VBA for a Veteran, questions were added to the surveys that result in a variation of length in the questionnaires. Questionnaires average 15 minutes to complete depending on the benefit line, completion method, and Veteran experience. The annual number of responses (Column 4 below) assumes a thirty percent (30%) completion rate by those who receive a survey instrument.

The volumes below reflect the estimated burden per year for the duration of FY 2016. These burdens are estimated based upon the FY15 actual response rates. In order to calculate the total volumes for the entire contract period, multiply the annual volumes by 1.5.

TABLE 1. FY 2016 TIME BURDEN, RESPONDANT COST, AND FREQUENCY BY RESPONDENT GROUP
Population surveyed	Number of respondents	Number of responses per respondent	Annual number of responses	Annual Frequency of Response	Annual burden per response (in minutes)	Total Annual Burden hours	Annual Cost Burden
Compensation Enrollment	160,000	1	48,000	Annually	15	12,000.00	$300,000.00
Compensation Servicing	60,000	1	18,000	Annually	15	4,500.00	$112,500.00
Pension Enrollment	10,000	1	3,000	Annually	15	750.00	$18,750.00
Pension Servicing	10,000	1	3,000	Annually	15	750.00	$18,750.00
Education Enrollment	16000	1	4,800	Annually	15	1,200.00	$30,000.00
Education Servicing	10000	1	3,000	Annually	15	750.00	$18,750.00
VR&E Enrollment	60,000	1	18,000	Annually	15	4,500.00	$112,500.00
VR&E Servicing	60,000	1	18,000	Annually	15	4,500.00	$112,500.00
VR&E Non-Participant	5,000	1	1,500	Annually	15	375.00	$9,375.00
Loan Guaranty Home Loan Process	40,000	1	12,000	Annually	15	3,000.00	$75,000.00
Specialty Adapted Housing Grant Process (Mail)	5000	1	1,500	Annually	15	375.00	$9,375.00
Totals	436,000	-	130,800	-	165	32,700	$817,500.00

According to the U.S. Bureau of Labor Statistics, Average Hourly Earnings, the cost to the respondent is $25, making the total cost to the respondents $817,500 (32,700 burden hours x $25 per hour).

13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

a. There is no capital, start-up, operation, or maintenance costs.

b. Cost estimates are not expected to vary widely. The only true cost is that for the time of the respondent (average of 15 minutes per respondent).

c. There are no anticipated capital start-up cost components or requests to provide information.

14. Provide estimates of annual cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies also may aggregate cost estimates from Items 12, 13, and 14 in a single table.

Table 2 below shows the labor and contracting costs for conducting the surveys for FY16. Operational costs are outsourced to a contractor and are included in the contractor’s total cost. In order to calculate the total volumes for the entire contract period, multiply the annual volumes by 1.5

TABLE 3: Estimated Cost to the Federal Government
Cost Item	Hours	Cost
VA-Labor	2,080	$145,028.00
Contractor Costs		$1,541,431.37
Printing and Postage		$637,458.66
TOTAL	2,080	$2,253,943.71

The VA Labor cost was estimated using a composite average salary and benefits figure of $47.87 per hour for 2,080 hours at a GS-13, step 4 and $58.28 per hour for 780 hours at a GS14, step 5.¹ The amount paid to the contractor for the survey effort includes as its major components the survey of recent claimants or existing beneficiaries for an annual cost of $1,471,457.08 in FY 2015. (In FY 2016, the annual cost is $1,541,431.37.) The cost of printing, mailing, and return postage for FY2015 was $625,219.52 in accordance to Government Printing Office (GPO) estimates. These costs include development of the instruments, development of the sampling plan, review of the instrument, locating of respondents, programming of the questionnaire for Web administration, questionnaire pretest, validation, data processing, providing a clean data file, project management and analysis, and reporting.

15. Explain the reason for any changes reported in Items 13 or 14 above.

There is a scheduled labor rate increase for the contractor that increases the annual contracting cost.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Based on previous survey experience, Section 16A below documents the proposed project plan for the tabulation and publication of survey data, including the analytical techniques and database management strategies to be employed. Following this, Section 16B provides an overview of the project timeline.

16A: TABULATION AND PUBLICATION

• Scan Surveys and Monitor Response Rates

During the survey field-period, the surveys are scanned as they are received to ensure the currency of the name and address data file. During the scanning process, VBA’s contractor inspects and removes duplicate surveys in case any individual inadvertently received a second survey after completing one from the first mailing. Data from damaged surveys could potentially be lost unless appropriate actions are rendered to make the data usable. Staples, paper clips, and notes are removed and stray pencil marks that would interfere with the scanning process are erased.

The contractor hosts a secure Enterprise Data Exchange (EDX) site that provides response rates from the survey. Throughout the period that the surveys are being fielded, response rate data is updated at a minimum on a monthly basis for the survey.

• Quarterly Reporting

Program lines receive an overview of findings that outline the following:

• Summary of key score matrices

• Key attributes

• Key diagnostics and demographics

• Verbatim (open-ended) response report

• Semi-annual program Line Briefings

• Summary of key findings at business-line level

The semi-annual program line briefing level report for each business line contains a summary of business line specific results, as well as a summary comparison of all business lines for measures that are common across business lines.

VBA’s contractor pre-tests automation routines for new designs in the analysis to ensure that the report exhibits graphics and data tables that can be reliably and accurately programmed for each report type. Development of the automated report system consists of several concurrent tasks, including a data crosswalk of the survey iterations to map identical variables and response sets, Statistical package for the Social Sciences (SPSS) syntax routines and development of prototype report templates using Microsoft Excel with Visual Basic for Applications.

• Conduct Final Briefing

The contractor develops data tabulations on survey highlights and presents the results to Compensation, Pension, EDU, LGY, VR&E and VBA staff. The contractor works closely with VBA to determine the most salient findings to present in the briefing. The contractor delivers bound color copies of the briefing presentation to all business lines.

• Load Data to Reporting Site

Data collected is loaded to the VOV Reporting Site for all program lines and regional offices. Reporting Site was developed under a different order.

• Deliver Final Products

The contractor submits the final deliverable containing electronic files as specified in the SOW. Documents and materials include:

• Raw survey data in SPSS format

• Copies of the reports in electronic format

• Copies of the questionnaire and survey mailing materials

• Survey respondent verbatim, grouped by Regional Office/Line of Business

These electronic files will provide VA with all materials used during the project, along with the final reports.

16B: PROJECT TIMELINE

The major activities for the VBA VOV survey project are structured by task, and are outlined below.

• Task 1: Conduct kickoff meeting and develop the Project Management Plan (PMP) and Risk Management Plan (RMP)

• Task 2: Monthly Status Meetings and written summaries

• Task 3: Methodology, Sampling Plans, and Production Schedule

• Task 4: Final Surveys, letters, postcards, and mailing materials to the print vendor

• Task 5: Sample cleaning, Imaging and Data Collection, and Disposition Reports

• Task 6: Host live survey websites and Disposition Reports

• Task 7: Host toll-free telephone and web helpdesk assistance and corresponding call/inquiry logs

• Task 8: Data Matrices, Verbatim Reports, Data Analysis

• Task 9: Data Load to VOV Reporting Site and Training Sessions

• Task 10: Reporting Site Training Sessions

• Task 11: Business line analysis and presentations

• Task 12: Semi-Annual Leadership Briefings

• Task 13: Action Plan development and sessions

• Task 14: Nonresponse Bias Reports per Office of Management and Budget (OMB) Standards and Guidelines for Statistical Surveys

Conduct kickoff meeting and develop the project management plan (PMP)	Continuous (annually)
Monthly Status Meetings and written summaries	Continuous
Methodology and Sampling Plans, and Production Schedule	Continuous (annually)
Sample cleaning, Imaging and Data Collection, and Disposition Reports	Continuous
Host live survey websites and Disposition Reports	Continuous
Host toll-free telephone and web helpdesk assistance and corresponding call/inquiry logs	Continuous
Data Matrices, Verbatim Reports, Data Analysis	Continuous
Data Load to VOV Reporting Site and Training Sessions	Continuous
Business line analysis and presentations	Continuous
Semi-Annual Leadership Briefings	Continuous (twice yearly)
Action Plan development and sessions	Continuous

17. If seeking approval to omit the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The collection survey instrument may be reproduced and/or stocked by respondents and Veterans organizations. Requiring VA to display an expiration date on the form would result in unnecessary waste of existing stocks. Inclusion of the expiration date could also result in a delay of the Department's action on the benefit being sought (respondent trying to obtain a newer version, while VA would have accepted the older version). For the reasons stated, VA continues to seek an exemption that waives the displaying of the expiration date.

18. Explain each exception to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB 83-I.

There are no exceptions.

1 2015 General Schedule Salary Table for the locality of Washington-Baltimore-Northern Virginia

12