Va 3

PILOT OF HCAHPS/SHEP SURVEY INSTRUMENTS, Continued

A. Justification

1. Explain the circumstances that make the collection of information necessary. Identify legal or administrative requirements that necessitate the collection of information.

The mission of VHA is to provide high quality medical care to eligible veterans. Executive Order 12862, Setting Customer Service Standards, dated September 11, 1993, calls for the establishment and implementation of customer service standards, and for agencies to “survey customers to determine the kind and quality of services they want and their level of satisfaction with current services”. Federal law, 38 U.S.C. Section 527, requires the Secretary of Veterans Affairs to evaluate programs and provision of services to beneficiaries. In response to these directives, VHA conducts both centrally and locally administered surveys to determine the level of satisfaction with existing services among VHA's customers. The surveys solicit voluntary opinions and are not intended to collect information required to obtain or maintain eligibility for a VA program or benefit.

Executive Order 12862 also calls on Agencies to “benchmark customer service performance against the best in business”. The Joint Commission on the Accreditation of Hospital Organizations (JCAHO) is poised to adopt the Hospital Consumer Assessment of Health Plan Survey (HCAHPS) as a national standard survey for inpatients and VA needs to conform with private sector surveys to the extent possible so that VA can have a means of comparing customer service performance with comparable non-VA healthcare facilities nationwide. On December 12, 2005, OMB approved the HCAHPS survey instrument for use by the Agency for Healthcare Research and Quality (AHRQ) under approval number 0938-0981. However, before Veterans Health Administration (VHA) implementation, it is necessary to test the possible consequences of using the HCAHPS alone versus in combination similar questions on the existing Veterans Health Administration (VHA) Survey of Healthcare Experiences of Patients (SHEP) Inpatient survey (approved under 2900-0227). The SHEP contains a broader range of questions mandated by Congress. A series of five pilots is proposed to develop the future patient satisfaction survey for hospitalized veterans.

After the results of these pilots have been analyzed and a decision made, VHA will furnish OMB an updated inpatient survey (2900-0227) via OMB 83-I.

2. Indicate how, by whom, and for what purposes the information is to be used; indicate actual use the agency has made of the information received from current collection.

Pilot results will determine content and survey methods for future rounds of customer satisfaction surveys. VHA proposes a three-part piloting of the HCAPHS survey instrument to better understand how the revised questionnaire containing HCAHPS (either long form or short form) will perform in measuring patient satisfaction, and what will be the effect on scores and response rates. In addition, we intend to determine how the HCAHPS sampling methods work in the population of Veteran inpatients, and how those results may differ from current (SHEP) sampling methods. This pilot study is designed to obtain baseline results on how veteran patients respond to healthcare related questions using the HCAHPS survey, and modified versions as compared to responses to similar questions on the existing Survey of Healthcare Experiences of Patients (SHEP) Inpatient survey (approved under 2900-0227), to identify and test methods for maintaining the ability to validly trend performance results should VHA pursue adopting HCAHPS as the standard survey for inpatients. The purpose of the pilot is to prepare VA to conform to future CMS standards for Medicare claims. The purpose of these patient satisfaction surveys is to determine how to improve services, customer satisfaction with existing services and how or if customer satisfaction has changed in response to reengineering efforts. The results will be used by management at all levels as a tool for assessing and improving the quality of services being provided to patients. Patient satisfaction scores are part of the evaluation system for senior VHA managers.

The purposes of each of the arms of the study are detailed in the table below:

Study Arm:	HCAHPS Short Form survey (existing SHEP sampling)	HCAHPS Short Form survey (HCAHPS sampling)	HCAHPS Long Form Survey (HCAHPS sampling)
Sample size:	2,700 outgo; 1,510 responses	2,700 outgo; 1,510 responses	2,700 outgo; 1,510 responses
Comparison 1:	Compare response rates, scores and demographics of the 2 sampling methods	Compare response rates, scores and demographics of the 2 sampling methods
Comparison 2:		Assess length effect	Assess length effect
Comparison 3:	Calibrate HCAHPS measure of healthcare satisfaction with SHEP overall quality (uses contemporaneous SHEP surveys as comparison)	Calibrate HCAHPS measure of healthcare satisfaction with SHEP overall quality (uses contemporaneous SHEP surveys as comparison)

Because the HCAHPS sampling protocol differs from the SHEP sampling, and it will be necessary to adhere to the HCAHPS sampling protocol after the transition to HCAHPS, the first comparison will be to assess differences in response rates, scores on overall quality, and characteristics of the two samples. Comparison 2 will assess scores, unit non response, and item non response between the SHEP HCAHPS short form and long form (length effect). Comparison 3 will be used to calibrate the SHEP measure of overall quality (poor, fair, good, very good, and excellent) with the HCAHPS measure of satisfaction with healthcare (11 point scale). This comparison will use the existing, contemporaneous administration of SHEP as a comparison. This will be necessary to establish any trending for the future.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The short term nature of this pilot and its limited scope make the use of automated, electronic, mechanical, or other technological collection techniques too costly to implement. Additionally, it is the position of VHA, based upon contractor feedback, that electronic submission of satisfaction surveys may corrupt the statistical validity of the data. Since individual patients complete the responses in their homes, there is no way to utilize technology to decrease the respondent burden other than posting the form on the Internet.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

The information to be gathered from these pilot surveys as a whole is unique and not available from any other sources with the appropriate level of specificity required.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

No small businesses or other small entities are impacted by this information collection.

6. Describe the consequences to Federal program or policy activities if the collection is not conducted or is conducted less frequently as well as any technical or legal obstacles to reducing burden.

Conducting the pilot study of the various versions of the HCAHPS and Inpatient SHEP surveys will enhance the VHA’s ability to understand differences in response patterns between the different pilot groups and interpret possible changes in performance scores and response rates, and identify changes that can be attributed to differences in the survey instrument versus those that can be attributed to improved system performance. The burden consists only of that information which is essential to maintain the validity and support the goals of the Executive Order. Given that the private sector of healthcare plans to adopt HCAHPS and that the Joint Commission may soon require HCAHPS to evaluate care for non-VA inpatient healthcare, the VA needs to be prepared to compare our data with outside agencies.

7. Explain any special circumstances that would cause an information collection to be conducted more often than quarterly or require respondents to prepare written responses to a collection of information in fewer than 30 days after receipt of it; submit more than an original and two copies of any document; retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years; in connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study and require the use of a statistical data classification that has not been reviewed and approved by OMB.

There are no special circumstances that require the collection of information to be conducted in a manner that is inconsistent with the guidelines in 5 CFR 1320.6.

8. a. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the sponsor’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the sponsor in responses to these comments. Specifically address comments received on cost and hour burden.

The notice of Proposed Information Collection Activity was published in the Federal Register on July 13, 2006 (Volume 71, Number 134, Page 39704). We received no comments in response to this notice.

b. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, clarity of instructions and recordkeeping, disclosure or reporting format, and on the data elements to be recorded, disclosed or reported. Explain any circumstances, which preclude consultation every three years with representatives of those from whom information is to be obtained.

VA, through the office of Quality and Performance, has expert staff available for advising, consulting, and working with individual facilities regarding local survey efforts. We are working in close concert with AHRQ, the originators of the HCAHPS survey, and with CMS and DOD, both current users of HCAHPS. In addition, there are a number of private sector and educational institutions that concentrate on satisfaction surveying which are available as external resources to all agency employees. These are utilized whenever necessary.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

There are no plans to provide payments or gifts to respondents.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statue, regulation, or agency policy.

These surveys are anonymous. Names and personal identifiers will be used to locate survey participants and will thereafter be stripped from any files as well as reports.

Each patient who participates is assured confidentiality. It is recognized that the survey must be completely voluntary to provide reliable results. Survey instructions to patients specify and underscore that responding to the survey is completely voluntary, confidential, and will have no effect on entitlement to or eligibility for VA medical benefits, and that the form does not need to be signed. The patient completes the questionnaire anonymously (giving neither name nor social security number). All returned survey documents are destroyed once the dataset created from those documents has been validated. In the many years that the VA has been conducting similar types of surveys, there has never been a single complaint by a veteran concerning a violation of this confidentiality pledge. Since the responses are not individually identifiable, there is no need to store or process these forms in accordance with the Privacy Act. Nonetheless, the VA adheres to 38 U.S.C., Section 3305, which mandates the confidentiality of medical quality-assurance records.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private; include specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

There are no questions of a sensitive nature.

12. Estimate of the hour burden of the collection of information:

a. The annual burden is estimated at 3,625 hours. The details are shown below:

	Target Sample	Respondents	Minutes	Equals	Divided by 60	Annual Burden Hours
Pilot 1: New SHEP HCAHPS Short Form survey (Existing sampling design) - 10-1465-2
	2,700	1,510	15	22,650	60	378
Pilot 2: New SHEP HCAHPS Short Form survey (HCAHPS sampling design) – 10-1465-2
	2,700	1,510	15	22,650	60	378
Pilot 3: New SHEP HCAHPS Long Form Inpatient survey (HCAHPS sampling design) – 10-1465-1
	2,700	1,510	20	30,200	60	503
TOTAL	8,100	4,430	50	75,400	60	1,259

b. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB 83-I.

The separate burden hour for each form is shown in subparagraph 12a.

c. Provide estimates of annual cost to respondents for the hour burdens for collections of information. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 14 of the OMB 83-I.

The cost to the respondents for completing these forms is $18,885 (1,259 hours x $15 per hour). We do not require any additional recordkeeping.

13. Provide an estimate of the total annual cost burden to respondents or recordkeepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

a. There are no capital, start-up, operation or maintenance costs.

b. Cost estimates are not expected to vary widely.

c. There are no anticipated capital start-up cost components or requests to provide information.

14. Provide estimates of annual cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies also may aggregate cost estimates from Items 12, 13, and 14 in a single table.

The cost to the government will be comprised of the following items:

a. A contractor has quoted a figure of $102,900 to print, mail, collect, and scan the survey for data entry. They will also develop and deliver the analysis data set to OQPDC.

One GS 14/step 3 analyst and one GS 13/Step 4 analyst will perform the analysis and prepare the report.

FEDERAL GOVERNMENT PROCESSING /ANALYZING COSTS
	NUMBER	HOURLY SALARY	HOURS EACH	TOTAL
CONTRACTOR CHARGES
Print, Mail, Collect and Tabulate Surveys				$102,900
ANALYSIS
1 GS 14/5	1	$42.24	80	$3,379
1 GS 13/5	1	$35.75	80	$2,860
TOTAL				$109,139

15. Explain the reason for any program changes or adjustments reported in Items 13 or 14 of OMB 83-I

This is a new collection so all burden hours are considered a program increase.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Results of the pilot study will be analyzed by Office of Quality Performance (OQP) staff to assess the performance of each of the 3 versions of the survey in the veteran population. Management decisions about the relative performance of the five different survey instruments will be based primarily on measurable differences in the response rates achieved and on differences in the positive score percentage on overall satisfaction. In addition, individual item response rates will be analyzed. There are no plans to publish any results of this study at this time.

17. If seeking approval to omit the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

VA continues to seek to minimize its cost to itself of collecting, processing and using the information by not displaying the expiration date. Therefore, we request an exemption that waives the displaying of the expiration date on VA Forms. Inclusion of the expiration date would place an unnecessary burden on the respondent as we have found that the term “expiration date” confuses respondents.

18. Explain each exception to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB 83-I.

There are no such exceptions.

B. collections of Information Employing Statistical Methods

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each stratum. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

The universe consists of 22,000 VA inpatients discharged in a given month from VHA Medical, Surgical, Psychiatry, Rehabilitation Medicine, Neurology, or Spinal Cord services to the community, and admitted and discharged from the same service. Of these, 8,100 will be randomly selected by one of two methods to receive one of three survey versions, comprising the three arms of the pilot study. The overall response rate, after two waves of mailing, is projected to be 56%.

2. Describe the procedures for the collection of information, including:

• Statistical methodology for stratification and sample selection

Because the SHEP sampling design differs from the mandated HCAHPS protocol sampling design, one objective of the study will be to assess the effect of sampling design on study population demographics, health status, performance measure scores, and response rate. To measure this sampling effect, the identical SHEP short form core instrument (Arm 1 and Arm 2) will be sent to equal numbers of patients, selected by one of the two methods. 2,700 patients will be selected using the existing SHEP sampling method, using discharging bed sections to identify medicine, surgery, psychiatry, spinal cord injury, neurology, and rehabilitation medicine patients; another 2,700 will be selected according to the HCAHPS method, using Diagnosis-related groups (DRG’s) to identify medicine and surgery patients, and excluding psychiatry patients. For Arm 3, 2,700 patients will be selected to receive the SHEP HCHHPS Long form according to the HCAHPS method. Therefore, 8,100 eligible patients will constitute the total sample; a 56% response rate has historically been achieved in the SHEP inpatient survey, so the study sample will be 4,531 respondents.

• Estimation procedure

Analysis plan: Positive scores for SHEP inpatient overall quality (the percent of respondents rating the quality of VHA care very good or excellent), HCAHPS rating of quality of healthcare, and response rates will be estimated in this study. Differences in these positive scores and response rates based upon sampling method, short form versus long form, will also be estimated. Analysis will consist of ANOVA, with response rate and score as the dependent variable and study arm as the independent variable. Health status (2 categories), age (2 categories), and gender will be included in models. Differences within 5 percentage points will be estimated using a 95% confidence interval. In addition, all of these estimates will be compared to contemporaneous estimates obtained from the ongoing SHEP survey itself. This will allow for calibration of the new HCAHPS measures with the existing SHEP measures.

• Degree of accuracy needed

Currently, inpatient overall quality scores run in the range of 78%, and response rates for inpatient SHEP with a second wave of mailing are projected to be 56%. We will be testing for differences in the magnitude of plus or minus 5 percentage points for overall quality scores and for response rate.

We are seeking a degree of accuracy of plus or minus 5 points on the measure of inpatient overall quality and on response rate, and a 95% confidence level for each of the five survey instruments, and the two sampling methods. No attempt will be made to detect score differences at the level of the 160 VA medical center facilities.

• Unusual problems requiring specialized sampling procedures

The pilot will be post-stratified to look at difference sin age, health status, and gender. Network differences will be assessed on a post-hoc basis.

• Any use of less frequent than annual data collection to reduce burden

The pilot survey project is a one-time effort, and will be conducted in the course of one month.

3. Describe methods to maximize response rate and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

The proposed sample of 8,100 will yield a response rate of 56% or 4,531 responses. We will maximize the response rate by utilizing a personalized prenotification letter from the VA, by prominently featuring the VA seal on the survey, by using an official VA cover letter, by sending a second wave of mailing to non responders three weeks after the first wave, and by sending a thank you/ reminder postcard. We will assess non-response bias using a model-based weighting approach to adjust for non response propensity of the various population subgroups. The proposed sampling scheme will result in ~ 72 respondents per survey arm per VISN, which will give the desired accuracy of the estimate of overall satisfaction for purposes of comparing the 3 surveys. The proposed sample size was calculated by:

Number of respondents= (Np)*(p)*(1-p) *12 (strata) = 4,531

((Np-1)*(B/C)²) + (p*(1-p))

Where: Ns=number of respondents

Np=number in population (22,000)

P=proportion of responses (0 or 1; 50%)

B=margin of error of estimate (0.05)

C=Z-score for 95% level of confidence (1.96)

Formula has been adjusted for stratification by 2 categories of health status, 3 categories of age, and 2 categories of gender.

Formula taken from Dillman D.A. Mail and Internet Surveys, Wiley, 2000, P 206

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions of 10 or more individuals.

This project itself represents a test. The purpose of the test is to determine the effects of survey length, survey content, sampling scheme, and survey formatting on scores and on response rates. In addition, we aim to determine the relationship between the existing SHEP overall quality measure and the new HCAHPS measure of satisfaction with healthcare, with adjustments for length effects. The results of the study will help us to trend results from the existing SHEP to the new HCAHPS-based SHEP after the transition.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Statistical aspects of this design were determined within the agency by John Elter, PhD (919-993-3035, Ext 224), Trang Lance, MPH (Biostatistician), and James Schaeffer, MPH (Biostatistician). The data will be collected and tabulated by NRC-Picker Corporation under contract, and the analyses will be performed by John Elter, PhD and Trang Lance, MPH.

Page 9