Justification B Veterans Outcome Assessment (2)

The Veterans’ Outcome Assessment (VOA)
OMB 2900-XXXX

VA Form 10-211017

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each strata. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

The potential respondent universe of Veterans encountering a new episode of mental health care will include all Veterans with a mental health encounter (mental health encounter stop codes that should result in a meaningful episode of mental health care) and no mental health encounter in the previous 9 months. In FY12, this represented 460,635 Veterans. In consultation with statistical experts, a sampling procedure has been developed to randomly select from these new mental health care patients, stratified by VISN (the 21 Veterans Integrated Service Networks). VISNs are relatively similar in the size of their eligible patient population (with a median of approximately 20,000 and a range from 9,645 to 40,770). In order to ensure that reliable comparisons can be made at the VISN level, each VISN will be equally represented in the overall sample, and, where relevant, national data will weight VISNs by their population of eligible Veterans.

Our goal is to collect 400 baseline and follow-up surveys per quarter. Our sampling estimates are based on previous experience with the Veterans Health Outcomes Improvement Project (VHOIP), a similar call-center based outcomes collection pilot project, which found about 40% of all calls resulted in contact with a Veteran, 50% of all contacts resulted in a Veteran starting a baseline survey, and about a 60% follow-up rate 3-months later (12% of initial sample completing follow-up survey). Based on this experience, we conservatively estimate that in order to collect baseline and follow-up surveys from 400 patients each quarter, we will need to make about 3300 calls, successfully contact 1200 Veterans, and administer about 670 baseline surveys each quarter, with successful follow-up completion with 400 respondents. We note that the present collection is briefer and has more streamlined questions than the VHOIP survey, and we therefore expect higher rates of completion than these.

2. Describe the procedures for the collection of information, including:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed

• Unusual problems requiring specialized sampling procedures

• Any use of less frequent than annual data collection to reduce burden

On a weekly basis, all eligible patients (patients with a new episode of mental health treatment occurring in the previous week) will be identified and checked to ensure that none are duplicates from any previous draws. From these eligible patients, stratified by VISN, a random sample of 12 Veterans will be drawn from each VISN. The call center will be provided with contact information for these veterans, with completed baseline and 3-month follow-up surveys expected from approximately 12%. Our initial plan assumes similar contact and response rates by VISN, but if this assumption proves incorrect, the sampling rates will be adjusted accordingly.

Precision analysis was used to determine the sample size needed in order to detect small effects at the national level on a quarterly basis and medium effects at the VISN level on a yearly basis.

3. Describe methods to maximize response rate and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Each veteran in the sample will be called at least 10 times during the first two weeks following initiation of mental health services. The protocol describes the number and timing of telephone contact attempts, which are spaced across different days and varying times of day. Call schedules will ensure that calls can be made in the morning and evening across all time zones. When contact cannot be made or in case of disconnected or wrong phone numbers, staff will determine whether updated contact information is available. When a Veteran is successfully contacted, the script will emphasis the importance of their input for improving the quality of mental health care in VA, and the Veteran will be asked whether he or she is willing to participate. For those not willing to participate immediately, a follow-up call will be scheduled. For reluctant respondents, the importance of the survey and its brevity will again be emphasized; anyone unwilling to participate at that point will be thanked and not contacted again.

To maximize the follow-up response rate, Veterans completing a baseline will be called at least 10 times for the follow-up survey; these contacts will begin 3 months after initiation of the mental health treatment episode. Veterans’ contact information will be updated quarterly to ensure that these calls are as productive as possible.

Some information will be available on the demographic characteristics and treatment histories of Veterans who cannot be contacted, are contacted but refuse to participate, complete the baseline survey only, or complete both surveys. These data will allow us to compare these groups and to begin to understand any non-response bias.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions of 10 or more individuals.

Feedback on the survey measure from the VAPAHCS Veteran & Family Advisory Council (9 Veterans) was obtained on Tuesday, July 23, 2013. For further information on this Council, see: http://www.paloalto.va.gov/vfcc.asp

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals consulted on statistical aspects of the design:

• Thomas R. Bowe, Statistician, Program Evaluation and Resource Center, (650) 493-5000 x22666

• Alex Sox-Harris, Associate Director, Program Evaluation and Resource Center, (650) 493-5000 x23423

VA units that will actually collect and analyze the information for VA:

• NorthEast Program Evaluation Center (NEPEC), VA Connecticut Healthcare System (182), 950 Campbell Avenue, West Haven, CT 06516 (203) 937-3851. Rani Hoff , PhD, MPH, Director

• Program Evaluation and Resource Center, VA Palo Alto Healthcare System (152MPD), 795 Willow Rd, Menlo Park, CA 94025 (650) 493-5000 x22808. Jodie Trafton, PhD, Director

Page 3