OMB Supporting Statment Part B 9 23 11_V3

Evaluation of the Substance Abuse and Mental Health Administration

Primary Care Behavioral Health Integration Grant Program

OMB Supporting Statement

B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection Methods to be Used

• Intervention and control sites: The ten grantee sites that will serve as intervention sites for RQ2 will be chosen from the 56 grantees based on five criteria: 1) Potential to identify a well-matched control site, 2) The quality of their data infrastructure, 3) A target population of greater than 750 individuals, 4) A target population that meets the SAMHSA SMI definition, and 5) The quality of the PBHCI program data submitted to date. Sampling of grantees for RQ will not be done at random. We will specifically select programs that reflect the diversity of the grantees in terms of location, populations served and integration models utilized, as well as those with sufficiently large patient populations and electronic data infrastructure that can support the evaluation. While our research design for RQ2 is quasi-experimental (we will select matched-control sites for each intervention site), we will also treat intervention sites as a series of case studies. Specifically, we will note in our report that sites with the potential were selected from the population of grantees in order to provide a strong exemplar of integrated program implementation.

The 10 matched control sites will be chosen based on 1) the absence of primary care services, 2)geographic proximity to their matched intervention sites (within the same state and preferably within the same county), 3) similarity to the intervention site in the size and characteristics of the population served and 4) similarity in the mental health services delivered. One limitation of our design is that there may be differences in the type and degree of match between individual intervention sites and their unique matched control; this is because we can only select intervention sites from the existing pool of community behavioral health clinics already in existence that meet our inclusion/exclusion criteria. In order to ensure that we have the best possible intervention-control site matched pairs, we will consider more than the necessary number of potential sites for the evaluation and then select those with the most potential to support the evaluation with the highest level of scientific rigor. We will also describe these limitations of the study design in any presentation or written summary of our results.

• Site visit interviews for leadership, care coordinators, and physical health and mental health providers: The evaluator will work with the Project Directors at each of the sites to be visited to identify individuals to include in each discussion. All staff who have significant roles in the implementation of the PBHCI program at each site will be included in site visit interviews. Information from grantee proposals and ongoing quarterly reports will allow the evaluation team to identify these staff by position. The Project Directors at each site will provide any updates to staffing mix, name individuals in positions, and facilitate meeting arrangements. Both individual and group interviews will be conducted as part of the site visit. The protocol for conducting individual interviews with those in certain positions, and group interviews with those in other positions, will be consistent across grantee sites. It is expected that 3-4 sites will be chosen from the first cohort and 6-7 will come from the second cohort.

• Web-based Survey: Project directors at each grantee site will receive an e-mail inviting them to participate in the survey and also asking them to forward the invitation e-mail to other appropriate staff (primary care providers, mental health providers, care managers, etc.) at their site.  A concerted effort will be made to field the web survey prior to the site visits, as timing of site visits permits.

• Client Physical Exam and Survey: PBHCI intervention sites will provide the evaluator with a de-identified list of clients who have enrolled in integrated care services, and the control sites will provide a complete de-identified list of adult clients to the evaluator for participation in the physical exam and survey. All clients within an enrollment time period will be invited to give consent to be contacted to participate in the study. The evaluator will provide client contact information to the contractor, who will contact all clients who have given consent and invite them to participate in the study. For most sites, we expect to contact all clients to invite them to participate. If a site is very large and in excess of 600 clients (at control sites) and 500 clients (at intervention sites) have given consent to be contacted, the evaluator will randomly select those to invite. We expect an average of n=300 client physical exams and surveys completed for the control sites and n=250 client physical exams and surveys completed for the intervention sites. The evaluator will complete physical exams with n=300 clients from the control sites at baseline so that (assuming ~15% attrition) n=250 participants complete the physical exam and survey per at 1 year follow-up. The evaluator will complete n=250 physical exams/surveys from the intervention sites because they will only complete the physical exam and survey at follow-up (baseline data will be extracted from medical records / registries and the TRAC tool). We propose this final sample size because a) we expect that most sites will be able to accrue at least n=250 patients by the middle of their second year of grant activities; b) it will give the evaluator > .80 power (alpha = 0.05) to detect all of the 7 outcomes of interest across all sites (RQ2); and c) it will give the evaluator >.80 power (alpha =0.05) to detect 5 of the 7 outcomes of interest (smoking cessation, high blood pressure, diabetes, and obesity) within clusters of 3 sites. That is, with this level of power, the evaluator can reliably answer questions about which models and model features (RQ3) are related to 5 of the 7 physical health outcomes of interest, provided that the evaluator can identify at least 3 intervention sites that share the model or model feature of interest. Within this size sample, the evaluator might also be able to address hypotheses related to RQ3 and the remaining 2 physical

health conditions (hypercholesterolemia, hyperlipidemia) if effect sizes are large.

2. Procedures for the Collection of Information

Client Physical Exam and Survey: Prospective participants will be asked by a staff person at their behavioral health center if they would like to be contacted in order to learn about a research study. Points of contact at the sites will be asked to have their case managers bring up the study to all eligible clients with serious mental illness. They are told to provide clients with a brief description of the study (e.g., it involves a physical exam and an interview, and that they will receive a gift card for $20 upon completing it) and to answer any questions consumers may have. RAND will review the Release of Name and Contact Information document with the point of contact in order to ensure that they have a full understanding of the study and the sign up process, and ask that they in turn review this document with case managers. RAND will provide standardized recruiting material to sites (e.g., posters, discussion guides for case managers) to assist sites in their recruiting efforts. Staff at the behavioral health clinic will compile the list of persons who agree to be contacted and they will send this list with names and contact information to RAND over a HIPAA-compliant secure SharePoint site. Staff at the sites will also keep a tally of the number of clients that have declined to be contacted so that response rates can be calculated for each site. Although we cannot know the demographic and clinical characteristics of the persons who decline to be contacted without creating greater data collection burden, we can and will compare these characteristics of the participating sample to the characteristics of each site’s overall PBHCI clientele (NOMs data can be used for this purpose). This comparison will help us to determine if we have recruited a representative sample of participants for this portion of the study. RAND will securely transmit the list of clients who wish to be contacted to our subcontractor, OnSite Health Diagnostics, who will send a letter (Attachment 13) to prospective participants describing the study. Letters will go out 1-2 weeks prior to the data collection date with information about the study and an invitation for consumers to come to their behavior health clinic on a specific date to participate. OnSite Health Diagnostics will also call prospective participants 1-2 days prior to the data collection date to provide additional information about the study and to further invite consumers to participate in the study (Attachment 14). Participants who do not show-up on the day of data collection but who had indicated that they would like to participate will be called by OnSite Health once more and invited to participate in the study on a second date shortly thereafter.

Individuals who have provided written informed consent (Attachments 14, 15) for primary data collection will participate in a physical health exam. The physical health exam will be conducted by trained and experienced staff from OnSite Health Diagnostics -- a national biometric and diagnostic health screening company (HIPAA compliant). OnSite Health will perform the following patient assessments during the physical health exam: Height, weight, body mass index (calculated from height and weight), blood pressure, waist circumference, breath carbon monoxide (an indicator of tobacco smoke exposure; participant blows into a tube). OnSite Health Diagnostics’ licensed and trained phlebotomists will also collect a blood sample (approximately 3 tablespoons) for the following tests: blood sugar and HgBA1C (diabetes indicators), cholesterol (total, HDL and LDL), and triglycerides. OnSite staff will also ask participants if they succesfully fasted for 8 hours prior to the health exam; an 8 hour fast is necessary prior to the blood tests for diabetes indicators. OnSite staff will issue $20 gift cards (to a local, major retailer e.g., Walmart) to participants who have completed the physical health exam and survey at the end of the 40 minute session.

• Site visit interviews for leadership, care coordinators, and physical health and mental health providers: Information will be gathered in small group interviews arranged by the evaluator and the grantee project director or control site leader, led by a researcher, and recorded by a note-taker. A written statement of informed consent (Attachments 11, 12) will be provided to each participant prior to the interview and will be reviewed with the group by the researcher.

• Web-based Surveys: Participants will access the survey through a web link provided in the invitation e-mail. Participants will provide informed consent for the survey using an electronic signature and all survey information will be maintained on a secure system. Respondents will be identified by unique numeric IDs and not names or other identifying information.

• Individual Service Utilization Data: Data on client-level service utilization will be extracted from clinical registries and/or medical records (depending on each sites’ data management system) and submitted via an Excel spreadsheet (Attachment 17) or similar database file (e.g., CSV, DBF) to the secure RAND SharePoint site. RAND and the SAMHSA Center for Integrated Services provide technical support to sites regarding data definitions and submission, including individual consultations with sites and webinars for all grantees.

• Quarterly Reports from grantees: The Project Director from each site will assemble the necessary information and submit it electronically in a Word document to the secure RAND SharePoint site.

• Physical Health Indicators to be reported through TRAC: Physical health data will be extracted from grantees’ clinical registries and/or electronic medical records and entered into the TRAC system at the time each consumer completes his/her TRAC-required bi-annual client interview; entering all TRAC data simultaneously will minimize grantees’ data management burden.

3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response

Client exam and survey: Several elements have been considered to maximize client participation in the exam and survey. In addition to payment for the individual’s time, the exam will take place at the center at which the client usually attends appointments, and it will be scheduled to coordinate with an existing appointment when possible. To address the potential issue of response bias, RAND will compare the characteristics of participants in the client physical exam and survey with the characteristics of sites’ overall PBHCI clientele using NOMs data. This comparison will help us to determine if we have recruited a representative sample of participants for this portion of the study.

For the control sites, it is also an opportunity for clients to be screened for physical health conditions and referred to appropriate care, if necessary. We do, however, project that there will be some attrition in the control group and have increased the initial sample by 15% to account for that.

Web-based survey: In order to ensure that we obtain sufficient participation in the web-based survey, project directors will assist with recruitment of eligible staff from their sites. Specifically, project directors will receive and e-mail inviting them to participate in the survey which they will be asked to then forward to the appropriate, eligible staff at their site. Project directors will also receive a series of e-mails following the original invitation encouraging them to make sure that all appropriate staff complete the web-based survey. At the end of the survey, all respondents will be encouraged to provide the name of their site for tracking purposes, which will not be linked to their survey responses. RAND staff will then be able to prompt project directors with a personalized e-mail noting if overall response from their site is low. Respondents will have 1 month to complete the survey following distribution of the original invitation e-mail.

4. Tests of Procedures or Methods to be Undertaken

The evaluation design team sought to use instruments/questions that have been tested previously. For example, the questions for the site visit interviews and web-based survey came from the Assessment of Chronic Illness Care, the NCQA standards for Patient Centered Medical Homes and the VA Survey of Mental Health Services (VA), among others. The client survey is primarily made up of questions from the TRAC client-level survey with several from the National Health and Nutrition Examination Survey (NHANES) and the National Survey on Prescription Drugs. The quarterly report has been pilot tested with the original 13 grantees during their first year of implementation, and modifications were made based upon their questions and feedback.

5. Consultants on Statistical Aspects of the Design and Persons Who Will Collect and Analyze the Information

The evaluation design was conceived by the RAND project team in conjunction with the representatives from SAMHSA and ASPE and with input from the Technical Advisory Group. Members of the RAND project team include (in alpha order): Audrey Burnam, Nicole Eberhart, Bing Han, Susan Lovejoy, Harold Pincus, Marcela Horvitz-Lenon, and Deborah Scharf. Representatives of SAMHSA include (in alpha order): Crystal Blyler, Trina Dutta, Fran Randolph, and Ken Thompson. Representatives of ASPE include (in alpha order): Hakan Aykan, Kirsten Beronio, Richard Frank, and William Marton. The evaluation will be conducted by RAND under contract with ASPE.

List of Attachments

Attachment 1: Site visit interview with leadership

Attachment 2: Site visit interview with mental health providers

Attachment 3: Site visit interview with primary care providers

Attachment 4: Site visit interview with care coordinators

Attachment 5: Site visit interview with key staff (control sites)

Attachment 6: Client physical exam and survey

Attachment 7: Individual client service utilization

Attachment 8: Quarterly reports from grantees

Attachment 9: Physical health indicators to be reported through TRAC

Attachment 10: Web-based survey

Attachment 11: Consent protocol for site visit interviews-grantee sites

Attachment 12 Consent protocol for site visit interviews-control sites

Attachment 13: Initial client contact letter for exam and survey

Attachment 14: Telephone script for client exam and survey

Attachment 15: Baseline consent for client exam and survey-control sites

Attachment 16: Baseline consent for client exam and survey-intervention sites

Attachment 17: Individual service utilization codebook

Attachment 18: RAND IRB package