BHW SUD Supporting Statement B -- NORC response CLEAN

Supporting Statement B

Bureau of Health Workforce Substance Use Disorder Evaluation

OMB Control No. 0906-XXXX-New

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The Bureau of Health Workforce (BHW) Substance Use Disorder (SUD) Evaluation does not employ any statistical methods for selecting respondents. The decision not to use statistical methods to select respondents was based on two reasons: (A) survey data are intended to complement already-available administrative data in a manner that maximizes the utility of administrative data enabling the most comprehensive evaluation possible; and (B) the diversity of the programs and the possible combination of types of sites, organizations, and individuals necessitates data collection from all eligible respondents to ensure an understanding of efforts to strengthen the SUD workforce.

Below we describe the available administrative data and the diversity across the survey respondent populations.

A. Available administrative data

The six types of respondents for this evaluation are: [1] National Health Service Corps (NHSC) sites, [2] NHSC Loan Repayment Program (LRP) Participants, [3] grantees in three BHW education and training programs—Behavioral Health Workforce and Education Training program (BHWET), Graduate Psychology Education program (GPE), and the Opioid Workforce Expansion Program (OWEP), [4] trainees of each of the grant programs, [5] partner sites of the grantees, and [6] alumni of the education and training programs. The Health Resources and Services Administration (HRSA) has collected administrative data for all six types of respondents as part of routine grant administration, including applications, performance data, and progress reports.

The administrative data contain certain background characteristics of the respondents, for example, the types of sites, and the education, training, and demographic backgrounds of NHSC participants and trainees. HRSA has invested a great deal of resources in administrative data systems and has engaged in efforts to facilitate greater use of administrative data for program monitoring and evaluation purposes. Data linkage, particularly administrative data linkage, has been an important part of the evidence-based policy act (PL-114-140 § 4(a)(1)), and complete information on all respondents strengthens the knowledge base for data-driven policy formulation. It is important that such data be used in a manner that helps guide HRSA programming.

While the administrative data are useful to understand who is participating in the programs, and can help characterize the diversity of the respondents in the evaluation, they do not capture key process and outcome information. To maximize the use of the administrative data, they will be merged with primary data collected through survey instruments. The primary data collection captures key process and outcome information that is lacking in the administrative sources. Having complete information, from multiple sources of data on all respondents, will enable HRSA to assess and understand performance in a wide variety of settings, understand impacts, and reasons for variation in performance, and guide future program decisions. The data collected can also allow HRSA to conduct subsequent, comparative analyses that may not be achieved within the timeline of this contract.

The primary data collection part of the evaluation will also rely on administrative data for outreach to the units of analysis. This reliance is because the administrative data contain the respondents’ email addresses, and the email address is a common identifier. Even though HRSA has taken great efforts to ensure the high quality of the administrative data, the primary data collection will be able to provide HRSA with insight on the quality of the common identifier—the validity of the email address. It is likely that the email address for some respondents may have been entered incorrectly (typos) or the email account is no longer valid, defunct, or no longer used by the respondent. In such cases, we will not be able to collect primary data; however, we will be able to assess for whom we cannot collect data, and study any systematic variation in nonresponse. Because there will be a certain, albeit small, percentage of non-matches, it will be prudent to reach out to all eligible participants to minimize nonresponse from any subgroup.

Collecting data on all NHSC LRP participants, grantees, and trainees can help HRSA understand where and how technical assistance should be targeted for the specific type of health professional and setting. It will allow HRSA to develop policies and guidance for specific operational goals. For example, gathering primary data (via the data collection instruments) from all grantees will help HRSA understand the many different recruitment and training efforts underway, and where future programs should target grant goals. In addition, there is minimal added cost for collecting data on all units, while there is much to be gained in the way of program monitoring, evaluation, and oversight. The data collection activities will not have a significant impact on any subjects while enabling all types of sites, grant programs, and individuals to contribute to the evaluation.

B. Diversity of programs

The NHSC programs’ sites and participants, the grantees, and their trainees and sites are extremely varied in their characteristics, settings, goals, and challenges. They also serve an extensive variety of patients, who have different socio-demographic characteristics and health care needs. The data collection instruments will capture the variation in characteristics and performance not captured elsewhere in the administrative data, and provide HRSA with a complete picture of all those delivering health care, as well as the recipients of health care services.

Given this diversity in respondents, the evaluation team determined that sampling could not ensure a representative sample and would not likely improve the robustness of data gathered. Furthermore, there were no reasonable criteria for purposefully sampling among the programs. As a standard objective, we aim to achieve a high response rate (80 percent). However, if our response rate is lower, performing a census can help us ensure the representativeness of subgroups, should a response be less than 80 percent.

A stratified sample may not yield sufficient responses to produce representative samples for all the various subgroup analyses that would need to be conducted to capture the diversity of outcomes across the different types of respondents. This is because the sample may not yield sufficient statistical power to determine variation in outcomes across the types of respondents. For example, to capture sufficient responses across all HRSA regions, we would need to capture all available respondents, not a sample of them, to ensure sufficient responses. Similarly, we would need to capture responses from all types of NHSC sites to examine differences across sites. If we do not achieve sufficient responses in these strata, even with a census approach, we will aggregate across groups. The census approach would be the optimal way to ensure that all groups are represented.

Collecting standardized data across the population of programs will allow the identification of similarities and variations in program efforts. The standardized information will inform HRSA leadership on program progress, permit corrective program actions, and provide actionable information for future program incentives and investment strategies. It will also allow HRSA to assess gaps in training and service delivery to populations in underserved areas.

Below we describe the diversity across the survey respondent populations.

NHSC Sites and Grantee Program Training Sites: These health care service delivery sites will vary with respect to HRSA priority area, Health Professional Shortage Area score, rural status, facility staffing mix, service mix, community characteristics, volume of patients served, payer mix, and technical needs. In addition, the NHSC sites may be any of the following types of facilities:

• Community Mental Health Centers (CMHC)

• Federally Qualified Health Centers (FQHC)

• American Indian Health Facilities

• Certified Rural Health Clinics

• State prisons

• Critical access hospitals (CAH)

• State or local health departments, community outpatient facilities (hospital-affiliated and non-hospital-affiliated)

• Private practices (solo or group)

• School-based clinics

• Mobile units

• Free clinics, or SUD treatment facilities, including Substance Abuse and Mental Health Services Administration (SAMHSA)-certified opioid treatment programs (OTPs)

• Office-based opioid treatments (OBOTs)

• Non-opioid outpatient SUD treatment facilities
  
  

NHSC Participants (Loan Repayment Program Awardees): Fourteen types of professionals may apply for loan repayment under this program and will vary in demographic, socioeconomic, and professional experience, including status of Drug Addiction Treatment Act of 2000 (DATA 2000) waiver approval.

Grantee Organizations: HRSA seeks to evaluate each type of education and training program, which vary by curriculum, trainee type, and types of training sites across three grant programs (OWEP, BHWET, and GPE). These programs vary with respect to:

• The types of institutions that could apply: e.g., accredited institutions of higher education or accredited behavioral health professional training programs; accredited master’s or doctoral-level programs in social work; accredited doctoral, internship, and post-doctoral residency programs of health service psychology

• The disciplines of training offered: psychiatry, psychology, school psychology, behavioral pediatrics, psychiatric nursing, social work, school social work, SUD prevention and treatment, marriage and family therapy, occupational therapy, school counseling, or professional counseling

• The activities and curriculum developed

• The number of trainees

Grant Program Trainees: Recipients will vary in discipline, specialty, training background, funding status, demographic characteristics, disadvantaged background, or veteran status.

The survey component of the evaluation will supplement, without duplicating, administrative data in a way that allows for a complete assessment of all units of analysis (NHSC LRP sites and participants, grantees, training sites, and trainees). To date, there has not been a census of the expansion effort or comprehensive assessment of these programs. A thorough understanding is critical to ensure that program implementation is succeeding. The totality of information is necessary to enable HRSA to assess the success of the funding and also ensure that funded organizations have demonstrated adequate training to the behavioral health workforce.

Respondent Universe

Below we define the eligible respondents for each survey.

NHSC SUD Workforce Loan Repayment Program/NHSC Rural Communities Loan Repayment Program/NHSC Loan Repayment Program- Participant Survey (NHSC LRP Participant Survey): All LRP participants who received awards in each fiscal year (FY) from 2019 to 2021 will be invited to respond.

NHSC Loan Repayment Programs- Site Survey (NHSC LRP Site Survey): All sites that were approved to receive NHSC resources in FY2019 to FY2021, regardless if they currently have a NHSC LRP participant on staff, will be invited to respond.

Grantee Training and Educational Programs- Grantee Organization Survey (Grantee Organization Survey): All grantee organizations that received awards in FY 2018 for the BHWET program, and received FY 2019 awards for the GPE and OWEP, will be invited to respond.

Grantee Training and Educational Programs- Trainee Survey (Grantee Trainee Survey): All individuals identified by a grantee as receiving funding (and who did not leave the program early) as part of the grantee training and educational programs will be invited to respond.

Grantee Training and Educational Programs- Alumni Trainee Survey (Grantee Alumni Survey): All individuals who received funding from the Grantee Training and Educational Programs in FY 2019 or FY 2020, but who had not completed their training at the time of the survey and did not receive funding in subsequent years, will be invited to respond. Respondents will be asked to provide a contact email for this follow-up survey as part of the Grantee Training and Educational Programs- Trainee Survey.

Grantee Training and Educational Programs- Site Survey (Grantee Site Survey): All sites that were approved to receive BHWET, OWEP, or GPE trainees and have had or ever had trainees, will be invited to respond.

The Grantee Organization, Grantee Site, and NHSC LRP Site surveys will be sent to a point of contact, but this point of contact may require information from others in the organization or site in order to complete the survey. As described below in Section 2, the instructions in the introductory email in these three surveys ask the point of contact to determine all relevant persons who can assist in answering the questions and to share the survey with them. These instructions will also be repeated in the surveys.

Response Rate

This is the first time survey data will be collected to evaluate the specific objectives of the opioid workforce investments. We aim to achieve an 80 percent response rate for each of the six surveys and will pursue a robust outreach plan, using general best practices in survey research and the follow-up procedures described below. We discuss our approach to maximize response rates in Section 3 below.

2. Procedures for the Collection of Information

We have designed data collection procedures to support a high response rate, reduce burden to respondents, and promote accuracy and completeness of responses. Specifically, the following steps are planned.

Data will be collected from respondents using a web-based survey that is sent to respondents via a secure email link. We will obtain email addresses for NHSC sites, NHSC participants, and grantee organizations from administrative data. We will request that grantee organizations provide email contact information for their participating sites and trainees. We will give grantee organizations at least two months advance notice of the planned survey to prepare the contact information, and will provide them with an Excel template to complete with the trainee and site contact data and email back to research staff. We will follow up weekly with grantee organizations that have not provided their trainee and site contact data for a period of six weeks. If grantee organizations are still nonresponsive, we will ask HRSA BHW leadership to reach out to these grantee organizations.

To ensure the trainee and site email addresses received from grantee organizations are correct, we will perform quality checks prior to survey implementation. During survey fielding, we will maintain a list of returned emails, which are accounts that are no longer valid or were incorrectly provided. We will follow up with sites and grantee organizations via phone calls (a maximum of three calls per respondent) to correct possible errors in the contact information.

The fielding of each survey will use multiple outreach strategies to engage respondents and maximize response rates. Table 1 summarizes the types and timing of the contact efforts. We will email all respondents to introduce the evaluation team, provide instructions on completing the survey, and send a secure link to the survey. For the duration of the fielding, the evaluation team will carry out outreach strategies tailored to each type of respondent to help obtain a high response rate.

Table 1: Respondent Outreach

Contact Timing	Respondent Type	Contact Details	Mode
First year of survey, during OMB approval	Grantee organizations	Advance introduction from survey team: Introductory email announcing survey and alerting grantee organizations as to the survey team’s upcoming request for trainee and site contact emails	Email
Immediately after OMB approval (START of survey fielding)	Grantee organizations	Follow-up introductory email from survey team reminding grantee organizations of the survey and requesting contact list of trainees and sites	Email
Immediately after OMB approval and survey team request	Grantee organizations outreach to sites and trainees	Grantee organizations reach out to sites and trainees to collect up-to-date contact information	Email
During the first two months following OMB approval	Grantee organizations	Regular follow-up with grantee organizations to ensure contact list of trainees and sites is provided to the survey team	Email
During the first two months following OMB approval	Grantee organizations	BHW Project Directors reach out to grantee organizations that have not responded to requests for site/trainee contact information	Email/phone
During survey fielding (as needed)	Grantee organizations/sites	Reach out to grantee organizations and sites for correct email addresses when invalid email addresses are identified from returned emails or, if needed, to collect this contact information if not yet provided	Phone (max three calls per grantee or per site)
Start of survey fielding	All respondents	Email with link to surveys	Email
Start +1 week	All respondents	Reminder email with link to surveys with additional language urging request to participate, if no indication respondent started survey	Email
Start +2 weeks	NHSC LRP sites and participants, grant program sites, trainees, and alumni	Reminder letter mailed to respondents with known USPS mailing address. The letter will include the HRSA and NORC logos, providing legitimacy and authority to the survey, and encouraging timely response	Mail
Start +3 weeks	All respondents	Reminder email with link to surveys	Email
Start +4 weeks	All respondents	Reminder email with link to surveys	Email
Start +5 weeks	NHSC LRP sites and participants	1. Reminder email with link to surveys 2. NHSC Regional Office Directors send additional email to encourage site participation	Email
Start +5 weeks	Grantee organizations, grant program sites, trainees, and alumni	1. Reminder email with link to surveys 2. BHW Grant Program Project Directors send additional email to encourage grantee participation 3. Grantee organizations prompt grant program sites and trainees to complete the survey	Email
Start +5 weeks	NHSC LRP participants, grant program trainees, and alumni	Text outreach to remind nonrespondents and may include a link to the survey	Text
Start +6 weeks	NHSC LRP sites and participants, grant program sites, trainees, and alumni	“Last chance” additional email	Email
Start +6 weeks	Grantee organizations	1. “Last chance” additional email 2. “Last chance’ phone calls to remind them	Phone (max two calls per grantee)
During survey fielding	All respondents	Include survey announcement in select communications (e.g., newsletters) from HRSA to respondents and possibly professional associations	Ad in newsletter

3. Methods to Maximize Response Rates and Deal with Nonresponse

Below we describe our strategies to maximize response rates, followed by a summary of our approaches to address nonresponse.

Questionnaire design and mode: To help facilitate cooperation and reduce item nonresponse by respondents, attention was placed on creating a logical, clear questionnaire with concrete question wording, closed-ended response choices, simple grammar, and questions grouped according to subject areas. In addition, the web-based Qualtrics survey platform will make it easy for respondents to participate. We will pre-populate fields where relevant to allow respondents to complete the survey more efficiently and skip questions that do not pertain to them. The web-based platform will also allow respondents to save and continue work, which adds convenience in completing the surveys.

Media platforms: During each year of data collection, we will work with HRSA BHW and its existing channels of communication, such as newsletters or other announcements from HRSA, to inform respondents of the surveys and to promote their importance and value. If possible, the survey team will also stand up a webpage for the evaluation surveys, both on the HRSA and NORC websites, in order to provide more background on each survey, post the questionnaires, and promote participation. We will also identify various associations and organizations to which respondents may belong, for example, the National Rural Health Association, and post information about the surveys via their media channels (e.g., listserv, website).

Mid-point outreach from HRSA leadership: During the third to fourth week of survey fielding, we will engage NHSC Regional Office Directors and BHW Project Officers to reach out via email to NHSC sites and grantees, respectively, who have not responded.

Respondent support (or technical assistance): We will provide contact information for the survey team should respondents have questions or concerns about the surveys, as well as contact information for our Institutional Review Board should respondents have concerns about their rights as a study participant. Respondents will be provided a toll-free number to speak directly with staff trained to assist survey respondents and be as responsive as possible in addressing their concerns.

Dealing with nonresponse: For all surveys, respondents who have not completed the survey within two weeks of receiving the link will receive a reminder email from the survey team. Respondents who have not completed the survey within three weeks will then receive a hard-copy letter in the mail. This mailing will be restricted to respondents for whom we have a mailing address, including sites and individuals but excluding grantees. Additional reminder emails will be sent each of the following two weeks. After six weeks, the survey team will pursue additional outreach efforts, such as involving HRSA Program Officers and NHSC Regional Office Directors to encourage grant programs and NHSC sites, respectively, to respond. We may also ask BHW grantees to prompt BHW sites and trainees to complete their surveys. In addition, we may send text messages to NHSC clinicians and BHW grantee trainees and alumni for whom we have working cell phone numbers. We will also conduct telephone reminder calls to grantee organizations. The surveys will be closed approximately eight weeks after opening

We will also consider the need for post-stratification weighting of the NHSC site and for clinician data. We will first determine what strata are relevant for comparisons, such as facility type and clinician background characteristics. Based on the responses received at the close of the survey, we will assess whether any group was disproportionally under-represented in the survey. We would then use “raking” to create weights to make responding sites and participants representative of the total population surveyed. This technique involves selecting a set of variables where the population distribution is known, and then iteratively adjusting the weight for each respondent until the weighted respondent distribution aligns with the total population for those variables. Post-stratification weight adjustments can reduce the variance and the bias in final survey estimates.

Nonresponse analysis: We have administrative data for the universe of respondents for each survey, and, as such, we will be able to conduct nonresponse analysis using administrative data. At the conclusion of each survey, we will conduct a nonresponse analysis to assess potential differences in socio-demographic and training background characteristics between those who respond and those who do not. This nonresponse analysis may also help identify opportunities for modifications to future follow-up approaches. For example, if paraprofessionals are the largest group of nonresponders, we can revise the advance notification emails or follow-up reminders to this group to specify the importance of their participation. Additionally, we may attempt to contact some nonresponders by phone at the end of the first fielding of the surveys to understand why they didn’t respond (e.g., is a lack of time, interest, or confidentiality most concerning?). This information may also help improve the outreach communications to respondents. Specifically, the survey team would randomly select up to nine nonrespondents from each survey and contact them twice via phone. At the conclusion of all data collection, we will compile the findings and incorporate recommendations to improve future response rates.

4. Tests of Procedures or Methods to Be Undertaken

The survey instruments were developed with input from HRSA staff, reviewed by NORC survey methodologists and SUD and OUD workforce subject matter experts, and edited by copy editors. Pilot tests of the instruments were conducted with three to seven respondents for each survey. Pilot testers were asked to comment on survey length, clarity of instructions and questions, and whether response categories were comprehensive and coherent. Based on feedback from pilot testers, content was modified as needed to clarify terminology, streamline questions, and refine response options.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Data collection and analysis will be conducted by the NORC at the University of Chicago evaluation team. Contact information for NORC’s principal staff on the project is listed below. All five persons were involved in designing the data collection.

Alana Knudson

Principal Investigator

Phone: 301/634-9326

Email: [email protected]

Kathy Rowan

Project Director; Data Analysis Lead

Phone: 301/634-5426

Email: [email protected]

Britta Anderson

Project Manager

Phone: 301/634-9356

Email: [email protected]

Jennifer Satorius

Survey Lead

Phone: 217/632-7161

Email: [email protected]

Ed Mulrow

Senior Vice President, Statistics and Methodology

Phone: 301/634-9441

Email: [email protected]

9