Supporting Statement B_09092024

Supporting Statement B

Evaluation of the Maternal and Child Health Bureau Pediatric Mental Health Care Access and Screening and Treatment for Maternal Mental Health and Substance Use Disorders Programs Project

OMB Control No. 0906-XXXX

2. Collection of Information Employing Statistical Methods

This section presents information about the collection of data for the evaluation of the Maternal and Child Health Bureau (MCHB) Pediatric Mental Health Care Access (PMHCA) and Screening and Treatment for Maternal Mental Health and Substance Use Disorders (MMHSUD) programs. As noted in Supporting Statement A, the goal of this project is to provide the Health Resources and Services Administration (HRSA) with information to guide future program decisions regarding increasing health professionals’ (HPs) capacity to address patients’ behavioral health and access to behavioral health services. A comprehensive, multimethod data collection effort is proposed to better understand the implementation of the PMHCA and MMHSUD programs, as well as outcomes related to enrolled/participating HPs’ and practices’ service delivery, capacity, and engagement.

1. Respondent Universe and Sampling Methods

The exhibit below describes the potential respondent universe for each data collection tool. The respondent universe for the HP and Practice-Level Surveys will comprise identified enrolled and/or participating HPs and practice managers from all 2021, 2022, and 2023 PMHCA awardees and 2023 MMHSUD awardees. The respondent universe for the Program Implementation Survey and SSI will comprise program implementers (e.g., program directors/principal investigators) from all 2021, 2022, and 2023 PMHCA awardees and 2023 MMHSUD awardees. The reason we elected to collect data from the respondent universe rather than using sampling is that, in actual practice, some sampling bias occurs in almost all studies to some extent; given the variability in how programs are implemented in different locations, a sample may not accurately represent enrolled/participating HPs and practices or program implementers. Selecting the universe will provide the largest number of respondents and be more appropriate for drawing comparisons to regional- and national-level data.

Specifically, for the HP Surveys and Practice-Level Surveys, obtaining responses from all participating HPs and practice managers is preferred over sampling to allow for examination within and/or across PMHCA and MMHSUD programs regarding (1) screening, assessment, treatment, and referral for behavioral health conditions among enrolled/participating HPs/practices; (2) use of behavioral health consultation, care coordination, and training/education delivered by the programs; and (3) access to behavioral health services for children, adolescents, and young adults for PMHCA programs and pregnant and postpartum people for MMHSUD programs.

For the Program Implementation Surveys and SSIs, obtaining responses from all awardee project directors/principal investigators is preferred to sampling to obtain important contextual information about how the program is being implemented across all awardees and in various settings, given the heterogeneity of implementation approaches.

The respondent universe for the Behavioral Health Consultation Provider SSI, Care Coordinator SSI, and Champion SSI will comprise 1 representative per stakeholder group for each of the 67 PMHCA and MMHSUD awardees to obtain important contextual information about each program, given the heterogeneity of implementation approaches. The Community-Based and Other Resources SSI will be a case study with up to 50 community-based and other resources representatives across all 67 PMHCA and MMHSUD programs.

Section B.3 provides information regarding expected response rates for the data collection activities. Attachments B1–B16 include data collection tools.

Exhibit: Potential Respondent Universe

Form Name	Number of Entities in the Universe
HP Survey	Based on awardee estimates of HPs to be enrolled in the PMHCA and MMHSUD programs, 23,256 HPs are eligible to be surveyed.
Practice-Level Survey	Based on awardee estimates of practices to be enrolled in the PMHCA and MMHSUD programs, 6,172 practice managers are eligible to be surveyed.
Program Implementation Survey	67 individuals (1 from each of the PMHCA and MMHSUD awardees) in the project director/principal investigator role are eligible to be surveyed.
Behavioral Health Consultation Provider SSI	67 individuals (1 from each of the PMHCA and MMHSUD awardees) in the behavioral health consultation provider role are eligible to be interviewed.
Care Coordinator SSI	67 individuals (1 from each of the PMHCA and MMHSUD awardees) in the care coordinator role are eligible to be interviewed.
Champion SSI	67 individuals (1 from each of the PMHCA and MMHSUD awardees) identified by awardees as being in a program champion role are eligible to be interviewed.
Community Resources SSI	A case study will include up to 50 community-based and other resources representatives across all PMHCA and MMHSUD programs.
Program Implementation SSI	134 individuals (2 from each of the PMHCA and MMHSUD awardees) in the program implementation role are eligible to be interviewed.

2. Procedures for the Collection of Information

To conduct the evaluation, JBS International, Inc. (JBS) will collect data using two data collection methods (i.e., surveys, SSIs). Each data collection method supports the evaluation’s goals and objectives. The data collection process will follow a systematic, mixed-methods data collection approach to gathering high-quality data from each 2021, 2022, and 2023 PMHCA and 2023 MMHSUD cooperative agreement-funded program and its participating/enrolled HPs and practices; project director/principal investigator(s), behavioral health consultation provider, care coordinator, and champion. Additionally, we will conduct a case study with up to 50 community-based and other resources representatives across all PMHCA and MMHSUD programs. We will use surveys and SSIs to collect data from the target population. A general description of key data collection procedures is provided below. Attachments in Supporting Statement A include emails referenced in the sections below.

HP Surveys. The HP Survey will be administered once, with anticipated data collection in 2025. The survey will take approximately 20 minutes to complete, be administered via a web-based platform (e.g., Alchemer), and sent directly to all HPs via an email with a link to the online survey. The survey link will be available for a minimum of 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.

Approximately 2 days prior to distributing the survey, HPs will be invited to participate in the survey either by JBS or by the programs, depending on decisions made while coordinating with the awardees. The introductory email will include an email address that HPs can use to submit any questions or concerns about the survey.

At the time of survey administration, JBS or the awardees will send a second email to the HPs that contains the link to complete the survey. If HPs have difficulties accessing the web-based survey (e.g., due to firewall issues or other technical problems), we will email a fillable and printable PDF and include instructions for completing and returning it.

Practice-Level Survey. The Practice-Level Survey will be administered once, with anticipated data collection in 2025. The survey will be administered via a web-based platform (e.g., Alchemer) and take approximately 20 minutes to complete. The survey link will be available for a minimum of 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.

Approximately 2 days prior to distributing the survey, each practice’s office manager/office leadership will be invited to participate in the survey either by JBS or by the cooperative agreement-funded programs, depending on the decision made while coordinating with the awardees. The introductory email to the practice representatives will include an email address that practice representatives can use to submit any questions or concerns about the survey.

At the time of survey administration, JBS or the awardees will send a second email to the office manager/office leadership that contains the link to complete the survey. If participants have difficulties accessing the web-based survey (e.g., due to firewall issues or other technical problems), we will email a fillable and printable PDF and include instructions for completing and returning it.

Program Implementation Survey. The Program Implementation Survey will be administered once, with anticipated data collection in 2025. The survey will be administered via a web-based platform (e.g., Alchemer) and take approximately 20 minutes to complete. The survey link will be available for a minimum of 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.

Approximately 2 days prior to distributing the survey, each cooperative agreement awardee will be emailed to inform it of the upcoming survey and to provide it with an overview of the content, procedures, and time commitment to complete it. The introductory email will include an email address that programs can use to submit any questions or concerns about the survey. Awardees can also use this email to suggest that the survey be sent to a different project representative within the program, but each program should provide only one response to the survey.

At the time of each survey administration, JBS will send a second email to the cooperative agreement-funded awardees, via the web-based platform, that contains the link to complete the survey. If awardees have difficulties accessing the web-based survey (e.g., due to firewall issues or other technical problems), we will email a fillable and printable PDF and include instructions for completing and returning it.

Behavioral Health Consultation Provider SSI. The SSIs will be conducted once in 2025 for all PMHCA and MMHSUD awardees. JBS will email program staff to identify and select 1 behavioral health consultation provider to participate in the SSI. We will then email the behavioral health consultation provider to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will also ask the behavioral health consultation provider to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the behavioral health consultation provider does not respond to the second email, JBS will inform HRSA and request that Project Officers (POs) reach out to their awardees to facilitate scheduling.

Two-person teams will conduct the SSIs. An evaluation team member will lead the interview, describe its purpose, ask respondents to describe their position and role within the project, guide the discussion to gather information on the topics of interest, and summarize the discussion at the end of the interview. They will prepare for the interview by reviewing any applicable awardee documents (e.g., cooperative agreement application, progress reports) and use the written SSI guide to conduct the discussion. A notetaker from the evaluation team will take detailed notes.

The interview will take about 45 minutes to complete. Interviewers will seek permission from participants to record the interview. The SSIs will be conducted and recorded (if permission is granted) via a web-based platform (e.g., Microsoft Teams, Zoom). JBS will transcribe the recorded interviews to facilitate qualitative content analysis and make follow-up telephone calls, if necessary, to clarify information obtained during the interview. After we complete the transcripts, the recordings will be destroyed.

Care Coordinator SSI. The SSIs will be conducted once in 2025 for all PMHCA and MMHSUD awardees. JBS will email program staff to identify and select one care coordinator (or care coordinator designee) to participate in the SSI. We will then email the care coordinator to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will also ask the care coordinator to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the care coordinator does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling.

Two-person teams will conduct the SSIs. An evaluation team member will lead the interview, describe its purpose, ask respondents to describe their position and role within the project, guide the discussion to gather information on the topics of interest, and summarize the discussion at the end of the interview. They will prepare for the interview by reviewing any applicable awardee documents (e.g., cooperative agreement application, progress reports) and use the written SSI guide to conduct the discussion. A notetaker from the evaluation team will take detailed notes.

The interview will take about 45 minutes to complete. Interviewers will seek permission from participants to record the interview. The SSIs will be conducted and recorded (if permission is granted) via a web-based platform (e.g., Microsoft Teams, Zoom). JBS will transcribe the recorded interviews to facilitate qualitative content analysis and make follow-up telephone calls, if necessary, to clarify information obtained during the interview. After we complete the transcripts, the recordings will be destroyed.

Champion SSI. SSIs will be conducted once in 2025 for all PMHCA and MMHSUD awardees. JBS will email program staff to identify and select one champion to participate in the SSI. We will then email the selected champion to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will also ask the champion to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the champion does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling.

Two-person teams will conduct the SSIs. An evaluation team member will lead the interview, describe its purpose, ask respondents to describe their position and role within the project, guide the discussion to gather information on the topics of interest, and summarize the discussion at the end of the interview. They will prepare for the interview by reviewing any applicable awardee documents (e.g., cooperative agreement application, progress reports) and use the written SSI guide to conduct the discussion. A notetaker from the evaluation team will take detailed notes.

The interview will take about 30 minutes to complete. Interviewers will seek permission from participants to record the interview. The SSIs will be conducted and recorded (if permission is granted) via a web-based platform (e.g., Microsoft Teams, Zoom). JBS will transcribe the recorded interviews to facilitate qualitative content analysis and make follow-up telephone calls, if necessary, to clarify information obtained during the interview. After we complete the transcripts, the recordings will be destroyed.

Community-Based and Other Resources SSI. SSIs will be conducted once in 2025. JBS will email program staff to identify program-level community-based and other resources representatives to participate in the SSI. After all programs, as applicable, have identified community-based and other resource representatives, JBS will create a crosswalk to map the identified representatives to characteristics of interest (e.g., type of service provided by/focus of the organization), which will inform sampling and scheduling. We will then email the selected community-based and other resources representatives to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will also ask the representatives to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the community-based or other resources representative does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. If we still do not hear back after the third attempt, we will reach out to the next selected representative to share participation details and to facilitate scheduling.

Two-person teams will conduct the SSIs. An evaluation team member will lead the interview, describe its purpose, ask respondents to describe their position and role within the project, guide the discussion to gather information on the topics of interest, and summarize the discussion at the end of the interview. They will prepare for the interview by reviewing any applicable awardee documents (e.g., cooperative agreement application, progress reports) and use the written SSI guide to conduct the discussion. A notetaker from the evaluation team will take detailed notes.

The interview will take about 30 minutes to complete. Interviewers will seek permission from participants to record the interview. The SSIs will be conducted and recorded (if permission is granted) via a web-based platform (e.g., Microsoft Teams, Zoom). JBS will transcribe the recorded interviews to facilitate qualitative content analysis and make follow-up telephone calls, if necessary, to clarify information obtained during the interview. After we complete the transcripts, the recordings will be destroyed.

Program Implementation SSI. SSIs will be conducted one time in 2025. JBS will email project leadership to invite them to participate in the interview and will provide an overview of the SSI goals, procedures, and time commitment. In the initial email, we will ask whether the project director or another individual should participate in the Program Implementation SSI and for the project director (or designee) to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the project director (or designee) does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling.

Two-person teams will conduct the SSIs. An evaluation team member will lead the interview, describe its purpose, ask respondents to describe their position and role within the project, guide the discussion to gather information on the topics of interest, and summarize the discussion at the end of the interview. They will prepare for the interview by reviewing any applicable awardee documents (e.g., cooperative agreement application, progress reports) and use the written SSI guide to conduct the discussion. A notetaker from the evaluation team will take detailed notes.

The interview will take about 60 minutes to complete. Interviewers will seek permission from participants to record the interview. The SSIs will be conducted and recorded (if permission is granted) via a web-based platform (e.g., Microsoft Teams, Zoom). We will transcribe the recorded interviews to facilitate qualitative content analysis and make follow-up telephone calls, if necessary, to clarify information obtained during the interview. After we complete the transcripts, the recordings will be destroyed.

For more detail on the analysis plan, please see Supporting Statement A, section 16.

3. Methods to Maximize Response Rates and Deal With Nonresponse

The ability to gain the cooperation of potential respondents is important to the success of the evaluation of the PMHCA and MMHSUD programs. We will inform all potential respondents of the data collection activities before contacting them to participate, which will maximize response rates. For the Program Implementation Survey and SSIs, participation from project leadership is especially important because it is integrally involved in the program and has been informed about all data collection activities and timelines through its participation in regular meetings with HRSA and regular evaluation capacity-building webinars presented by JBS.

Awardees are aware that they are required to participate in this evaluation as a condition of funding. For the HP Survey and Practice-Level Survey, participating providers and practices are informed of the evaluation upon participation/enrollment in PMHCA and MMHSUD programs. For the Champion SSI and the Community-Based and Other Resources SSI, awardees will identify and select participants who are eligible and interested in participating in data collection. For the Behavioral Health Consultation Provider SSI and Care Coordinator SSI, programs will inform their designated behavioral health consultation provider and care coordinator that they will be asked to participate in the SSI. JBS will provide awardees with materials (e.g., graphics, PowerPoint slides, newsletter blurbs) to promote engagement with their programs and with feedback on the evaluation data collection. A discussion of methods to maximize response rates for the different data collection methods is presented below.

HP Survey. We anticipate that the survey will be completed by 20–30 percent of HPs asked to participate; this estimate is based on previous experience from the initial evaluation. In the initial evaluation, approved under 0906-0052 and 0906-0074, we administered the Health Care Professional/HP Survey yearly from 2020 to 2023 and had response rates ranging from 18.6% to 24.3%, with an average of 20.5 percent. To address response rates, following the invitation email to HPs, we will access the web-based system and track surveys each week to assess the number of HPs who have initiated and completed the HP survey. After sending the initial invitation email, JBS or the awardees will send a reminder email to HPs who have not completed the web-based survey and continue to send them at specified intervals to those who have not responded; JBS estimates between six and eight reminder emails, depending on completions and on coordination with HRSA. If an awardee has decided to communicate directly with their enrolled/participating HPs during the survey administration process, JBS will provide the cooperative agreement-funded program with draft email content for all necessary communications discussed above. We will update HRSA on the response rates during the standing, biweekly contract calls.

To assess the presence of non-response bias, we will first conduct a demographic comparison that considers such factors as provider specialty and the PHMCA program with which the provider is registered. Chi-square or t-tests will be performed to compare the groups. In addition, we will test for nonresponse bias by conducting a wave analysis that compares the characteristics of those who responded to the early email requests to complete the survey with those who completed the survey only after later email requests. Appropriate parametric and nonparametric statistics will be calculated. If significant nonresponse bias is detected, a sensitivity analysis will be performed to model the potential impact of nonresponse were all nonrespondents to answer with the lowest, average, or highest value for each survey question.

Practice-Level Survey. We anticipate that the survey will be completed by 20–30 percent of practices asked to participate; this estimate is based on previous experience from the initial evaluation. In the initial evaluation, approved under 0906-0052 and 0906-0074, the Practice-Level Survey was administered yearly from 2020 to 2023 and had response rates ranging from 14.2 percent to 33.8 percent, with an average of 22.4 percent. To address response rates, following the invitation email to office manager/office leadership, JBS will access the web-based system and track surveys on a weekly basis to assess the number of people who have initiated and completed the Practice-Level Survey. After sending the initial invitation email, JBS or the awardees will send a reminder email to participants who have not completed the web-based survey and continue to send them at specified time points to the office manager/office leadership representatives who have not responded; JBS estimates between six and eight reminder emails, depending on completions and on coordination with HRSA. If an awardee has decided to communicate with their enrolled/participating practices during the survey administration process, JBS will provide the cooperative agreement-funded program with draft email content for all necessary communications discussed above. We will update HRSA on the survey response rates during the standing, biweekly contract calls.

To assess the presence of non-response bias, we will first conduct a demographic comparison that considers such factors as provider specialty and the PHMCA program with which the provider is registered. Chi-square or t-tests will be performed to compare the groups. In addition, we will test for nonresponse bias by conducting a wave analysis that compares the characteristics of those who responded to the early email requests to complete the survey with those who completed the survey only after later email requests. Appropriate parametric and nonparametric statistics will be calculated. If significant nonresponse bias is detected, a sensitivity analysis will be performed to model the potential impact of nonresponse were all nonrespondents to answer with the lowest, average, or highest value for each survey question.

Program Implementation Survey. We anticipate that Program Implementation Survey will be completed by 100 percent of awardee staff members asked to participate, based on previous experience from the initial evaluation. To maximize participation rates, introductory emails to the awardees will indicate that completion is important to the HRSA MCHB evaluation. Following the invitation email to awardees for each time point, we will access the web-based system and track surveys each day to assess the number of awardees who have initiated and completed the survey. After sending the initial invitation email, JBS will send a reminder email to awardees who have not completed the web-based survey and continue to send them at specified time points to awardees who have not responded until we receive a response, or they refuse to complete the survey (which we do not anticipate). JBS will update HRSA on the response rates during the standing, biweekly contract calls and determine if HRSA would like for the POs to reach out to awardees to encourage their completion of the surveys.

To assess the presence of non-response bias, we will first conduct a demographic comparison that considers such factors as provider specialty and the PHMCA program with which the provider is registered. Chi-square or t-tests will be performed to compare the groups. In addition, we will test for nonresponse bias by conducting a wave analysis that compares the characteristics of those who responded to the early email requests to complete the survey with those who completed the survey only after later email requests. Appropriate parametric and nonparametric statistics will be calculated. If significant nonresponse bias is detected, a sensitivity analysis will be performed to model the potential impact of nonresponse were all nonrespondents to answer with the lowest, average, or highest value for each survey question.

Behavioral Health Consultation Provider SSI. We anticipate that PMHCA and MMHSUD Behavioral Health Consultation Provider SSIs will be completed by 100 percent of the awardee behavioral health consultation providers asked to participate. JBS will email programs to identify and select one behavioral health consultation provider to participate in the Behavioral Health Consultation SSI. Once the program has identified and selected the behavioral health consultation provider, JBS will email the selected provider to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will then ask them to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the behavioral health consultation provider does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. JBS will employ methods to reduce the burden on participating behavioral health consultation providers (e.g., accommodating schedules, providing advance notification of timing of the SSI). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate awardee behavioral health consultation providers.

To minimize non-response bias, we may request awardees support additional communication efforts to reach their identified behavioral health consultation provider. To avoid non-response bias based on technology, JBS will offer a call-in phone option for participants who cannot use the web-based platform and will provide technology support during the interview.

During qualitative analysis, JBS will also use multiple approaches to minimize non-response bias in the form of omitted data. JBS will establish qualitative analysis protocols for use by the analysis team that direct exploring the meaning of outlier data or extreme cases in addition to themes of most common responses to help avoid omission of important data. We will also employ triangulation of qualitative data and quantitative data to develop a complete understanding of concepts of interest.

Care Coordinator SSI. We anticipate that PMHCA and MMHSUD Care Coordinator SSIs will be completed by 100 percent of the awardee care coordinators asked to participate. JBS will email the programs to identify and select one care coordinator (or designee) to participate in the Care Coordinator SSI. Once the program has identified and selected the care coordinator, JBS will email the selected care coordinator to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will then ask the care coordinator to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the care coordinator does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. JBS will employ methods to reduce the burden on participating care coordinators (e.g., accommodating schedules, providing advance notification of timing of the SSI). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate awardee care coordinators.

To minimize non-response bias, we may request awardees support additional communication efforts to reach their identified care coordinator (or designee). To avoid non-response bias based on technology, JBS will offer a call-in phone option for participants who cannot use the web-based platform and will provide technology support during the interview.

During qualitative analysis, JBS will also use multiple approaches to minimize non-response bias in the form of omitted data. JBS will establish qualitative analysis protocols for use by the analysis team that direct exploring the meaning of outlier data or extreme cases in addition to themes of most common responses to help avoid omission of important data. We will also employ triangulation of qualitative data and quantitative data to develop a complete understanding of concepts of interest.

Champion SSI. We anticipate that PMHCA and MMHSUD Champion SSIs will be completed by 100 percent of the awardee champions who are asked to participate. Once programs have identified and selected a program champion, JBS will email the selected champion to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will then ask the champion to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the champion does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. JBS will employ methods to reduce the burden on participating champions (e.g., accommodating schedules, providing advance notification of timing of the SSI). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate awardee champions.

To minimize non-response bias, we may request awardees support additional communication efforts to reach their identified champion. To avoid non-response bias based on technology, JBS will offer a call-in phone option for participants who cannot use the web-based platform and will provide technology support during the interview.

During qualitative analysis, JBS will also use multiple approaches to minimize non-response bias in the form of omitted data. JBS will establish qualitative analysis protocols for use by the analysis team that direct exploring the meaning of outlier data or extreme cases in addition to themes of most common responses to help avoid omission of important data. We will also employ triangulation of qualitative data and quantitative data to develop a complete understanding of concepts of interest.

Community-Based and Other Resources SSI. We anticipate that Community-Based and Other Resources SSIs will be completed by 100 percent of participants (i.e., we will conduct 50 interviews). After all programs, as applicable, have identified community-based and other resource representatives, JBS will create a crosswalk to map the identified representatives to characteristics of interest (e.g., type of service provided by/focus of the organization), which will inform sampling and scheduling. JBS will then email the selected community-based and other resources representatives to ask them to participate in the interview and provide an overview of the SSI goals, procedures, and time commitment. We will also ask the representatives to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the community-based and other resources representative does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. If we do not hear back after the third attempt, we will reach out to the next selected representative to share participation details and to facilitate scheduling. JBS will employ methods to reduce the burden on participating community-based and other resources representatives (e.g., accommodating schedules, providing advance notification of timing of the SSI). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate community resource representatives.

We will use multiple approaches to minimize non-response bias with the Community-Based and Other Resources SSIs during recruitment and data collection. Throughout participant recruitment and on a rolling basis, we will review the number of potential participants in each of the categories of interest (sampling groups) to ensure a diverse pool. If the number of potential participants in a category is skewed low or high in comparison to other categories, we may request additional recruitment efforts from awardees. To avoid non-response bias based on technology, we will offer a call-in phone option for participants who cannot use the web-based platform and will provide technology support during the interview.

During qualitative analysis, we will also use multiple approaches to minimize non-response bias in the form of omitted data. We will establish qualitative analysis protocols for use by the analysis team that direct exploring the meaning of outlier data or extreme cases in addition to themes of most common responses to help avoid omission of important data. We will also employ triangulation of qualitative data and quantitative data to develop a complete understanding of concepts of interest.

Program Implementation SSI. We anticipate that Program Implementation SSIs will be completed by 100 percent of the awardee staff members asked to participate. JBS will email project leadership to invite it to participate in the interview and will provide an overview of the SSI goals, procedures, and time commitment. In the initial email, we will ask whether the project director or another individual should participate in the Program Implementation SSI and for the project director (or designee) to select an interview timeslot. If we do not hear back on the initial email request, we will follow up by email. If the project director (or designee) does not respond to the second email, JBS will inform HRSA and request that POs reach out to their awardees to facilitate scheduling. We will employ methods to engage awardee staff to maximize participation rates and to reduce the burden on participating awardee staff (e.g., engaging HRSA POs, accommodating schedules, providing advance notification of timing of the SSI). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate awardee staff members. We will inform awardee staff, in advance, of the purpose and significance of the interview.

To minimize non-response bias, we may request POs support additional communication efforts to reach the awardees. To avoid non-response bias based on technology, JBS will offer a call-in phone option for participants who cannot use the web-based platform and will provide technology support during the interview.

During qualitative analysis, JBS will also use multiple approaches to minimize non-response bias in the form of omitted data. JBS will establish qualitative analysis protocols for use by the analysis team that direct exploring the meaning of outlier data or extreme cases in addition to themes of most common responses to help avoid omission of important data. We will also employ triangulation of qualitative data and quantitative data to develop a complete understanding of concepts of interest.

4. Tests of Procedures or Methods to Be Undertaken

For the previous OMB submission for the evaluation (OMB Control No. 0906-0074; expiration 12-31-25), JBS conducted pilot tests of all the data collection tools, except for the Behavioral Health Consultation Provider SSI, to be used in the evaluation with:

• Representative subsamples of the target populations

• Representatives with experience in health, public health, and/or behavioral health research and evaluation, including study design; instrument development; and data collection, analysis, and reporting

• Medical advisors (e.g., physicians specializing in family medicine, preventive medicine, or addiction medicine) with current or previous practice in clinical practice sites (e.g., private practice, nonacademic hospital-based practice)

• Individuals with positions in academic settings and the Federal Government

We previously administered portions of these instruments to 2021 and 2022 PMHCA awardees.

Prior to this OMB submission, we piloted the Behavioral Health Consultation Provider SSI with:

• Representatives with experience in health, public health, and/or behavioral health research and evaluation, including study design; instrument development; and data collection, analysis, and reporting

• Medical advisors (e.g., physicians specializing in family medicine, preventive medicine, and addiction medicine) with current or previous practice in clinical practice sites (e.g., private practice, nonacademic hospital-based practice)

• HRSA MCHB technical assistance staff who work directly with the cooperative agreement-funded programs

Attachment B17 provides a summary of pilot test feedback for the Behavioral Health Consultation Provider SSI instrument and outlines the changes made to the data collection tools based on this feedback. We asked pilot test participants to review the questions and to provide feedback on the clarity, appropriateness of the questions and procedures for the intended audience, and estimated amount of time to complete the SSI.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

As Section A.8 notes, consultations on the evaluation design, data collection instruments (i.e., HP Survey, Practice-Level Survey, Program Implementation Survey, Program Implementation SSI) and protocols, survey and SSI questions, data management, and analysis of the initial evaluation of programs funded by HRSA in 2018 and 2019 (September 2018–September 2021) occurred throughout the planning phase of the initial project. The current evaluation (September 2021–September 2026) is a continuation of the initial evaluation to (1) incorporate the 2023 PMHCA and MMHSUD awardees in the evaluation and (2) integrate the 2021 and 2022 PMHCA awardees into the revised data collection instruments presented in this package. We have refined the current evaluation design and data collection instruments based on information learned in the previous evaluation and to account for differences in the new awardee program requirements. These consultations have provided, and will continue to provide, the opportunity to:

• Ensure the technical quality and appropriateness of the overall evaluation design and data analysis plans

• Obtain advice and recommendations concerning the data collection instruments

• Structure the evaluation and instruments to minimize overall and individual response burden

Consultations have occurred with the following individuals in connection with this study:

• Medical Director of Behavioral Health, Chicago Department of Public Health. Years and areas of consultation: 2022, clinical expertise

• Director of Special Projects Massachusetts Behavioral Health Partnership, Co-Founder National Network of Child Psychiatry Access Programs. Years and areas of consultation: 2018‒present; representative of those from whom information is to be obtained

• Professor, Behavioral and Community Health, University of Maryland School of Public Health. Years and areas of consultation: 2019–2021, methodological and analytic expertise

JBS staff designed the data collection instruments and will lead the data collection and analysis efforts. Consultants provided feedback on the data collection design and support on the data analysis plan.

5