Supporting Statement B
Evaluation of the Maternal and Child Health Bureau Pediatric Mental Health Care Access and Screening and Treatment for Maternal Depression and Related Behavioral Disorders Programs Project
OMB Control No. 0906-xxxx
1. Respondent Universe and Sampling Methods
The potential respondent universe for each data collection tool is described in Exhibit 1. Sampling or other respondent selection methods will not be used; instead, the respondent universe will comprise all participants from all 28 awardees. The reason we elected to collect data from the respondent universe rather than using sampling is that in actual practice some sampling bias occurs in almost all studies to some extent and given the variability in how states are implementing their programs, a sample may not accurately represent enrolled health care providers. Selecting the universe will provide the largest number of respondents and will be more appropriate for drawing comparisons to state and national-level data, as appropriate. Specifically, for the HCP Surveys and Practice-level Surveys obtaining responses from all participating health care providers and practice managers is preferred over sampling to allow for examination within a state and/or across PMHCA and MDRBD programs regarding (1) enrolled providers/practices for screening, referral, and care coordination efforts for behavioral health conditions; (2) provision of behavioral health services for mental health conditions in primary care settings by enrolled health care providers; (3) use of consultative services; and (4) facilitation of access to behavioral health services for mental health conditions. For the Program Implementation Surveys and SSIs, obtaining responses from all awardee project directors and principal investigators is preferred to sampling to obtain important contextual information about how the program is being implemented across all awardees and in various settings given the heterogeneity of implementation approaches. Expected response rates for the data collection activities as a whole are expected to meet or exceed 80% (see Section B.3 for additional information). Data collection tools are included in Attachments B1–B8.
Exhibit 1. Potential Respondent Universe
Form Name |
Number of Entities in the Universe |
PMHCA HCP Survey |
Based on awardee estimates of health care providers to be enrolled in the PMHCA program; 9,315 health care providers are eligible to be surveyed. |
MDRBD HCP Survey |
Based on awardee estimates of health care providers to be enrolled in the MDRBD program; 3,720 health care providers are eligible to be surveyed. |
PMHCA Practice-level Survey |
Based on awardee estimates of practices to be enrolled in the PMHCA program; 2,955 practice managers are eligible to be surveyed. |
MDRBD Practice-level Survey |
Based on awardee estimates of practices to be enrolled in the MDRBD program; 1,210 practice managers are eligible to be surveyed. |
PMHCA Program Implementation Survey |
21 individuals (1 from each of the 21 PMHCA awardees) who are in the project director/principal investigator role are eligible to be surveyed. |
MDRBD Program Implementation Survey |
7 individuals (1 from each of the 7 MDRBD awardees) who are in the project director/principal investigator role are eligible to be surveyed. |
PMHCA Program Implementation SSI |
21 individuals (1 from each of the 21 PMHCA awardees) who are in the project director/principal investigator role are eligible to be interviewed. |
MDRBD Program Implementation SSI |
7 individuals (1 from each of the 7 MDRBD awardees) who are in the project director/principal investigator role are eligible to be interviewed. |
2. Procedures for the Collection of Information
To conduct the evaluation, data will be collected using two data collection methods (i.e., surveys and SSIs). Each data collection method supports the evaluation’s goals and objectives. The data collection process will follow a systematic, mixed-methods data collection approach to gathering high-quality data from each HRSA MCHB cooperative agreement-funded program and its participating health care providers and practices. As shown in Exhibit 1, selection methods will not be used and the entire potential respondent universe will be surveyed. Surveys and SSIs will be used to collect data from the target population. A general description of key data collection procedures is provided below. Emails referenced in the sections below are included as attachments in Section A.
HCP Surveys. HCP Surveys will be administered annually to health care providers enrolled in PMHCA and MDRBD programs with anticipated data collection in 2020, 2021, and 2022. The survey will take approximately 10 minutes to complete, will be administered via a web-based platform (e.g., SurveyMonkey, Qualtrics), and will be sent directly to all health care providers via an email with a link to the online survey. At each time point, the survey link will be available for at least 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.
Approximately two weeks before distributing each survey, health care providers will be notified by email of the upcoming survey. The introductory email will include an email address that they can use to submit questions about the survey.
At the time of each survey administration, a second email will be sent to health care providers that contains the link to complete the online survey. If participants have difficulties accessing the web-based survey because of firewall issues or other technical problems, we will coordinate with them to identify an alternative method for completion (e.g., a fillable, printable PDF file with instructions for completing and returning the form). A unique identifier will be assigned to each participant regardless of administration method to match and compare the participants’ survey results over time.
Practice-level Survey. The Practice-level Surveys will be administered annually with anticipated data collection in 2020, 2021, and 2022. The survey will take approximately 15 minutes to complete. At each time point, the survey link will be available for at least 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.
Approximately two weeks before distributing each survey, practice managers will be notified by email of the upcoming survey. The introductory email will include an email address that they can use to submit questions about the survey.
At the time of each survey administration, a second email will be sent to the practice manager that contains the link to complete the online survey. If participants have difficulties accessing the web-based survey because of firewall issues or other technical problems, we will coordinate with them to identify an alternative method for completion (e.g., a fillable, printable PDF file with instructions for completing and returning the form). A unique identifier will be assigned to each participant regardless of administration method to match and compare the participants’ survey results over time.
Program Implementation Survey. The Program Implementation Survey will be administered annually in 2020, 2021, and 2022. The survey will be administered via a web-based platform (e.g., SurveyMonkey) and will take approximately 20 minutes to complete. Identified participants will be sent a link to complete it. As with the other surveys, participants can complete the survey in more than one sitting, if needed, saving information as they progress through the questions. At each time point, the survey link will be available for at least 31 days. To facilitate completion, the web-based platform will allow participants to complete the survey in more than one sitting, if needed.
Approximately two weeks before distributing each survey, an email will be sent to each cooperative agreement-funded awardee to inform it of the upcoming survey and to provide an overview of the content, procedures, and time commitment to complete the survey. An email address will be provided to awardees that they can use to submit questions about the survey and other data collection activities. Awardees can also use this email to suggest that the survey be sent to a different project representative within the state.
At the time of each survey administration, a second email will be sent to the cooperative agreement-funded awardees via the web-based platform that contains the link to complete the online survey. If awardees have difficulties accessing the web-based survey because of firewall issues or other technical problems, we will coordinate with them to identify an alternative method for completion (e.g., a fillable, printable PDF file with instructions for completing and returning the form).
Program Implementation Semi-Structured Interviews. SSIs will be conducted one time in 2022. Each PMHCA and MDRBD awardee project director will be sent an email inviting him or her to participate in the interview and describing the goals, procedures, and time commitment of the interview. The initial email will ask whether the project director or another individual should participate in the Program Implementation SSI. In a second email, each project director (or designee) will be asked to provide at least three dates and times that he or she is available during a 2-week period to facilitate scheduling the SSI. If the project director (or designee) does not respond to the initial email request, we will follow up with another email. If the project director (or designee) does not respond to the second email, JBS will inform HRSA and request permission to reach out to the awardee’s Project Officer (PO) to facilitate scheduling.
Two-person teams will conduct the SSIs. The interviews will be led by an experienced senior evaluation team member, who will describe the purpose of the interview, ask respondents to describe their position and role within the project, guide the discussion to gather pertinent information on the topics of interest, and summarize the discussion at the end of the interview. The interviewer will prepare for the interview by reviewing applicable awardee documents (e.g., cooperative agreement application, progress reports) and results from the Program Implementation Survey (described above). The interviews will provide an opportunity to elaborate on the survey responses. The interviewer will use the written SSI guide to conduct the discussion, and a note taker from the evaluation team will take detailed notes.
The interview will take approximately 60 minutes to complete. Interviewers will seek permission from participants to audio record the interview. The SSIs will be conducted and recorded (if permission is granted) via WebEx. Recorded interviews will be transcribed to facilitate qualitative content analysis. After the transcripts are completed, the audio recordings will be destroyed.
3. Methods To Maximize Response Rates and Deal With Nonresponse
The ability to gain the cooperation of potential respondents is important to the success of the evaluation of MCHB PMHCA and MDRBD programs. All potential respondents will be informed of the data collection activities before being contacted to participate, which will maximize response rates. For the Program Implementation Survey and SSIs, project directors are especially important because they are integrally involved in the program and have been informed about all data collection activities and timelines through their participation in regular meetings with HRSA and quarterly evaluation capacity-building webinars presented by JBS. Awardees are aware that they are required to participate in this evaluation as a condition of funding. For the HCP Survey and Practice-level Survey, participating providers and practices are informed of the evaluation upon enrollment in PMHCA or MDRBD programs. A discussion of methods to maximize response rates for the different data collection methods is presented below.
HCP Surveys. Following each invitation email sent to health care providers in 2020, 2021, and 2022, JBS will access the web-based survey platform and track survey activity each week to assess the number of health care providers who have initiated and completed the HCP Survey. Reminder emails will be sent to health care providers who have not completed the web-based survey. We will send email reminders at specified intervals to health care providers who have not responded; JBS estimates between four and six reminder emails depending on completions and coordination with HRSA.
Practice-level Surveys. Following each invitation email sent to practice managers in 2020, 2021, and 2022, JBS will access the web-based survey platform and track survey activity each week to assess the number of people who have initiated and completed the Practice-level Survey. Reminder emails will be sent to participants who have not completed the web-based survey. We will send email reminders at specified intervals to practice managers who have not responded; JBS estimates between four and six reminder emails depending on completions and coordination with HRSA.
Program Implementation Surveys. It is anticipated that PMHCA and MDRBD Program Implementation Surveys will be completed by 100% of awardee staff members who are asked to participate. To maximize participation rates, introductory emails to the awardees will indicate that completion is important to the HRSA MCHB evaluation. Following the invitation email to awardees, JBS will access the web-based survey platform and track survey activity each day to assess the number of awardees who have initiated and completed the survey. One, two, and three weeks after sending the initial invitation email, a reminder email will be sent to awardees who have not completed the web-based survey. JBS will coordinate with HRSA to have the POs reach out to awardees who have not responded to encourage their completion of the surveys. Pending discussion with HRSA POs, JBS may also conduct follow-up phone calls to nonrespondents.
Program Implementation SSIs. Similar to Program Implementation Surveys, it is anticipated that PMHCA and MDRBD Program Implementation SSIs will be completed by 100% of the awardee staff members who are asked to participate. JBS will employ methods to engage awardee staff to maximize participation rates and reduce the burden on participating awardee staff (e.g., engagement of HRSA POs, accommodating schedules, advance notification of timing of the survey). Planning and preparation in advance of the interviews are crucial to these protocols, which will include proper timing and scheduling of the interviews to accommodate awardee staff members. Awardee staff will be informed, in advance, of the purpose and significance of the interview.
4. Tests of Procedures or Methods To Be Undertaken
Pilot tests of the data collection tools to be used in the evaluation were conducted with representative subsamples of the target populations. All pilot tests were conducted with nine or fewer individuals. Attachments B9–16 provides a summary of pilot test feedback for each data collection tool tested and outlines the changes that were made to the data collection tools based on this feedback.
PMHCA HCP Survey. A pilot test of the PMHCA HCP survey was conducted with health care providers (eg., advanced practice nurse/nurse practitioner, pediatrician) who practice in various primary care clinical practice sites (e.g., private practice, non-academic hospital-based practice). Pilot test participants were asked to complete the survey questions and to provide feedback on the time to complete the survey and the clarity and format of the questions.
MDRBD HCP Survey. A pilot test of the MDRBD HCP survey was conducted with health care providers who practice in various primary care clinical practice sites. Pilot test participants were asked to complete the survey questions and to provide feedback on the time to complete the survey and the clarity and format of the questions.
PMHCA Practice-level Survey. Pilot test feedback on the PMHCA Practice-level Survey has not been received at the time of submission of the Draft OMB Package. Pilot test results will be included in the Final OMB Package submission.
MDRBD Practice-level Survey. Pilot test feedback on the MDRBD Practice-level Survey has not been received at the time of submission of the Draft OMB Package. Pilot test results will be included in the Final OMB Package submission.
PMHCA and MDRBD Program Implementation Surveys and SSI Guides. Pilot study participants for the Program Implementation Surveys and SSI guides and procedures consisted of professionals with extensive expertise managing and implementing state or local programs supporting maternal and pediatric behavioral health that received Federal funding. Pilot test participants were asked to review the survey and interview questions and to provide feedback on the clarity of the questions. They were also asked to comment on the appropriateness of the questions and procedures for the intended audience.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
As noted in Section 8A, consultations on the evaluation design, data collection instruments and protocols, survey and SSI questions, data management, and analysis of this evaluation have occurred throughout the planning phase of the project. These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall evaluation design and data analysis plans, to obtain advice and recommendations concerning the data collection instruments, and to structure the evaluation and instruments to minimize overall and individual response burden. Consultations have occurred with the following individuals in connection with this study (listed in alphabetical order):
John Straus, M.D. Director of Special Projects Massachusetts Behavioral Health Partnership, Co-Founder National Network of Child Psychiatry Access Programs, 617-790-4120, [email protected]. Years and areas of consultation: 2018‒present, serves as a representative of those from whom information is to be obtained.
Min Qi Wang, Ph.D. Professor, Behavioral and Community Health, University of Maryland School of Public Health, 301-405-6652, [email protected]. Years and areas of consultation: 2019–present, methodological and analytic expertise.
JBS staff designed the data collection instruments and will lead the data collection and analysis efforts. Dr. Straus and Dr. Wang provided consultation on the data collection design, and Dr. Wang will provide support on the data collection analysis.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Instructions for writing Supporting Statement B |
| Author | Jodi.Duckhorn |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |