Supporting Statement A_final

Supporting Statement A

Evaluation of the Maternal and Child Health Bureau Pediatric Mental Health Care Access and Screening and Treatment for Maternal Depression and Related Behavioral Disorders Programs Project

OMB Control No. 0906-xxxx

Terms of Clearance: None

A. Justification

1. Circumstances Making the Collection of Information Necessary

In compliance with Section 3506(c)(2)(A) of the Paperwork Reduction Act of 1995, this submission requests Office of Management and Budget (OMB) approval of a 4-year clearance for the Health Resources and Services Administration (HRSA) to conduct an evaluation of the Maternal and Child Health Bureau (MCHB) Pediatric Mental Health Care Access (PMHCA) and Screening and Treatment for Maternal Depression and Related Behavioral Disorders (MDRBD) programs. This project will collect data to provide HRSA with information to guide future program and policy decisions regarding increasing health care providers (HCPs)/health professionals’ (HPs) (e.g., pediatricians, family physicians, physician assistants, advanced practice nurse/nurse practitioners, licensed practical nurses, registered nurses, counselors, social workers, medical assistants, patient care navigators) capacity to address patients’ behavioral health and access to behavioral health services.

Title X of the 21st Century Cures Act (Cures Act) strengthens mental health and substance use disorder care for women, children, and adolescents. The PMHCA and MDRBD Notice of Funding Opportunity announcements were based on the Cures Act and articulate the requirements for each cooperative agreement-funded program. Section 10002 of the Cures Act supports increased access to pediatric mental health care. It authorizes HRSA to provide funding to promote behavioral health integration in pediatric primary care by supporting the development and improvement of existing statewide or regional pediatric mental health care telehealth access programs. Section 10005 supports screening, assessment, and treatment of maternal depression. It provides funds for states to establish, improve, or maintain existing programs that screen, assess, and treat women who are pregnant or gave birth in the preceding 12 months, for maternal depression. The services must include culturally and linguistically appropriate components.

Section 2712 of the American Rescue Plan Act, as specified below, allowed for additional funding for PMHCA programs. In addition to amounts otherwise available, Section 2712 appropriates to the Secretary of Health and Human Services for fiscal year (FY) 2021, out of any money in the Treasury not otherwise appropriated, $80,000,000 to remain available, until expended, for carrying out Section 330M of the Public Health Service Act (42 U.S.C. 254c–19).

The PMHCA and MDRBD programs aim to increase identification of behavioral health conditions by screening specified populations (e.g., children, adolescents, young adults, pregnant and postpartum women), especially those in rural, isolated, and/or underserved areas; providing clinical behavioral health consultation, care coordination support (i.e., communication/collaboration, accessing resources, referral services), and training to HCPs/HPs; and increasing access to clinical interventions, including by telehealth. Provider/HP education and training will support the knowledge and skills acquisition needed to accomplish this goal. This evaluation will allow HRSA to determine the extent to which the PMHCA and MDRBD programs have met these objectives. The evaluation will be implemented by JBS International, Inc. (JBS), as part of a contract that is funded by HRSA (Contract No. 75R60219D00046).

2. Purpose and Use of Information Collection

As stated in Section A.1, the goal of this project is to provide HRSA with information to guide future program and policy decisions regarding increasing HCPs/HPs’ capacity to address patients’ behavioral health and access to behavioral health services.

The evaluation uses a mixed-methods design with data collection activities across all HRSA MCHB awardees. Methodologies for this study include surveys (e.g., online, mailed), semi-structured interviews (SSIs), and focus group discussions (FGDs).

As indicated in Exhibits 1 and 2 below, the project will collect data from those enrolled/participating in the 2021 PMHCA program (i.e., enrolled/participating HPs and practices) and project leadership implementing the programs (e.g., program-level project directors and principal investigators), as well as data from program champions, informal and formal community resources partner representatives, and program-level care coordinators participating in the 2018/2019/2021 PMHCA and 2018 MDRBD programs. For the 2021 PMHCA programs, HPs, practices, and project leadership will complete surveys annually in 2023, 2024, and 2025. SSIs with project leadership, SSIs with informal and formal community resources partner representatives, and SSIs/FGDs with program champions will be conducted once in 2025. SSIs with program-level care coordinators will be conducted twice, once in 2023 and once in 2025. For 2018/2019 PMHCA and 2018 MDRBD awardees, the SSIs/FGDs with champions and the SSIs with informal and formal community resources partner representatives and care coordinators will be conducted once in 2023. Additional data collection for the 2018/2019 PMHCA and 2018 MDRBD programs (i.e., HCP Survey, Practice-Level Survey, Program Implementation Survey, Program Implementation SSI) has previously been approved (OMB Control No. 0906-0052).

Exhibit 1. 2021 PMHCA Data Collection Activities

Tool	2022	2023	2024	2025	2026
HP Survey	N/A	Spring 2023	Spring 2024	Spring 2025	N/A
Practice-Level Survey	N/A	Spring 2023	Spring 2024	Spring 2025	N/A
Program Implementation Survey	N/A	Spring 2023	Spring 2024	Spring 2025	N/A
Program Implementation SSI	N/A	N/A	N/A	Summer 2025	N/A
Champion SSI or Champion FGD	N/A	N/A	N/A	Fall 2025	N/A
Community Resources SSI	N/A	N/A	N/A	Fall 2025	N/A
Care Coordinator SSI	N/A	Spring 2023	N/A	Spring 2025	N/A

Exhibit 2. 2018/2019 PMHCA and 2018 MDRBD Data Collection Activities

Tool	2022	2023
Champion SSI or Champion FGD	N/A	Spring 2023
Community Resources SSI	N/A	Spring 2023
Care Coordinator SSI	N/A	Spring 2023

Specifically, HRSA is requesting approval for the following:

HP Survey (2021 PMHCA only) – survey of enrolled/participating 2021 PMHCA program HPs, examining their experiences with the programs (e.g., HP training, how the programs meet their consultation and care coordination support needs, access to consultations and referrals, capacity to address behavioral health).

Practice-Level Survey (2021 PMHCA only) – survey of enrolled/participating 2021 PMHCA program office managers/office leadership about how the program is being implemented within their practice (e.g., enrolled/participating practices’ behavioral health screening, consultation, treatment, and referral practices; community linkages; businesses processes; financial sustainability).

Program Implementation Survey (2021 PMHCA only) – survey of 2021 PMHCA program project directors/principal investigators examining HP/practice recruitment and enrollment, HP training, clinical behavioral health consultation, care coordination support, community linkages, program outreach and dissemination, and sustainability.

Program Implementation SSIs (2021 PMHCA only) – interview with 2021 PMHCA program project directors/principal investigators examining their programs’ implementation to complement and expand on data collected in the program implementation surveys.

Champion SSIs or FGDs (All awardees) – interviews/discussions with program champions regarding program implementation and involvement (e.g., program involvement, program implementation, HCP/HP communications, program outcomes, sustainability, health equity, advocacy activities) to complement and expand on data collected in other data collection activities (e.g., HCP/HP Survey).

Community Resources SSI (All awardees) – interview with program-level informal and formal community resources partner representatives, examining their collaboration with their state, political subdivision of a state, Indian tribe, or tribal organization’s awardee program (e.g., collaboration/involvement with the awardee program; impacts of collaboration, health equity, and sustainability). The SSIs will be a case study with (1) up to 5 programs that have identified informal partnerships and (2) up to 5 programs that have identified formal partnerships with community resources. Selected programs will select up to 5 informal and up to 5 formal partner representatives to participate in the SSI.

Care Coordinator SSIs (All awardees) – interview with program-level care coordinators about their program’s care coordination (e.g., communication/collaboration, accessing resources, referral services) processes and systems to complement and expand on data collected in other data collection activities (e.g., Program Implementation Survey/SSI).

These data will assist in understanding the implementation and outcomes of the PMHCA and MDRBD programs, which were initially funded in 2018, 2019, and 2021 in areas that have not yet been evaluated. Specifically, the collected data will be used to:

• Study the efforts of awardee programs to achieve key outcomes

• Measure whether and to what extent awardee programs are associated with changes in these key outcomes

• Examine changes over time; within a state, political subdivision of a state, Indian tribe, or tribal organization; and/or across PMHCA and MDRBD programs, regarding (1) enrolled/participating HCPs/HPs/practices related to screening, referral, and care coordination for behavioral health conditions; (2) provision of behavioral health services for mental illness in primary care settings; (3) use of consultative services; and (4) provision of access to behavioral health services for mental illness

• Provide the data in reports to HRSA MCHB

• Develop resources for dissemination by HRSA MCHB

Supporting Statement B contains additional information on study procedures on the collection of information using these data collection tools. The data collection tools are also included as attachments in Supporting Statement B.

3. Use of Improved Information Technology and Burden Reduction

The evaluation of the MCHB PMHCA and MDRBD programs will follow a multimethod approach. Data collection methodologies for this evaluation will use surveys (i.e., web-based, email), as well as SSIs and FGDs (i.e., Microsoft Teams, Zoom). All technology used for the survey administration (i.e., web-linked survey administered via email and via survey platform) will meet Federal requirements for Section 508 accessibility. Information technology will be used in the following ways:

• All survey participants will receive the web-linked survey via email. Electronic responses will be downloaded directly into a securely stored database.

• All SSIs and FGDs will be conducted via a web-based platform (e.g., Microsoft Teams, Zoom). For respondents who agree to be recorded, interviewers will record responses as they are given and will upload the recordings to a secured drive. For respondents who do not agree to be recorded, a notetaker will record responses and upload call notes to a secured drive.

• Reports and materials (e.g., resources) generated from this project may be made available to the public through email or the HRSA MCHB website.

The data collection methods were selected for the evaluation because they will reduce participant burden while providing the evaluation with necessary data. Offering a web-based survey reduces burden to participants by eliminating the time it takes to write responses on a paper-based, mail-in survey. In addition, having participants respond to an online survey eliminates the time needed to mail back a paper-based survey. The burden is reduced for respondents participating in interviews via a web-based platform (e.g., Microsoft Teams, Zoom) because they will not have to write down responses to the questionnaires.

Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost or of data being lost in transit or delivered to an incorrect location. However, because not all respondents may prefer to complete a web-based survey, and to maximize completion rates, we may use alternative forms of administration (e.g., providing a printable PDF to participants). In this case, the printable PDF surveys can be returned either as attachments through encrypted emails or via mail, depending on the respondent’s preference. All hard copies will be submitted to JBS with unique alphanumeric identifiers, and the data will be entered into the online system at JBS. Hard copies will be stored in a locked file cabinet, with no name or identifying information attached.

4. Efforts to Identify Duplication and Use of Similar Information

Prior to an initial evaluation of the programs funded by HRSA in 2018 and 2019 (September 2018 – September 2021), of which the current evaluation (September 2021 – September 2026) is a continuation to incorporate additional awardees, there were no prior studies or evaluations of HRSA MCHB PMHCA and MDRBD programs among any study population. The lack of evaluative studies is due to the Cures Act legislation authorized these programs in 2016, and the first cooperative agreement programs were funded starting in fiscal year 2018. Because evaluation of the HRSA MCHB PMHCA and MDRBD cooperative agreement-funded programs is ongoing, there is no similar or existing information available on these programs; hence, there is no duplication of efforts. No other bureaus or agencies are currently evaluating the programs.

In addition, there is no duplication of information within this evaluation because the data collection surveys and SSI/FGD guides were developed while taking into account the data awardees are required to report to HRSA based on their cooperative agreements. All potential data items were mapped to the evaluation questions to ensure no duplication of information and to reduce participant burden.

5. Impact on Small Businesses or Other Small Entities

Physicians, as part of participating HPs (see Section A.12 for additional information), and participating practices are included in the data collection efforts (e.g., surveys) for this evaluation. Although a portion of physicians may be employed by large hospitals or health systems, none of which are considered small businesses, some may be in a private practice or practice in small groups of physicians. Similarly, some participating practices may be part of large systems and, therefore, are not considered small businesses, whereas others may be private practices. Information collection for this evaluation is not anticipated to have a significant impact on physicians or their practices.

The information to be obtained from physicians and participating practices is the minimum required for the intended use of the data and to achieve the objectives of the evaluation; however, completion of survey instruments will likely induce minimum burden. To reduce this burden, the surveys have been developed to be as short as possible, while still collecting necessary data, and attempts have been made to move respondents quickly through questions. For example, skip patterns have been added to the surveys so respondents do not need to answer questions that may not be relevant to them.

6. Consequences of Collecting the Information Less Frequently

As noted above in Section A.2, the collection of these data is critical to assessing the efforts of awardee programs to achieve key outcomes; measuring whether and to what extent awardee programs are associated with changes in these key outcomes; and examining changes over time; within a state, political subdivision of a state, Indian tribe, or tribal organization; and/or across PMHCA and MDRBD programs, with regard to evaluation variables of interest. The frequency of data collection as specified below is held to the minimum necessary to meet the needs of the evaluation goals and objectives.

HP Surveys (2021 PMHCA only). HP Surveys will be administered annually to HPs enrolled in 2021 PMHCA programs, with anticipated data collection in 2023, 2024, and 2025. Annual HP Survey administration will allow for data collection from all potentially enrolled/participating HPs because they may enroll/participate in PMHCA programs on a rolling basis. Annual HP Survey administration will also allow for examination of changes over time in behavioral health capacity and in screening, assessment, and treatment of behavioral health conditions for HPs who complete the survey more than one time.

Practice-Level Surveys (2021 PMHCA only). Practice-Level Surveys will be administered annually to practice managers for practices enrolled/participating in 2021 PMHCA programs, with anticipated data collection in 2023, 2024, and 2025. Annual Practice-Level Survey administration will allow for data collection from all potentially enrolled/participating practices because practices may enroll in PMHCA programs on a rolling basis. Annual Practice-Level Survey administration will also allow for examination of changes over time in behavioral health services, practice behavioral health capacity, community linkages, practice operations, and staff training for practices that complete the survey more than one time.

Program Implementation Surveys (2021 PMHCA only). Program Implementation Surveys will be administered annually to project directors/principal investigators from each of the 24 2021 PMHCA awardees, with anticipated data collection in 2023, 2024, and 2025. Program Implementation Survey data will be collected annually to allow for examination of changes in HP/practice recruitment and enrollment, HP training, clinical behavioral health consultation, care coordination support, community linkages, program outreach and dissemination, and sustainability.

Program Implementation SSIs (2021 PMHCA only). SSIs will be administered one time via a web-based platform (e.g., Microsoft Teams, Zoom) to project directors/principal investigators from each of the 24 2021 PMHCA awardees, with anticipated data collection in 2025. Topics will be similar to the Program Implementation Surveys but will provide project directors/principal investigators an opportunity to discuss program implementation toward the end of the project period.

Champion SSIs and FGDs (All awardees). SSIs and FGDs will be administered once via a web-based platform (e.g., Microsoft Teams, Zoom) to up to 3 program champions from each of the 52 PMHCA and MDRBD awardees, with anticipated data collection in 2023 for 2018/2019 PMHCA and 2018 MDRBD awardees and in 2025 for 2021 PMHCA awardees. The Champion SSIs and FGDs collect in-depth, contextual, qualitative information from champions regarding program implementation to complement and expand on data collected in other data collection activities.

Community Resources SSI (All awardees). Community Resources SSIs will be administered once via a web-based platform (e.g., Microsoft Teams, Zoom) to informal and formal community resources partner representatives partnering with PMHCA and MDRBD programs, with anticipated data collection in 2023 for 2018/2019 PMHCA and 2018 MDRBD programs and in 2025 for 2021 PMHCA programs. The SSIs will be a case study with (1) up to 5 cooperative agreement-funded programs that have identified informal partnerships and (2) up to 5 cooperative agreement-funded programs that have identified formal partnerships with community resources. JBS will work with designated staff from the cooperative agreement program to identify and select up to 5 informal and up to 5 formal partner representatives who should participate in the SSI.

Care Coordinator SSI (All awardees). Care Coordinator SSIs will be administered via a web-based platform (e.g., Microsoft Teams, Zoom) to program-level care coordinators from each of the 52 PMHCA and MDRBD awardees, with anticipated data collection (1) once in 2023 for 2018/2019 PMHCA and 2018 MDRBD awardees and (2) twice: once in 2023 and once in 2025 for 2021 PMHCA awardees. The SSIs will address community linkages implementation and communication, health equity, and sustainability.

There are no legal obstacles to reduce the burden.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The request fully complies with the regulation.

8. Comments in Response to the Federal Register Notice/Outside Consultation

Section 8A

A 60-day Federal Register Notice was published in the Federal Register on July 5, 2022, Vol. 87, No. 127, pp. 39841-42 (see Attachment A1). There were public comments. Public comments and responses to comments are provided as an attachment to this supporting statement (see Attachment A75).

Section 8B

Consultations on the evaluation design, data collection instruments (i.e., HCP Survey, Practice-Level Survey, Program Implementation Survey, and Program Implementation SSI) and protocols, survey and SSI questions, data management, and analysis of the initial evaluation of programs funded by HRSA in 2018 and 2019 (September 2018 – September 2021) occurred throughout the planning phase of the initial project. The current evaluation (September 2021 – September 2026) is a continuation of the initial evaluation to incorporate additional awardees. The current evaluation design and data collection instruments have been refined based on information learned in the previous evaluation and to account for differences in the new awardee program requirements. These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall evaluation design and data analysis plans, obtain advice and recommendations concerning the data collection instruments, and structure the evaluation and instruments to minimize overall and individual response burden. Consultations have occurred with the following individuals in connection with this study (listed in alphabetical order):

John Straus, MD, Director of Special Projects Massachusetts Behavioral Health Partnership, Co-Founder National Network of Child Psychiatry Access Programs, 617-790-4120, [email protected]. Years and areas of consultation: 2018‒present, serves as a representative of those from whom information is to be obtained.

Min Qi Wang, PhD., Professor, Behavioral and Community Health, University of Maryland School of Public Health, 301-405-6652, [email protected]. Years and areas of consultation: 2019–present, methodological and analytic expertise.

9. Explanation of Any Payment/Gift to Respondents

Respondents will not receive any payments or gifts.

10. Assurance of Confidentiality Provided to Respondents

The current project will fully comply with the Privacy Act of 1974 (5 U.S.C. Section 552a, 1998; https://www.justice.gov/opcl/privacy-act-1974). The Privacy Act may apply to some data collection activities (e.g., the study will collect email addresses from some respondents).

All respondents will be assured that their data will be kept private to the extent allowed by law. In addition, emails to inform participants about the data collection and any other introductory materials about the data collection will indicate HRSA’s Federal status and the purpose of the data collection. Please see Attachments A2‒A73 for email notifications. Supporting Statement B contains additional information on study procedures related to email communication.

The study meets the Common Rule definitions for human subjects research (45 CFR 46, Regulations for Protection of Human Subjects); however, the JBS Institutional Review Board (IRB) determined that this research is eligible for exemption under 45 CFR 46.101(b)(5) from 45 CFR Part 46 requirements (see Attachment A74). The JBS IRB requires that data collected be kept secure. Participants will be informed that the information they provide will be kept private. HRSA will only receive the de-identified data. To protect the subjects’ privacy, each subject will create a unique study ID number. All documents that identify participants by name will be kept in a locked file cabinet in the office of the research project managers. The documents will be stored for 3 years (along with all data) and then destroyed. All databases related to the study will, therefore, not contain subjects’ names or other personal identification (e.g., email addresses). This information will be stored in password-protected databases with well-established security systems to prevent unauthorized access.

11. Justification for Sensitive Questions

Personally identifiable information (PII), including participants’ names and email addresses, will be collected for administration of the surveys, SSIs, and FGDs. The surveys do not ask for information of a sensitive nature (e.g., sexual practices, alcohol or drug use, religious preference) other than race/ethnicity. Specifically, HP Survey respondents will be asked for their race and ethnicity. Collection of these data are necessary for the evaluation because a diverse workforce is important to patient-clinician communication and access to care for patients belonging to minority populations (U.S. Department of Health and Human Services, 2017). All data and information from participants will be stored in the secure facilities for 3 years after the study is completed, and we will adhere to Federal requirements regarding collection and storage of PII.

12. Estimates of Annualized Hour and Cost Burden

This section summarizes the total burden hours for this information collection effort in addition to the cost associated with those hours.

12A. Estimated Annualized Burden Hours

Exhibit 3 contains estimated response burdens for each subject population participating in the evaluation’s data collection activities.

Estimates for the response-hour burden were calculated (1) based on the methodology being used with each respondent population and (2) using the average completion time based on instrument pilot testing. For example, for the Program Implementation Survey, the average time of completion among pilot testers was approximately 15 minutes; however, we have rounded the burden estimate to 20 minutes for this particular survey to allow for additional time to provide responses to open-ended questions. Supporting Statement B contains additional information on pilot tests of the data collection tools to be used in the evaluation, as well as summaries of pilot test feedback and changes that were made to the data collection tools based on this feedback.

It should be noted that the list of forms in the table below does not match the list of forms in the 60- and 30-day Federal Register Notices because the forms were aggregated across program cohorts (i.e., 2021 PMHCA, and 2018/2019 PMHCA and 2018 MDRBD) and types of respondent (e.g., physician; nurse practitioner; physician assistant; counselor, social worker, and other community and social service specialist; practice manager; project director/principal investigator) in the 60- and 30-day Federal Register Notice and are disaggregated here.

Exhibit 3. Estimated Annualized Burden Hours

Type of Respondent	Form Name	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours
Physician	2021 PMHCA HP Survey	5,122	3	.25	3,841.5
Nurse Practitioner	2021 PMHCA HP Survey	1,499	3	.25	1,124.25
Physician Assistant	2021 PMHCA HP Survey	479	3	.25	359.25
Counselor, Social Worker, and Other Community and Social Service Specialist	2021 PMHCA HP Survey	664	3	.25	498
Other Health Care Professional/ Support Worker	2021 PMHCA HP Survey	267	3	.25	200.25
Practice Manager	2021 PMHCA Practice-Level Survey	2,950	3	.25	2,212.5
Project Director/ Principal Investigator	2021 PMHCA Program Implementation Survey	24	3	.33	23.76
Project Director/ Principal Investigator	2021 PMHCA Program Implementation SSI	24	1	1	24
HP/ Champion	PMHCA/MDRBD Champion SSI	48+56= 104	1	.5	52
HP/ Champion	PMHCA/MDRBD Champion FGD	24+28= 52	1	1	52
Counselors, Social Workers, and Other Community and Social Service Specialists	PMHCA/MDRBD Community Resources SSI	50+50= 100	1	.5	50
Health Education Specialist/ Community Health Worker	2021 PMHCA Care Coordinator SSI	24	2	.5	24
Health Education Specialist/ Community Health Worker	2018/2019 PMHCA/2018 MDRBD Care Coordinator SSI	28	1	.5	14
Total		11,337			8,475.51

12B. Estimated Annualized Burden Costs

Exhibit 4 summarizes the estimated annualized cost burden to respondents of the evaluation. Average hourly wage estimates and occupational profile codes were obtained from the U.S. Department of Labor, Bureau of Labor Statistics using wage estimates from 2021 (the most recently available estimates). The total respondent cost is calculated as hourly wage rate X time spent on the instrument X number of respondents.

Exhibit 4. Estimated Annualized Burden Costs

Type of Respondent (Occupational Profile Code)	Total Burden Hours	Hourly Wage Rate*	Total Respondent Costs
Physicians (29-1215;29-1216;29-1221)	3,842	$108.42	$416,549.64
Nurse Practitioners (29-1171)	1,124	$56.75	$63,787.00
Physician Assistants (29-1071)	359	$57.43	$20,617.37
Counselor, Social Worker, and Other Community and Social Service Specialist (21-000)	498+50= 548	$25.94	$14,215.12
Other Health Care Professionals (29-0000)	200	$29.53	$5,906.00
Practice Manager (11-9111)	2,213	$57.61	$127,490.93
Project Director/Principal Investigator (19-3099)	24+24=48	$43.70	$ 2,097.60
HP/Champion	52+52=104	$27.75	$2,886.00
Health Education Specialist/ Community Health Worker	38	$23.49	$892.62
Total	8,476		$654,442.28

*SOURCE: U.S. Department of Labor, Bureau of Labor Statistics. (2021, May). Occupational employment and wage statistics. https://www.bls.gov/oes/current/oes_stru.htm

13. Estimates of Other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

Other than time, there is no cost to respondents.

14. Annualized Cost to Federal Government

The cost to the Federal Government for this 5-year project is $4,023,331 or $804,666 per year on average. The total average cost for the project is $804,666 over a 1-year period. These costs cover all aspects of survey design, testing, data collection, and analysis. The method used to estimate the cost includes preparation of a detailed line-item budget that specifies all staff/consultant rates and labor hours by task, along with operational and other direct costs (e.g., telephone calls, reproduction).

In addition, it is estimated that one full-time equivalent HRSA staff member (Grade 13, Step 5) will spend 20% of his or her time (384 hours) to manage and administer the project. Assuming an annual salary of $148,445, Government personnel costs will be $29,689 over a 1-year period.

15. Explanation for Program Changes or Adjustments

This is a new information collection effort. However, previously approved OMB tools (OMB Control No. 0906-0052; expiration 06/30/2023) will continue to be administered with the 2018/2019 PMHCA and 2018 MDRBD program cohorts.

16. Plans for Tabulation, Publication, and Project Time Schedule

Project Time Schedule. As shown in Exhibit 5, the project covers a 3-year period commencing upon receipt of OMB approval.

Exhibit 5. Project Time Schedule

Activity	Time Schedule
Obtain OMB approval	Fall 2022/Winter 2023
Administer HP Survey, Practice-Level Survey, and Program Implementation Survey	3-4 months after OMB approval and at same timeframe 1 and 2 years after OMB approval
Administer Program Implementation SSI	2 years after OMB approval
Administer 2021 PMHCA Champion SSI/FGD and Community Resources SSI	2 years after OMB approval
Administer 2021 PMHCA Care Coordinator SSI	3-4 months after OMB approval and at same timeframe 2 years after OMB approval
Administer 2018/2019 PMHCA and 2018 MDRBD Champion SSI/FGD, Community Resources SSI, and Care Coordinator SSI	3-4 months after OMB approval
Data analysis	Beginning 3 months after OMB approval
Dissemination of findings through interim reports, infographics, and final report	Beginning 3 months after OMB approval through 2026

Analysis Plan. The HRSA MCHB evaluation will encompass the use of multiple instruments, collection of information, and analytical strategies. Both qualitative and quantitative data will be collected and analyzed to assess HCPs/HPs’ capacity to address patients’ behavioral health and access to behavioral health services among PMHCA and MDRBD programs. Qualitative data analysis will use a thematic approach to uncover underlying themes among the SSI and FGD responses. Quantitative data analyses will include the use of descriptive statistics, univariate analysis, and multivariate analysis. Finally, triangulation of methods (i.e., qualitative and quantitative data), when feasible, will be used to examine additional aspects of program achievements that may not be accomplished with individual methods. The planned qualitative and quantitative data analyses are explained in more detail in the remainder of this section.

Qualitative Data Analysis: The qualitative data for this evaluation will come from the SSI and FGD responses. The information collected will provide contextual information to better understand the nuances related to program implementation approaches and will complement and expand on data collected from the Program Implementation Surveys. The data will be analyzed to identify themes from the responses of project directors/principal investigators, program champions, informal and formal community resources partner representatives, and program-level care coordinators. As the first step in the data-cleaning process, audiotapes of the interviews will be transcribed and cleaned to remove any respondent-identifying information and any transcription mistakes; these transcriptions will serve as the qualitative data used for the study.

To facilitate the systematic analysis of the interview/discussion data, a computer-assisted qualitative and mixed-methods data analysis software package (e.g., MAXQDA, Version 18.2.0) will be used. Before the analyses begin, HRSA MCHB evaluation team staff will participate in a training workshop that specifies the qualitative analytic procedures and explains the importance of adherence to the procedures and of examination of inter-rater reliability. A codebook will be developed to guide the deductive coding process that contains the descriptive codes and their operational definitions, based on the specific evaluation questions under investigation and on the topics covered in the SSI/FGD guides. The purpose of deductive coding will be to apply the descriptive codes in the codebook to the SSI/FGD transcripts. The text passages to which these descriptive codes are applied will then be used in the inductive qualitative analysis.

The initial step in the inductive analysis process will be reading the raw data (i.e., cleaned and coded interview transcripts) to discover underlying raw data themes. The raw data themes will then be grouped into lower order themes based on common topics. Next, following the same coding procedures for grouping raw data themes, lower order themes will be grouped into higher order themes. Finally, higher order themes will be grouped into major categories.

Consensus among HRSA MCHB evaluation team members conducting the analyses will be reached at each step of the analytical process (i.e., raw data themes, lower order themes, higher order themes, major categories) before proceeding to the next step to achieve inter-coder reliability. This process ensures a consistent understanding and interpretation of the data.

Quantitative Data Analysis: The quantitative data for this evaluation will come from the HP, Practice-Level, and Program Implementation Surveys and will assess HPs’ capacity to address patients’ behavioral health and access to behavioral health services. Quantitative data will be collected from 2021 PMHCA enrolled/participating HPs, medical practice managers, and project directors/principal investigators. Selection of statistical analyses are determined by the evaluation questions, measurements of variables, type of sampling, and number of independent variables and outcome variables, as well as sample size.

Descriptive Statistics: The HRSA MCHB evaluation will use descriptive statistics to describe 2021 PMHCA cooperative agreement-funded programs and their programmatic activities. The purpose of descriptive analysis will be to understand the distribution of variables of interest, as well as to assess the accuracy of measurements, identify sources of error, and provide descriptive information. Frequencies will be run or the means and standard deviations of each variable will be calculated to examine the central tendency and distribution of variables. Knowledge of the distribution of data will inform the use of proper statistical techniques to conduct further analyses moving forward.

Analyses will also be conducted to identify random or systematic errors (e.g., instrumental noise) and to assess for missing values, because it is important to determine the potential bias of missing values. To address these potential limitations, the analysis team will consider the imputation of missing values for all variables with a large number of missing values. The team will review, select, and apply the most efficient method, based on careful consideration of the data set and type of missing data. The quantitative analysis team will also conduct cross tabulations to examine the relationship between the variables. The degree and statistical significance of association between variables is important not only for reporting relationships of interest, but also for supporting higher level analyses.

Univariate Analysis: Conducting univariate analysis will allow the analysis team to examine associations to identify variables associated with the outcomes of interest. These analyses may include contingency tables and chi-square tests for independence (Pearson’s), t-tests and univariate analysis of variance, linear logistic models when the dependent variable is binary, linear regression when the dependent variable is continuous, and/or Poisson regression when the dependent variable is measured in counts. In addition to these univariate analyses, the significant variables and the nonsignificant variables can be separated and decisions made on what to include in multivariate analysis.

Multivariate Analysis: The multivariate analysis will provide information about the impacts produced by 2021 PMHCA programs—direct and indirect and intended and unintended. Based on the evaluation questions and variable identification through the descriptive statistics and univariate analysis, the appropriate multivariate analysis will be determined and applied (e.g., linear or logistic regression, generalized estimating equations, multilevel analysis, cluster analysis, structural equation modeling). When interpreting the results of the analyses, both the statistical significance and the practical importance of the findings will be evaluated. The magnitude of changes will be compared with the literature and with practically meaningful standards.

For example, to determine whether and how HPs’ access to clinical behavioral health consultation has changed over time, a key consideration is the identification of a variable for “health professionals’ access.” The analysis team will review the data from the descriptive statistics and univariate analysis, as well as from a cluster analysis, if performed, and will confirm the data time points that can be used to examine change over time. Based on this information, the analysis team will determine the appropriate multivariate analyses.

Publication Plan. As stated in Section A.2, the goal of the evaluation of the PMHCA and MDRBD programs is to provide HRSA with information to guide future policy decisions regarding increasing HCPs/HPs’ capacity to address patients’ behavioral health and access to behavioral health services. It is therefore important to prepare and disseminate information that clearly and concisely presents evaluation results so that they can be appreciated by both technical and nontechnical audiences. Publication activities will include:

• Preparing and submitting to HRSA annual interim evaluation reports and a final evaluation report

• Preparing and submitting to HRSA up to 278 resources incorporating evaluation data

17. Reason Display of OMB Expiration Date Is Inappropriate

The OMB number and expiration date will be displayed on every page of every form/instrument.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

Reference

U.S. Department of Health and Human Services. (2017, August). Sex, race, and ethnic

diversity of U.S. health occupations (2011‒2015). Health Resources and Services Administration, National Center for Health Workforce Analysis. https://bhw.hrsa.gov/sites/default/files/bhw/nchwa/diversityushealthoccupations.pdf

6