Supporting Statement A_clean_5-13-2020

Supporting Statement A

Evaluation of the Maternal and Child Health Bureau Pediatric Mental Health Care Access and Screening and Treatment for Maternal Depression and Related Behavioral Disorders Programs Project

OMB Control No. 0906-xxxx

Terms of Clearance: None

A. Justification

1. Circumstances Making the Collection of Information Necessary

In compliance with Section 3506(c)(2)(A) of the Paperwork Reduction Act of 1995, this submission requests Office of Management and Budget (OMB) approval of a 3-year clearance for the Health Resources and Services Administration (HRSA) to conduct an evaluation of the Maternal and Child Health Bureau (MCHB) Pediatric Mental Health Care Access (PMHCA) and Screening and Treatment for Maternal Depression and Related Behavioral Disorders (MDRBD) programs. This project will collect data to provide HRSA with information to guide future program and policy decisions regarding increasing health care providers’ (i.e., physicians, nurse practitioners, physician assistants, nurse midwives, and other health care professionals) capacity to address patients’ behavioral health and access to behavioral health services.

Title X of the 21st Century Cures Act (Cures Act) strengthens mental health and substance use disorder care for women, children, and adolescents. The PMHCA and MDRBD Notice of Funding Opportunity (NOFO) announcements (PMHCA 2018 NOFO Number: HRSA-18-122, PMHCA 2019 NOFO Number: HRSA-19-096, MDRBD 2018 NOFO Number: HRSA-18-101) were based on the Cures Act and articulate the requirements for each cooperative agreement-funded program. Section 10002 of the Cures Act supports increased access to pediatric mental health care. It authorizes HRSA to provide funding to promote behavioral health integration in pediatric primary care by supporting the development and improvement of existing statewide or regional pediatric mental health care telehealth access programs. Section 10005 supports screening, assessment, and treatment of maternal depression. It provides funds for states to establish, improve, or maintain existing programs that screen, assess, and treat women who are pregnant or gave birth in the preceding 12 months, for maternal depression. The services must include culturally and linguistically appropriate components. PMHCA and MDRBD programs aim to increase identification of behavioral health conditions by screening specified populations (e.g., children, adolescents, young adults, pregnant and postpartum women), especially those in rural, isolated, and/or underserved areas; providing clinical behavioral health consultation, care coordination support (i.e., communication/collaboration, accessing resources, and referral services), and training to health care providers; and increasing access to clinical interventions, including by telehealth. Provider education and training will support the knowledge and skills acquisition needed to accomplish this goal. This evaluation will allow HRSA to determine the extent to which PMHCA and MDRBD programs have met these objectives. The evaluation will be implemented by JBS International, Inc. (JBS), as part of a contract that is funded by HRSA (Contract No. HHSH250201400044I/HHSH25034007T).

2. Purpose and Use of Information Collection

As stated in Section A.1, the goal of this project is to provide HRSA with information to guide future program and policy decisions regarding increasing health care providers’ capacity to address patients’ behavioral health and access to behavioral health services.

The evaluation uses a mixed-methods design with data collection activities across all HRSA MCHB awardees. Methodologies for this study include surveys (e.g., online, mailed) and semi-structured interviews (SSIs).

The project will collect data from two groups: those enrolled in PMHCA and MDRBD programs (i.e., participating health care providers and practices) and project leadership implementing the programs, such as state-level project directors and principal investigators. Both groups will complete surveys annually in 2020, 2021, and 2022 and an SSI with project leadership will be conducted in 2022. Specifically, HRSA is requesting approval for the following:

PMHCA Health Care Provider (HCP) Survey – survey of enrolled PMHCA program HCPs examining their behavioral health capacity and their screening, assessment, and treatment of behavioral health conditions.

MDRBD HCP Survey – survey of enrolled MDRBD program HCPs examining their behavioral health capacity and their screening, assessment, and treatment of behavioral health conditions.

PMHCA Practice-level Survey – survey of enrolled PMHCA program office managers/office leadership examining their practices’ behavioral health services, behavioral health capacity, community linkages, operations, and staff training.

MDRBD Practice-level Survey – survey of enrolled MDRBD program office managers/office leadership examining their practices’ behavioral health services, behavioral health capacity, community linkages, operations, and staff training.

PMHCA Program Implementation Survey – survey of PMHCA program Project Directors examining HCP/practice recruitment and enrollment, HCP training, clinical behavioral health consultation, care coordination support, community linkages, program outreach and dissemination, and sustainability

MDBRD Program Implementation Survey – survey of MDRBD program Project Directors examining HCP/practice recruitment and enrollment, HCP training, clinical behavioral health consultation, care coordination support, community linkages, program outreach and dissemination, and sustainability

PMHCA Program Implementation Semi-Structured Interviews – interview with PMHCA program project leadership examining their programs’ implementation to complement and expand on data collected in the program implementation surveys.

MDBRD Program Implementation Semi-Structured Interviews – interview with MDRBD program project leadership examining their programs’ implementation to complement and expand on data collected in the program implementation surveys.

These data will assist in understanding the implementation and outcomes of PMHCA and MDRBD programs, which were initially funded in 2018 and have not yet been evaluated. Specifically, the collected data will be used to:

1. Study the efforts of awardee programs to achieve key outcomes.

2. Measure whether and to what extent awardee programs are associated with changes in these key outcomes.

3. Examine changes over time, within a state and/or across PMHCA and MDRBD programs, regarding (1) enrolled providers/practices related to screening, referral, and care coordination for behavioral health conditions; (2) provision of behavioral health services for mental health conditions in primary care settings by enrolled health care providers; (3) use of consultative services; and (4) facilitation of access to behavioral health services for mental health conditions.

4. Provide the data in reports to HRSA MCHB.

5. Develop infographics for dissemination by HRSA MCHB.

6. Inform future program implementation (e.g., future NOFOs) or other policy recommendations.

Section B contains additional information on study procedures on collection of information using these data collection tools. The data collection tools are also included as attachments in Section B.

3. Use of Improved Information Technology and Burden Reduction

The evaluation of MCHB PMHCA and MDRBD programs will follow a multimethod approach. Data collection methodologies for this evaluation will use surveys (i.e., web-based and email) and SSIs (i.e., telephone). All technology used for the survey administration (i.e., web-linked survey administered via email and web-linked survey administered via survey platform) will meet Federal requirements for Section 508 accessibility. Information technology will be used in the following ways:

• All survey participants will receive the web-linked survey via email. Electronic responses will be downloaded directly into a securely stored database.

• All SSIs will be conducted by telephone. For respondents who agree to be recorded, interviewers will record responses as they are given and will upload the recordings to a secured drive. For respondents who do not agree to be recorded, a notetaker will record responses and upload call notes to a secured drive.

• Reports and materials (e.g., infographics) generated from this project may be made available to the public through the HRSA MCHB website.

The data collection methods were selected for the evaluation because they will reduce participant burden while providing the evaluation with necessary data. Offering a web-based survey reduces burden to participants by eliminating the time it takes to write responses on a paper-based, mail-in survey. This same burden is reduced for respondents participating in telephone interviews because they will not have to write down responses to the questionnaires. In addition, having participants respond to an online survey eliminates the time needed to mail back a paper-based survey.

Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost or of data being lost in transit or delivered to an incorrect location. However, because not all respondents may prefer to complete web-based survey and to maximize completion rates, we may use alternative forms of administration (e.g., providing a printable PDF to participants). In this case, the printable PDF surveys can be returned either as attachments through encrypted emails or via mail, depending on the respondent’s preference. All hard copies will be submitted to JBS with unique alphanumeric identifiers, and the data will be entered into the online system at JBS. Hard copies will be stored in a locked file cabinet, with no name or identifying information attached.

4. Efforts To Identify Duplication and Use of Similar Information

There have been no prior studies or evaluations of HRSA MCHB PMHCA and MDRBD programs among any study population. The lack of evaluative studies is due to the Cures Act legislation having authorized these programs in 2016 and the first cooperative agreement programs having been funded starting in fiscal year 2018. Because HRSA MCHB PMHCA and MDRBD cooperative agreement-funded programs have not yet been evaluated, there is no similar or existing information available on these programs; hence, there is no duplication of efforts. No other bureaus or agencies are currently evaluating the programs.

In addition, there is no duplication of information within this evaluation because the data collection surveys and SSI guide were developed while taking into account that data awardees are required to report to HRSA based on their cooperative agreements. All potential data items were mapped to the evaluation questions to ensure no duplication of information and to reduce participant burden.

5. Impact on Small Businesses or Other Small Entities

Physicians, as part of participating health care providers (see section A-12 for additional information), and participating practices are included in the data collection efforts (e.g., surveys) for this evaluation. Although a portion of physicians may be employed by large hospitals or health systems, none of which are considered small businesses, some may be in a private practice or practice in small groups of physicians. Similarly, some participating practices may be part of large systems and, therefore, are not considered small businesses, whereas others may be private practices. Information collection for

this evaluation is not anticipated to have a significant impact on physicians or their practices.

The information to be obtained from physicians and participating practices is the minimum required for the intended use of the data and to achieve the objectives of the evaluation; however, completion of survey instruments will likely induce minimum burden. To reduce this burden, the surveys have been developed to be as short as possible, while still collecting necessary data, and attempts have been made to move respondents quickly through questions. For example, skip patterns have been added to the surveys so respondents do not need to answer questions that may not be relevant to them.

6. Consequences of Collecting the Information Less Frequently

As noted above in Section A.2., the collection of these data is critical to assessing the efforts of awardee programs to achieve key outcomes; measuring whether and to what extent awardee programs are associated with changes in these key outcomes; and examining changes over time, within a state and/or across PMHCA and MDRBD programs, with regard to evaluation variables of interest. The frequency of data collection as specified below is held to the minimum necessary to meet the needs of the evaluation goals and objectives.

HCP Surveys. HCP Surveys will be administered annually to health care providers enrolled in PMHCA and MDRBD programs with anticipated data collection in 2020, 2021, and 2022. Annual HCP Survey administration will allow for data collection from all potentially enrolled health care providers because health care providers may enroll in PMHCA and MDRBD programs on a rolling basis. Annual HCP Survey administration will also allow for examination of changes over time in behavioral health capacity and in screening, assessment, and treatment of behavioral health conditions for health care providers who complete the survey more than one time.

Practice-level Surveys. Practice-level Surveys will be administered annually to practice managers for practices enrolled in PMHCA and MDRBD programs with anticipated data collection in 2020, 2021, and 2022. Annual Practice-level Survey administration will allow for data collection from all potentially enrolled practices because practices may enroll in PMHCA and MDRBD programs on a rolling basis. Annual Practice-level Survey administration will also allow for examination of changes over time in behavioral health services, practice behavioral health capacity, community linkages, practice operations, and staff training for practices that complete the survey more than one time.

Program Implementation Surveys. Program Implementation Surveys will be administered annually to project directors/principal investigators from each of the 28 PMHCA and MDRBD state awardees with anticipated data collection in 2020, 2021,

and 2022). Program Implementation Survey data will be collected annually to allow for examination of changes in health care provider/practice recruitment and enrollment, health care provider training, clinical behavioral health consultation, care coordination support, community linkages, program outreach and dissemination, and sustainability.

Program Implementation Semi-Structured Interviews. SSIs will be administered one time via telephone to project directors/principal investigators from each of the 28 PMHCA and MDRBD awardees with anticipated data collection in 2022. Topics will be similar to the Program Implementation Surveys but will provide project directors/principal investigators an opportunity to discuss program implementation toward the end of the project period.

There are no legal obstacles to reduce the burden.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The request fully complies with the regulation.

8. Comments in Response to the Federal Register Notice/Outside Consultation

Section 8A

A 60-day Federal Register Notice was published in the Federal Register on October 17, 2019, Vol. 84, No. 201, pp. 55579-80 (see Attachment A1). Public comments are provided and responses to comments are provided as attachments to this supporting statement.

Section 8B

Consultations on the evaluation design, data collection instruments and protocols, survey and SSI questions, data management, and analysis of this evaluation have occurred throughout the planning phase of this project. These consultations have provided, and will continue to provide, the opportunity to ensure the technical quality and appropriateness of the overall evaluation design and data analysis plans, to obtain advice and recommendations concerning the data collection instruments, and to structure the evaluation and instruments to minimize overall and individual response burden. Consultations have occurred with the following individuals in connection with this study (listed in alphabetical order):

John Straus, M.D., Director of Special Projects Massachusetts Behavioral Health Partnership, Co-Founder National Network of Child Psychiatry Access Programs, 617-790-4120, [email protected]. Years and areas of consultation: 2018‒present, serves as a representative of those from whom information is to be obtained.

Min Qi Wang, Ph.D., Professor, Behavioral and Community Health, University of Maryland School of Public Health, 301-405-6652, [email protected]. Years and areas of consultation: 2019–present, methodological and analytic expertise.

9. Explanation of Any Payment/Gift to Respondents

Respondents will not receive any payments or gifts.

10. Assurance of Confidentiality Provided to Respondents

The current project will fully comply with the Privacy Act of 1974 (5 U.S.C. Section 552a, 1998; https://www.justice.gov/opcl/privacy-act-1974). The Privacy Act may apply to some data collection activities (e.g., the study will collect email addresses from some respondents).

All respondents will be assured that their data will be kept private to the extent allowed by law. In addition, emails to inform participants about the data collection and any other introductory materials about the data collection will indicate HRSA’s Federal status and the purpose of the data collection. Please see Attachments A2‒37 for email notifications. Section B contains additional information on study procedures related to email communication.

The study meets the Common Rule definitions for human subjects research (45 CFR 46, Regulations for Protection of Human Subjects); however, the JBS Institutional Review Board (IRB) determined that this research is eligible for exemption under 45 CFR 46.101(b)(5) from 45 CFR Part 46 requirements (see Attachment A38). The JBS IRB requires that data collected be kept secure. Participants will be informed that the information they provide will be kept private. HRSA will only receive the deidentified data. To protect the subjects’ privacy, each subject will be assigned a unique study ID number. All documents that identify participants by name will be kept in a locked file cabinet in the office of the research project managers. The documents will be stored for 3 years (along with all data) and then destroyed. All databases related to the study will, therefore, not contain subjects’ names or other personal identification (e.g., email addresses). This information will be stored in password-protected databases with well-established security systems to prevent unauthorized access.

11. Justification for Sensitive Questions

Personally identifiable information (PII) including participants’ names and email addresses will be collected for administration of the surveys and SSIs. The surveys do not ask for information of a sensitive nature (e.g., sexual practices, alcohol or drug use, religious preference) other than race/ethnicity. Specifically, HCP Survey respondents will be asked for their race and ethnicity. Collection of these data are necessary for the evaluation because a diverse workforce is important to patient‒clinician communication and access to care for patients belonging to minority populations (U.S. Department of Health and Human Services, 2017). All data and information from participants will be stored in the secure facilities for 3 years after the study is completed and we will adhere to Federal requirements regarding collection and storage of PII.

12. Estimates of Annualized Hour and Cost Burden

This section summarizes the total burden hours for this information collection effort in addition to the cost associated with those hours.

12A. Estimated Annualized Burden Hours

Exhibit 1 contains estimated response burdens for each subject population participating in the evaluation’s data collection activities.

Estimates for the response-hour burden were calculated based on the methodology being used with each respondent population and were calculated using the average completion time based on instrument pilot testing. For example, for the Program Implementation Survey, the average time of completion among pilot testers was 16 minutes; however, we have rounded the burden estimate to 20 minutes for this particular survey to allow for additional time to provide responses to open-ended questions. Section B contains additional information on pilot tests of the data collection tools to be used in the evaluation as well as summaries of pilot test feedback and changes that were made to the data collection tools based on this feedback.

Exhibit 1. Estimated Annualized Burden Hours

Form Name	Number of Respondents	Number of Responses per Respondent	Total Responses	Average Burden per Response (in hours)	Total Burden Hours
Health Care Provider Survey	13,035	3	39,105	0.17	6,648
Practice-Level Survey	4,165	3	12,495	0.25	3,124
Program Implementation Survey	28	3	84	0.33	28
Program Implementation Semi-Structured Interview	28	1	28	1	28
Total	17,256		51,712		9,828

12B. Estimated Annualized Burden Costs

Exhibit 2 summarizes the estimated annualized cost burden to respondents of the evaluation. Average hourly wage estimates and occupational profile codes were obtained from the U.S. Department of Labor, Bureau of Labor Statistics using wage estimates from 2018 (the most recently available estimates). The total respondent cost is calculated as total burden hours X (hourly wage rate + the cost of fringe benefits and overhead, calculated at 100 percent of hourly wage rate).

Exhibit 2. Estimated Annualized Burden Costs

Type of Respondent (Occupational profile code)	Total Burden Hours	Hourly Wage Rate* ($/hr)	Fringe Benefits and Overhead ($/hr)	Total Respondent Costs
Physicians (29-1069)	3,903	$98.02	$98.02	$765,144.12
Nurse Practitioners (29-1171)	1,556	$52.90	$52.90	$164,624.80
Physician Assistants (29-1071)	770	$52.13	$52.13	$80,280.20
Nurse Midwives (29-1161)	112	$51.40	$51.40	$11,513.60
Other Health Care Professionals (29-0000)	308	$39.42	$39.42	$24,282.72
Practice Manager (11-9111)	3,124	$54.68	$54.68	$341,640.64
Project Director/Principal Investigator (19-3099)	56	$41.22	$41.22	$4,616.64
Total				$1,392,102.72

*SOURCE: U.S. Department of Labor, Bureau of Labor Statistics. (2018). Occupational employment statistics. Retrieved from https://www.bls.gov/oes/current/oes_stru.htm

13. Estimates of Other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

Other than time, there is no cost to respondents.

13. Annualized Cost to Federal Government

The cost to the Federal Government for this 5-year project is $2,456,817, or $491,363 per year on average. The total average cost for the project is $491,363 over a 1-year period. These costs cover all aspects of survey design, testing, data collection, and analysis. The method used to estimate the cost includes preparation of a detailed line-item budget that specifies all staff/consultant rates and labor hours by task, along with operational and other direct costs (e.g., telephone calls, reproduction).

In addition, it is estimated that one full-time equivalent HRSA staff member (Grade 13, Step 5) will spend 20% of his or her time (384 hours) to manage and administer the project. Assuming an annual salary of $148,445, government personnel costs will be $29,689 over a 1-year period.

Total costs to the Federal Government are, therefore, $521,052 annually.

13. Explanation for Program Changes or Adjustments

This is a new information collection effort.

13. Plans for Tabulation, Publication, and Project Time Schedule

Project Time Schedule. As shown in Exhibit 3, the project covers the 3-year data collection period commencing upon receipt of OMB approval.

Exhibit 3. Project Time Schedule

Activity	Time Schedule
Obtain OMB approval	Spring 2020
Administer Program Implementation Survey	1–3 months after OMB approval and at same timeframe 1 and 2 years after OMB approval
Administer HCP Survey and Practice-level Survey	2–4 months after OMB approval and at same timeframe 1 and 2 years after OMB approval
Administer Program Implementation SSI	2 years after OMB approval
Data analysis	Beginning 3 months after OMB approval
Dissemination of findings through interim reports, infographics, final report	Beginning 3 months after OMB approval through 2023

Analysis Plan. The HRSA MCHB evaluation will employ a mixed-methods approach to data collection and analysis through the use of multiple data collection instruments and both quantitative and qualitative analytical strategies. The ultimate aim is to use this mixed-methods approach to assess health care providers’ capacity to address patients’ behavioral health and access to behavioral health services among PMHCA and MDRBD programs. Qualitative data analysis will use a deductive approach to uncover underlying or recurring themes among the SSI responses. Quantitative data analyses will allow for examination of the association between independent variables (i.e., related to awardee program activities or awardee characteristics) and the outcomes of interest (e.g., questions related to health care providers’ capacity to address patients’ behavioral health). While causal inferences cannot be made, the study will examine factors that contribute to, or are associated with, outcomes of interest. Finally, triangulation of methods (i.e., qualitative and quantitative data), when feasible, will be used to examine additional aspects of program achievements that may not be accomplished through the use of either method alone. The planned quantitative and qualitative data analysis approaches are provided below followed by examples of analyses addressing the overarching evaluation questions in the remainder of this section.

Quantitative Data Analysis Approach: The quantitative data for this evaluation will be collected through the HCP, Practice-level, and Program Implementation Surveys administered to PMHCA- and MDRBD-enrolled health care providers, medical practice managers, and project directors/principal investigators. Selection of statistical analyses will be determined by the evaluation questions, the measurement characteristics of variables, the type of sampling and sample size, and the specific independent variables and outcome variables of interest. The following three-step analytic approach is planned to guide data analyses:

Step 1: Assessing the distribution of data and provider characteristics.

The HRSA MCHB evaluation will use descriptive statistics to describe the PMHCA and MDRBD awardees and their programmatic activities. The purpose of these analyses will be to examine the distribution of data for variables of interest, assess the accuracy of measurements, identify potential sources of bias/error, and assess for missing values to determine potential bias of missing values. Analyses will produce frequencies, measures of central tendency (i.e., means), and standard deviations. Knowledge of the distribution of data will inform the use of appropriate statistical strategies moving forward.

The most practical technique for testing missing completely at random (MCAR) is to create dummy variables for whether a variable is missing (i.e., 1 = missing; 0 = observed) and run t-tests and chi-square tests between this variable and other variables in the database to see if missingness on this variable is related to the values of other variables. Algorithms have been developed for doing this automatically without having to create all those dummy variables. When this test statistic is not significant, the data are considered MCAR. If the test statistic is significant, this significant variable will be included in the multiple imputations together with other predictors. Missing data can be continuous or categorical and they will be handled differently as described below.

• For continuous data, the most popular method for multiple imputation of missing data is the Markov Chain Monte Carlo (MCMC) algorithm. This method assumes multivariate normality which implies that valid imputations may be generated by linear regression equations. The algorithm is widely available and, most importantly, can handle arbitrary patterns of missing data.

• For categorical variables, we will adopt the regularized iterative Multiple Correspondence Analysis (MCA) algorithm. The regularized iterative MCA algorithm first consists of coding the categorical variables using the indicator matrix of dummy variables. Then, in the initialization step, missing values are imputed with initial values such as the proportion of the category for each category using the non-missing entries. The second step of the regularized iterative MCA algorithm consists of performing MCA on the completed dataset. Then, it imputes the missing values with the regularized reconstruction formulae of the fitted matrix for the regularized scores and loadings. These steps of estimation of the parameters via MCA and imputation of the missing values using the regularized fitted matrix are iterated until convergence.

The team will review, select, and apply the most efficient method, based on careful consideration of the data set and type of missing data. Any final reporting issuing from the data will indicate that missing values were imputed, if necessary, and will indicate the imputation method as described above. The quantitative analysis team will also conduct cross tabulations to examine the relationship between the variables. The degree and statistical significance of association between variables is important not only for reporting relationships of interest, but also for supporting higher level analyses.

Step 2: Assessing associations between variables to identify significant variable(s) associated with the outcome in the evaluation questions.

To identify variables associated with the outcomes of interest, we will use (1) contingency tables and chi-square tests for independence (Pearson’s), t-tests and analysis of variance, linear logistic models when the dependent variable is binary, (2) linear regression when the dependent variable is continuous, and/or (3) Poisson regression when the dependent variable is measured in counts. Through this step, the significant variables and the nonsignificant variables can be separated, and decisions made on what to include in multivariate analysis.

Step 3. Conducting the multivariate analysis in order to investigate the evaluation questions.

Multivariate analyses will provide information about the grantee and programmatic characteristics that may be attributed to the PMHCA and MDRBD programs. Based on the evaluation questions and variable identification through Steps 1 and 2, the appropriate multivariate analysis strategy will be determined and applied (e.g., linear or logistic regression models or multinomial regression, depending on the measurement of the outcome variable; generalized estimating equations (GEE) when the data consist of multiple time points; multilevel analysis, which can be used for the hierarchical data structure where data are considered clustered; cluster analysis if the data contain a series of variables reflecting the characteristics of samples; structural equation modeling [SEM]), if we are testing theory [as outlined in the theoretical/conceptual framework] or interrelationship among variables. When interpreting the results of the analyses, both the statistical significance and the practical importance of the findings will be evaluated. The magnitude of changes will be compared with the literature and with practically meaningful standards.

Approach for Quantitative Analyses Addressing Overarching Evaluation Questions. With regard to the overarching evaluation questions of interest, to examine evaluation questions related to programs’ activities supporting providers’ capacity to address patients’ behavioral health, it is expected that data analyses will include the frequency distribution, percentages, binomial test to determine whether the observed proportion is significantly different from a known proportion, and Chi-square (χ²) test to determine whether a statistically significant association exists between rows and columns in a contingency table. Associations may also be made using logistic regression to predict the outcome measures with a selected group of predictors.

Data analyses to examine evaluation questions related to changes in health care providers’ capacity to address patients’ behavioral health will include GEE and mixed models will be adopted to investigate the longitudinal trend of behavioral health changes. The health care providers’ capacity may serve as time-varying variable in relation to the changes of the outcome. For example, to determine whether and how health care providers’ access to clinical behavioral health consultation has changed over time, a key consideration is the identification of a variable for “health care provider access.” Using the steps outlined below, Following the steps above, the analysis team will review the data from the descriptive statistics (Step 1) and associations between variables (Step 2), as well as a cluster analysis if performed, and will confirm the data time points that can be used to examine change over time. Based on this information, the analysis team will determine the appropriate multivariate analyses (Step 3).

Similarly, examination of evaluation questions related to changes in health care practices’ capacity to address patients’ behavioral health will include GEE and mixed models will be adopted to investigate the longitudinal trend of behavioral health changes. As with health care providers, the health care practices’ capacity may serve as time-varying variable in relation to the changes of the outcome.

Data analyses to examine evaluation questions related to how program activities are associated with changes over time in access to behavioral health services will include mediation and moderation modeling to examine interrelations and potentially causal effect of factors associated with these changes over time in access to behavioral health services. For example, when examining which variables are associated with changes in provider delivery of screening over time, we will examine the assessment of the data distribution and characteristics from Step 1 and significant variables from Step 2 and then include the significant variables into a multivariate logistic model or multiple linear regression to screen out a group of the most robust and significant predictors (Step 3). Furthermore, the SEM may be adopted to examine the interrelationships among independent variables in relation to the dependent variable.

Qualitative Data Analysis: The qualitative data for this evaluation will come from the SSI responses. The information collected will provide contextual information to better understand the nuances related to program implementation and will complement data collected from the Program Implementation Surveys. The interview data will be analyzed to identify themes from the responses of project directors/principal investigators about their program. As the first step in the data-cleaning process, audiotapes of the interviews will be transcribed, deidentified, and cleared of any transcription mistakes; these transcriptions will serve as the qualitative data used for the study.

To facilitate the systematic analysis of the interview data, a computer-assisted qualitative and mixed-methods data analysis software package such as MAXQDA (Version 18.2.0) will be used. Before the analyses begin, HRSA MCHB evaluation team staff will participate in a training workshop that specifies the qualitative analytic procedures and explains the importance of adherence to the procedures and examination of inter-rater reliability. A codebook will be developed to guide the deductive coding process that contains the descriptive codes and their operational definitions based on the specific evaluation questions under investigation and on the topics covered in the SSI guides. The purpose of deductive coding will be to apply the descriptive codes in the codebook to the SSI transcripts. The text passages to which these descriptive codes are applied will then be used in the inductive qualitative analysis. The initial step in the inductive analysis process will be reading the raw data (i.e., cleaned and coded interview transcripts) to discover underlying raw data themes. The raw data themes will then be grouped into lower order themes based on common topics. Next, following the same coding procedures for grouping raw data themes, lower order themes will be grouped into higher order themes. Finally, higher order themes will be grouped into major categories. Consensus among HRSA MCHB evaluation team members conducting the analyses will be reached at each step of the analytical process (i.e., raw data themes, lower order themes, higher order themes, and major categories) before proceeding to the next step to achieve inter-coder reliability. This process ensures a consistent understanding and interpretation of the data.

Publication Plan. As stated in Section A.2, the goal of the evaluation of MCHB PMHCA and MDRBD programs is to provide HRSA with information to guide future policy decisions regarding increasing health care providers’ capacity to address patients’ behavioral health and access to behavioral health services. It is therefore important to prepare and disseminate information that clearly and concisely presents evaluation results so that they can be appreciated by both technical and nontechnical audiences. Publication activities will include:

• Preparing and submitting to HRSA annual interim evaluation reports and a final evaluation report.

• Preparing and submitting to HRSA a series of 17 infographics incorporating evaluation data.

13. Reason Display of OMB Expiration Date Is Inappropriate

The OMB number and Expiration date will be displayed on every page of every form/instrument.

13. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

Reference

U.S. Department of Health and Human Services. (2017). Sex, race, and ethnic diversity of U.S. health occupations (2011‒2015). Rockville, MD: Health Resources and Services Administration, National Center for Health Workforce Analysis. Retrieved from https://bhw.hrsa.gov/sites/default/files/bhw/nchwa/diversityushealthoccupations.pdf

8