Supporting Statement B_Final

Health Resources and Services Administration

Maternal and Child Health Bureau

Autism CARES Act Evaluation

OMB Control No. 0915-0335

Request for Clearance,

Supporting Statement,

and Data Collection Instruments

Part

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for the collection of data are representatives from all grantee sites receiving Health Resources and Services Administration (HRSA) grant support under the Autism Collaboration, Accountability, Research, Education, and Support (CARES) Act of 2014 and the reauthorized Autism CARES Act of 2019 (Attachment A). Grantee representatives may include principal investigators (PIs), project directors (PDs), and/or research assistants. The following are the grantee sites:

Training Grant Programs (64)

• Leadership Education in Neurodevelopmental and Related Disabilities (LEND) training program grantees (52)

• Developmental Behavioral Pediatrics (DBP) training program grantees (12)

Research Networks and Research Grants (24)

• Research Networks (4)

  • Autism Intervention Research Network on Behavioral Health (AIR-B)

  • Autism Intervention Research Network on Physical Health (AIR-P)

  • Developmental Behavioral Pediatrics Research Network (DBPNet)

  • Healthy Weight Research Network (HW-RN)

• Autism Single Investigator Innovation Program (Autism SIIP) grantees (2)

  • Autism Transitions Research Project

  • Autism Longitudinal Data Project

• Autism Field-Initiated Innovative Research Studies Program (Autism FIRST) grantees (6)

• Autism Secondary Data Analysis Research Program (Autism SDAR) grantees (12)

State Systems Grant Programs (5)

• Innovations in Care Coordination Grants (5)

Resource Centers (2) (These centers provide technical assistance and disseminate information to the grant programs.)

• Interdisciplinary Technical Assistance Center (ITAC) on Autism and Developmental Disabilities

• State Public Health Autism Resource Center (SPHARC)

Due to the small number and unique nature of each grant program, a census of all programs will be conducted. The findings will not be generalized beyond the respondents.

2. Procedures for the Collection of Information

The study is designed to measure the collective impact of MCHB’s Autism CARES Act Initiative, as well as to assess grantees’ progress in meeting the objectives of HRSA/Maternal and Child Health Bureau under the legislation. These objectives include increasing awareness of autism spectrum disorder (ASD) and other developmental disabilities (DD); training professionals to use valid, reliable, screening and diagnostic tools; providing evidence-based interventions for ASD/DD; supporting research to advance evidence-based or consensus-based guidelines for interventions; improving the overall system of services for ASD/DD; and reducing barriers to screening and diagnosis.

Information will be collected from each grantee in several ways. First, each grantee will submit specific program performance measures annually to the HRSA program officer through the Discretionary Grant Information System. These measures have already been cleared through the Office of Management and Budget. Additional performance data for the training grantees will be obtained through the National Information and Reporting System (NIRS), a web-based data reporting and retrieval system maintained by the Association of University Centers on Disabilities (AUCD) that enables grantees to manage data on their training programs, projects, activities, and products and helps them comply with Federal reporting requirements. Next, grantees will be invited annually to participate in an online Grantee Survey (Attachment B1). The annual Grantee Survey will provide information about the grantees’ previous year’s activities to inform assessment of annual progress and collective impact of grantee programs.

Third, information in the form of qualitative data will be gathered from a one-time semi-structured interview with the PI or PD from each grant program. Finally, grantee PIs and/or PDs for select grant programs (i.e., Research Networks and Autism SIIPs) will complete the Research Quantitative Data Collection Form that provides detailed quantitative information about project implementation.

The semi-structured interviews will provide data pertaining to the grantees’ implementation activities (e.g., training of maternal and child health [MCH] providers) and the intermediate results of those activites (e.g., screening/evaluation of underserved children by trained fellows or faculty), which are expected to lead to the long-term goal of early sngrescreening, diagnosis, and treatment of individuals with ASD/DD.

General Data Collection Procedures

Grantee Survey (Attachment B1). Each year in the summer, HRSA Project Officers will notify all Training and Research grantees about the Grantee Survey via email. The Study Team will follow up by sending all Training, Research, and State Systems grantees an email that includes a link to the Grantee Survey. This email will explain the purpose of the study, a list of frequently asked questions, instructions on how to access the web survey, and requested time to return the completed survey (10 business days). Grantees will receive the Study Team’s email and a phone number which they can use if they encounter any technical problems while completing the survey.

After the 10th business day, the Study Team will send up to two reminder emails to any grantees who have not completed their web survey. The reminder emails will include survey information, a link to the web survey, contact information for the help desk, and a reminder of the web survey due date.

Toward the end of the survey data collection period, a member of the Study Team will follow up by telephone with grantees who have not completed the Grantee Survey to encourage web survey completion, answer any questions they may have, and offer an opportunity to complete the survey over the telephone. If any grantees elect to complete the survey via telephone, a trained member of the Study Team will administer the Grantee Survey and record survey responses using the grantee’s unique survey link.

At the conclusion of the Grantee Survey data collection period, a thank-you email will be sent to all participating grantees. HRSA anticipates that all Training and Research grantees will complete the Grantee Survey annually for 3 years.

Semi-Structured Interviews (Attachments C–G). In the final year of each grant (or by November 2022), all grantees funded between FY 2017 through FY 2021 will be invited to participate in a semi-structured interview. Because grant periods vary, grantees will be recruited to participate in the semi-structured interview on a rolling basis.

In the final year of the grant, the Study Team, working in conjunction with the HRSA Project Officers, will notify select grantees about the semi-structured interview via email. This invitation email will include information about the purpose of the study and the interview and a list of frequently asked questions about the study. One week after sending the invitation email, a trained interviewer from the Study Team will begin contacting grantees via telephone to confirm receipt of the email, answer any questions, and schedule a semi-structured interview. Two weeks after sending the invitation email, the interviewer will send a reminder email to those grantees who have not scheduled their semi-structured interviews. For these remaining grantees, a second telephone call will be made to attempt to schedule an interview. Within 1 week of the scheduled interview, the interviewer will send an email to the grantee to confirm the interview date and time.

The interviews will be conducted over the telephone using the appropriate semi-structured guide (Attachments C–G), and with the permission of the respondent(s), the interviews will be recorded using a conference call interface with recording capability, such as WebEx. The interviews are expected to average between 60 and 90 minutes, depending on respondent type. At the conclusion of the entire data collection, a thank-you email will be sent to all grantees that participated in the semi-structured interview.

Table B.1 provides a crosswalk of each of the evaluation questions and objectives by data source. Each instrument referenced in this table is included in full as Attachments B–H. Not all evaluation questions may be completely answered by these instruments; however, these data will be supplemented by the Discretionary Grant Information System performance measures data.

Table B.1. Evaluation Question and Objective by Data Source

Evaluation Question	Data Collection Instrument
	Training Interview Guide (Attachment C)	Research Interview Guide (Attachment D)	State Systems Interview Guide (Attachment E)	ITAC and SPHARC Interview Guides (Attachments F and G)	Research Quantitative Data Collection Form (Attachment H)	Grantee Survey (Attachment B1)
Increase Awareness of ASD/DD
1a. How have HRSA Autism CARES Initiative efforts increased awareness of ASD/DD among maternal and child health (MCH) professionals, providers, and/or researchers?	X	X	X			X
1b. How have HRSA Autism CARES Initiative efforts increased public awareness of ASD/DD?	X	X	X			X
Reduce Barriers to Developmental Screening and Diagnosis
2a. What are the barriers to ASD/DD care identified by the HRSA Autism CARES efforts?	X		X			X
2a.1 How have the HRSA Autism CARES efforts addressed selected barriers to ASD/DD care?	X	X	X	X	X	X
2a.1a. What efforts have been most successful to reducing barriers and why?	X	X	X
2b. How have the HRSA Autism CARES efforts improved access to care?	X	X	X			X
Support Research to Advance Evidence-Based Interventions for ASD/DD
3a. How have the HRSA Autism CARES efforts supported or contributed to research to advance evidence-based interventions for children and youth with ASD/DD?	X	X	X		X	X
3b. How have the HRSA Autism CARES efforts worked to translate research into practice to support adoption of evidence-based interventions?	X	X	X		X	X
3c. What is the reach of the research that has resulted from the HRSA Autism CARES efforts?		X			X
3d. To what extent has the research addressed the needs of the underserved populations by filling knowledge gaps regarding factors that contribute to disparities?		X			X
3e. To what extent have DBP and research grantees mentored and developed new investigators in the field of ASD/DD?	X	X			X	X
Promote the Development of Evidence-Based Guidelines for Interventions (Improve System of Services)
4a. What efforts did the LEND, DBP, and state programs make to improve systems of care for children and youth with ASD/DD and their families?	X	X	X			X
4b. How have HRSA Autism CARES efforts promoted family-centered care for children and youth with ASD/DD and their families?	X	X	X			X
4c. How have HRSA Autism CARES efforts worked to reduce health disparities and promote health equity for children and youth with ASD/DD?	X	X	X
4d. How have HRSA Autism CARES efforts worked to improve the quality of care for children and youth with ASD/DD?	X	X	X
4e.1 How have the research programs advanced the development and validation of evidence-based guidelines for interventions?		X			X
4e.2. What has been the impact of evidence-based guidelines on practice (clinical and other settings)?		X
4f. How have the HRSA-funded ASD/DD programs expanded the coordination of care and partnerships in the area of ASD/DD?	X	X	X	X		X
4g. Has there been an improvement in access to care for children and youth with ASD/DD?	X	X	X			X
4h. Has there been an improvement in the early identification of ASD/DD manifestations in children and youth?	X
Train Professionals to Use Valid and Reliable Developmental Screening and Diagnostic Tools and Provide Evidence-Based Interventions for ASD/DD
5a. How have the LEND and DBP programs impacted the number of professionals trained in the screening, diagnosis, and evidence-based interventions related to ASD/DD?	X
5b. In what ways have the LEND and DBP programs affected the training of professionals who can provide screening, diagnosis, and evidence-based interventions related to ASD/DD?	X
5c. Has there been an increase in the number of trained professionals providing services to ASD/DD populations?	X
5c.1. Has there been an increase in the interdisciplinary workforce capable and available to address children's and youth’s ASD/DD health issues?	X
5c.2. How have the training programs broadened the diversity of the professionals and the interdisciplinary workforce trained in the screening, diagnosis, and evidence-based interventions related to ASD/DD?	X
5d. How have HRSA’s efforts supported or promoted timely referral and enrollment in early intervention services for children and youth with ASD/DD?	X		X
5e. How have HRSA Autism CARES efforts supported the enhancement of leadership skills among providers, professionals, and/or researchers at the local, state, and/or national levels?	X	X	X
5f. Has there been an increase in the number of leaders in the ASD/DD field at the local, state, and national levels?	X	X	X
5g. How have HRSA Autism CARES efforts improved the lives of children and youth and their families affected by ASD/DD?	X	X	X			X
Determine the Strategies Grantees are Utilizing to Sustain and Evaluate Their Programs
6a. What strategies are Autism CARES grantees implementing to sustain their programs?	X	X	X			X
6b. How have Autism CARES grantees evaluated their own efforts?	X	X	X	X
Assess How the National Resource Centers are Working Toward Their Objectives and Their Ability to Help Grantees Achieve Their Goals
7a. How have the National Resource Centers supported Autism CARES grantees (e.g., providing technical assistance, facilitating collaboration, convening Autism CARES grantees, collecting and managing data, producing and disseminating resources, providing a central resource center)?	X	X	X	X		X
7b. What strategies have the National Resource Centers found to be successful in serving grantees, and what factors have facilitated their successes?				X
7c. What major challenges do the National Resource Centers face in serving grantees?				X
7c.1. What challenges do Autism CARES grantees face in using the services of the National Resource Centers?	X	X	X	X
7c.2. How have the National Resource Centers and Autism CARES grantees worked to overcome these challenges?	X	X	X	X
7d. How have the National Resource Centers evaluated their own efforts to support Autism CARES grantees?				X

3. Methods to Maximize Response Rates and the Issue of Nonresponse

HRSA grantees who receive support through Autism CARES are required, as a condition of their grants, to provide the requested information to HRSA and to participate in the evaluation with the Study Team. Therefore, nonresponse will not be an issue.

4. Tests of Procedures or Methods to be Undertaken

The Grantee Survey and five semi-structured interview guides were pilot-tested from July–October 2019. Nine grantees participated in the pilot test, including three State Systems grantees, two LEND grantees, two DBP grantees, one Research Network grantee, and one Research Grants grantee.

Respondents were contacted via email and telephone. For the Grantee Survey, pilot test respondents were asked about the ease of comprehension (e.g., confusing wording or layout, confusing concepts) and length of time to complete. Recruitment materials were also pilot tested for comprehension. The Study Team conducted an internal debriefing on the pilot test of the semi-structured interviews, focusing on whether the questions were clear and elicited the desired types of information. All recruitment materials and instruments were revised to incorporate pilot test results including clarifying statements, removing repetitive questions, and adding definitions of key terms throughout the Grantee Survey (Attachment J).

The semi-structured interview guides used in the previous data collection effort (OMB 0915-0335) were updated to streamline the questions and improve clarity. All interview guides (Attachments C–G), the Research Quantitative Data Collection Form (Attachment H), and the Grantee Survey (Attachment B1) have either been pilot-tested or previously approved, and all procedures were developed with input from the HRSA Project Officers and HRSA grantees. All semi-structured interview guides (both pilot-tested and previously approved) and the Grantee Survey were designed and/or revised with HRSA and grantee input and follow a standard administration approach to ensure high-quality data are obtained across all grant programs while mininmizing burden on participants.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Table B.2 presents a summary of individuals consulted on study design, data collection, and/or analysis.

Table B.2. Individuals Consulted on Study Design, Data Collection, or Analysis

Name	Title	Telephone Number	Email
Primary Contractor – 2M Research
Regina James, MD	Project Director	703-721-8359	[email protected]
Amy Windham, PhD, MPH	Evaluation Lead	703-214-1512	[email protected]
Molly Matthews-Ewald, PhD, MS	Senior Public Health Research Analyst	817-856-0875	[email protected]
Amy Wieczorek Basl, MPH	Public Health Research Analyst	817-856-0866	[email protected]
Kathleen Santos, MS	Data Collection and Analysis Lead	703-214-1520	[email protected]
HRSA Staff
Leticia Manning, MPH	Lieutenant Commander, United States Public Health Service and Contracting Officer’s Representative	301-443-8335	[email protected]

Attachments to Supporting Statement B

Attachment A: Autism CARES Act of 2014 (P.L. 113–157) and Autism CARES Act of 2019 (P.L. 116-60)

Attachment B1: Grantee Survey

Attachment B2: Screenshots of Grantee Survey

Attachment C: Training Interview Guide

Attachment D: Research Interview Guide

Attachment E: State Systems Interview Guide

Attachment F: ITAC Interview Guide

Attachment G: SPHARC Interview Guide

Attachment H: Research Quantitative Data Collection Form

Attachment I: Definitions of Key Terms Used in the Grantee Survey