Supporting Statement A_OHS TIC Prog Eval_111221

Head Start Evaluation of a Trauma-Informed Care Program

OMB Information Collection Request

0970 - NEW

Supporting Statement Part A - Justification

November 2021

Submitted By:

Office of Head Start

Administration for Children and Families

U.S. Department of Health and Human Services

SUPPORTING STATEMENT A – JUSTIFICATION

1. Circumstances Making the Collection of Information Necessary

The Office of Head Start (OHS) is proposing to collect data for a new evaluation of a trauma-informed care program. The Head Start Evaluation of a Trauma-Informed Care Program is necessary to determine the intensity of professional development supports needed to enable teachers to effectively implement trauma-informed care in early care and education programs, and to evaluate the relative benefit of more intensive supports. Literature has identified practice-based coaching as a critical aspect of technical assistance for enhancing practices of early childhood educators, and this approach has been recommended by OHS (Practice-Based Coaching (PBC) | ECLKC (hhs.gov). However, there is enormous variability in the intensity and duration of coaching and little is known about how much is needed and what specific approaches may be most useful¹.

The evaluation will include a small randomized controlled trial across 10 sites within Head Start Region V. The information collected will be used to inform ongoing training and technical assistance (TTA) work provided by the National Centers funded by OHS.

This information collection is authorized by section 648 of the Head Start Act (42 USC 9801) and will be conducted on behalf of OHS by the National Center on Health, Behavioral Health, and Safety (NCHBHS), in partnership with the University of North Carolina at Chapel Hill (UNC-CH) and the Center for Childhood Resilience at the Anne & Robert H. Lurie Children’s Hospital of Chicago (Lurie).

2. Purpose and Use of the Information Collection

Purpose and Use

The overarching objective of this project is to evaluate the relative benefits of trauma-informed coaching and consultation (i.e., TTA) for enhancing Head Start teachers’ interactions with children, above and beyond standard TTA. Information obtained will be shared with a variety of stakeholders including ACF, OHS-funded Centers and grantees, and TTA professionals and early childhood education leaders across the country. In particular, information may suggest that future TTA for early childhood educators include ongoing coaching supports and expert content consultation. It will identify specific aspects of TTA (including duration and intensity) that may be most useful to coaches and teachers and that could be incorporated into future funding announcements and could inform best practices in early childhood TTA.

Additionally, information will be obtained about areas of teacher practice that may be most sensitive to change from a Trauma-Informed Care (TIC) approach, and the extent to which this is related to knowledge change. Although findings may be most applicable to TIC interventions in early care settings, we expect lessons learned to be more broadly applicable across different types of professional development models for educators. We will obtain information about the utility of a relatively low-cost method of assessing teacher practice change (Ecological Momentary Assessment), that could be incorporated into future federally-funded research on professional development interventions in education. Finally, information about this project will be disseminated in peer reviewed publications and presentations, with clearly identified limitations regarding its generalizability beyond Head Start sites with similar characteristics.

Research Questions

Research Question 1: What is the feasibility and fidelity of the RLR intervention within targeted HS sites (i.e., to what extent are each of the components implemented as intended)?

Research Question 2: How satisfied are coaches and teachers with the training and toolkit materials they are offered? Does this vary across group (low-intensity vs. high-intensity)?

2a. How satisfied are teachers with the coaching they receive, and does this vary by type of coaching (e.g., with or without Lurie staff)?

Research Question 3: Do participating TTA staff, coaches, and teachers gain knowledge of and change attitudes toward trauma-informed care from the beginning to end of the school year, and does the extent of improvement differ between the high- and low-intensity groups?

Research Question 4: How do teacher practices change over time in the four key areas of TIC: (1) creating a safe environment, (2) establishing healthy relationships and connectedness, (3) supporting and teaching emotion regulation, and (4) using self-care.

1. Does the high-intensity group improve more over time than the low-intensity group?

2. Does the rate of practice change over the school year vary between groups as additional supports are provided to the high-intensity group?

3. What implementation and contextual factors appear associated with practice change across groups (e.g., coaching intensity, coach background, site characteristics, other supports such as those provided by a mental health consultant, etc.)?

Study Design

The proposed evaluation will address important practice-based questions utilizing a randomized controlled design, the gold standard in evaluating intervention efficacy, with repeated measure of teacher practice. Information will be collected from staff from 10 sites within Head Start Region V, including approximately 75 teachers, administrators, and coaches. Sites will be randomized to either a low- or high-intensity TTA condition. The low-intensity condition will receive 4 hours of training, a “toolkit” of activity-based handouts, and access to virtual TA office hours. The high-intensity condition will include 4 hours of additional training on use of the toolkit modules, 6 hours of implementation support, and monthly classroom coaching.

Data collection will include virtual semi-structured interviews and focus groups at the end of the intervention, web-based surveys (baseline and post intervention), monthly web-based coaching logs, and teacher reports of classroom practices four times per day on a mobile app during 5 separate one-week periods across the school year. Lurie staff will also observe classrooms and record information about staff attendance at trainings and on TA calls (these activities do not impose burden on respondents and are not subject to the Paperwork Reduction Act²). See table below for specific activities and Supporting Statement B2 for additional details on our approach.

The primary limitation of this project is that the sample is small and is not intended to generalize beyond Head Start sites with similar characteristics. However, we have intentionally planned for as diverse a sample as possible and will carefully describe participant and site characteristics to identify these limitations, inform next steps for the field, and lay the groundwork for future, larger impact studies. We have intentionally planned for as diverse a sample as possible and will carefully describe participant and site characteristics to identify these limitations, inform next steps for the field, and lay the groundwork for future, larger impact studies. We are also unable to collect data on children as this extends beyond the scope of this project; this would clearly be important for validating the impact of TTA for early childhood teachers on children in the future. These and any other limitations identified will be clearly included in written products generated from this project.

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
Recruitment	Trauma-Informed System Change Instrument (TISCI) Questionnaire	Respondents: All site staff Content: Readiness for an organization to implement a TIC initiative at the practice and policy level Purpose: Assess site readiness as consideration for site selection	Mode: Web-based survey Duration: 10 mins.
Recruitment and post- intervention	Attitudes Related to Trauma-Informed Care (ARTIC) Questionnaire	Respondents: All site staff Content: Attitudes towards TIC Purpose: Assess site readiness as consideration for site selection and to evaluate change over time with intervention	Mode: Web-based survey Duration: 15 mins.
Recruitment	Site Application Form	Respondents: Site administrators Content: Characteristics of site, teachers, and students; implementation supports Purpose: Assess site readiness as consideration for site selection	Mode: Web-based survey Duration: 1 hour
Post-intervention	Site Administrator Interview	Respondents: Site administrators Content: Program experience, perceived impact, implementation supports and barriers Purpose: Assess feasibility and identify relevant implementation factors	Mode: Virtual interview Duration: 1 hour
Baseline	Background Forms	Respondents: Coaches and teachers Content: Demographics, education and experience Purpose: Describe sample	Mode: Web-based survey Duration: 5 mins.
Monthly during intervention	Coaching Logs	Respondents: Coaches Content: Activities, strategies, and topics addressed during coaching Purpose: Describe intervention and assess its fidelity	Mode: Web-based survey Duration: 15 mins.
Post-intervention	Coach Satisfaction Survey	Respondents: Coaches Content: Satisfaction and perceived utility of coaching methods, implementation supports and barriers Purpose: Assess feasibility of intervention	Mode: Web-based survey Duration:15 mins.
Post-intervention	Coach Interview	Respondents: Coaches Content: Program and coaching experience, implementation supports and barriers Purpose: Assess feasibility and identify relevant implementation factors	Mode: Virtual interview Duration: 1 hour
Baseline and post-intervention	Professional Self-Care Scale (PSCS)	Respondents: Teachers Content: Personal and professional self-care activities Purpose: Evaluate change over time with intervention	Mode: Web-based survey Duration: 5 mins.
Baseline and during intervention (4x/day for 5 separate school weeks)	Ecological Momentary Assessment (EMA) Survey (teachers only)	Respondents: Teachers Content: Report of specific strategies used with children and self-care practices Purpose: Evaluate change over time with intervention and between intervention groups	Mode: Survey on phone app Duration: 3 mins.
Post-intervention	Teacher Satisfaction Survey	Respondents: Teachers Content: Satisfaction and perceived helpfulness of program components, coaching, implementation supports and barriers Purpose: Assess satisfaction and differences between intervention groups	Mode: Web-based survey Duration: 15 mins.
Post-intervention	Teacher Focus Group	Respondents: Teachers Content: Program experiences, coaching experiences, implementation supports and challenges Purpose: Assess satisfaction with intervention and identify relevant implementation factors	Mode: Virtual focus group Duration: 1 hour
During intervention	Classroom Observation2	Respondents: Lurie intervention staff Content: Environmental supports for social-emotional development Purpose: Assess change over time between groups with intervention	Mode: Classroom observation ratings
During intervention	TTA Log2	Respondents: Lurie intervention staff Content: Record of attendance at trainings and on TA calls Purpose: Describe intervention dosage and explore relationship with outcomes	Mode: Web-based survey

3. Use of Improved Information Technology and Burden Reduction

Participating sites will be asked to provide data primarily through web-based surveys that can be completed on mobile devices meeting high levels of data security to reduce burden and support data quality. In addition, semi-structured interviews and focus groups will be conducted virtually to eliminate travel demands for participants. Remote information collection methods will also support site participation across a large geographical area by evaluators who are not local.

4. Efforts to Identify Duplication and Use of Similar Information

Some program information requested in the site application form is similar to what sites submit for the required Program Information Report (PIR) annual report³ (e.g., numbers of teachers and students). However, information in the PIR will not be available within the timeline of this project and is essential for selecting appropriate program sites and informing TTA. By aligning our questions with the PIR, sites will have data available which will minimize respondent burden.

5. Impact on Small Businesses or Other Small Entities

The collection of information does not impact small businesses or other small entities.

6. Consequences of Collecting the Information Less Frequently

Should information be collected less frequently than proposed, we anticipate they would not be reliable and/or valid (see explanation in item 7). We will also not be able to address the research questions of primary interest.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

We plan to collect information more frequently than quarterly for two measures - the primary outcome measure (Ecological Momentary Assessment completed by 30 Head Start teachers) and Coaching Logs (completed by up to 14 coaches who are educational or mental health specialists working with teachers).

Ecological Momentary Assessment (EMA) is a brief survey that respondents complete daily on a phone app in less than 5 minutes, within a response window tailored around each individual’s availability. This methodology requires repeated daily assessments over a period of time. We have chosen the minimal number of responses that have been demonstrated as

valid in prior studies (4 x/day for 5 days)⁴. Each response is a snapshot of the respondent’s experiences and interactions, which are only meaningful when there is adequate data collected over time to capture important contextual variability. We have planned for 5 assessments with this EMA measure in order to align with the specific practices the teachers are learning, and also to provide a large enough sample of within-person data to reliably interpret variability in individual scores. This information is the primary method by which we will evaluate if the professional development is effective in changing teacher practice. This is a much more cost-efficient and scalable method than classroom observation.

Coaching Logs will be completed monthly in order to reliably capture specific implementation data about how coaches are supporting teachers. This information is critical in order to understand the type and intensity of support teachers need to achieve practice change.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

Federal Register Notice

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on August 12, 2021, Volume 86, Number 153, page 44375, and provided a sixty-day period for public comment. During the notice and comment period, ACF received and responded to two requests for copies of the measures. No comments were received during the sixty-day comment period.

Efforts to Consult Outside the Agency

Head Start Region V leadership provided consultation on data collection measures and activities, and their input was incorporated to enhance clarity and minimize burden while maximizing utility of the information requested.

9. Explanation of Any Payment or Gift to Respondents

Respondents will not be provided any payment or gifts.

10. Assurance of Confidentiality Provided to Respondents

Respondents will be informed that every effort will be taken to protect their privacy, particularly that we will be asking for PII (names and email addresses) in order to create IDs to match participant data over time and distribute surveys. Participants will be identified with an ID rather than names in our records and will not be identified in any report or publication from the study. They will be informed that there may be times when federal or state law requires the disclosure of such records, although UNC-Chapel Hill will take steps allowable by law to protect the privacy of personal information. It is possible that information could be reviewed by representatives of the University for purposes such as quality control or safety, such as through an audit by the UNC-CH Institutional Review Board (IRB). Procedures to protect participant privacy will be detailed in consent forms approved by the UNC-CH IRB before any data collection begins (see Appendices A-C for director, coach, and teacher consents).

11. Justification for Sensitive Questions

Respondents will not be asked questions that are sensitive in nature.

12. Estimates of Annualized Burden Hours and Costs

The table below provides estimates of the hour burden of the collection of information, based on the number of respondents, frequency of the response, and an annual hour burden, calculated separately for each type of measure/method to be used.

Information Collection Title	Total Number of Respondents	Total Number of Responses Per Respondent	Average Burden Hours Per Response	Total/Annual Burden Hours	Average Hourly Wage (including fringe)	Total Annual Cost
Trauma-Informed System Change Instrument (TISCI) Questionnaire (all site staff)	75	2	0.17	26	$57.50	$1495
Attitudes Related to Trauma-Informed Care (ARTIC) Questionnaire (all site staff)	75	2	0.25	38	$57.50	$2185
Site Application Form (site administrators)	20	1	1	20	$52.82	$1056.40
Site Administrator Interview	10	1	1	10	$52.82	$528.20
Coach/Teacher Background Form	50	1	0.10	5	$67.46	$337.30
Coaching Logs	20	14	0.25	70	$67.46	$4722.20
Coach Satisfaction Survey	20	1	0.25	5	$67.46	$337.30
Coach Interview	20	1	1	20	$67.46	$1349.20
Professional Self-Care Scale (PSCS) (teachers only)	30	2	0.10	6	$35.14	$210.84
Ecological Momentary Assessment (EMA) Survey (teachers only)	30	100	0.07	210	$35.14	$7379.40
Teacher Satisfaction Survey	30	1	0.25	8	$35.14	$281.12
Teacher Focus Group	15	1	1	15	$35.14	$527.10
Total				433		$20,409.06

The cost to respondents was calculated using the relevant Bureau of Labor Statistics (BLS) job codes and wage data as follows:

• For all site staff, 25-000 Educational Instruction and Library Occupations from May of 2020 at $28.75, which is multiplied by 2 to account for fringe and benefits for an hourly wage of $57.50

https://www.bls.gov/oes/current/oes250000.htm

• For site administrators, 11-9031 Education Administrators, Preschool and Childcare Center/Program from May of 2020 at $26.41, which is multiplied by 2 to account for fringe and benefits for an hourly wage of $52.82

https://www.bls.gov/oes/current/oes119031.htm

• For coaches, 25-9031 Instructional Coordinators from May of 2020 at $33.73 which is multiplied by 2 to account for fringe and benefits for an hourly wage of $67.46.

https://www.bls.gov/oes/current/oes259031.htm

• For teachers, 25-2011 Preschool Teachers from May of 2020 at $17.57 which is multiplied by 2 to account for fringe and benefits for an hourly wage of $35.14

https://www.bls.gov/oes/current/oes252011.htm

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

No additional costs are anticipated for respondents or recordkeepers beyond their time.

14. Annualized Cost to the Federal Government

The core costs associated with information collection activities by a team at UNC-CH is estimated at $150,000 which includes 1) personnel time for the collection, management, analysis and dissemination of information, 2) costs for supplies and fees for a data collection survey platform, and 3) overhead costs (55.5%).

15. Explanation for Program Changes or Adjustments

This is a new information collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Data collection is expected to take place over approximately 1 year. Information with be tabulated and analyzed following completion of data collection activities in the summer of 2023. Mixed model trajectory analysis (MMTA), a form of hierarchical regression utilized with small samples, will be conducted with the EMA time series data by a statistician with expertise in this analytic approach. We plan to prepare at least one peer-reviewed publication to submit to a practitioner-focused journal such as Early Childhood Education Journal or Early Education and Development that shares findings relevant to practice-based coaching. Key limitations as described in A2 will be included.

Year	2021-2022				2022 - 2023				2023-2024
Quarter	Oct-Dec	Jan-Mar	Apr-Jun	July-Sept	Oct-Dec	Jan-Mar	Apr-Jun	July-Sept
Evaluation Activities
Site recruitment, selection, and randomization
Set up data collection protocols and platforms
Data collection
Analysis and Reporting
Data analyses
Preparation of final report for NCHBHS
Other dissemination activities (e.g., peer reviewed publication and presentation)

17. Reason(s) Display of OMB Expiration Date is Inappropriate

Not applicable.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1Artman-Meeker, K., Fettig, A., Barton, E. E., Penney, A., & Zeng, S. (2015). Applying an evidence-based framework to the early childhood coaching literature. Topics in Early Childhood Special Education, 35(3), 183-196.

Catriona Elek & Jane Page (2019) Critical features of effective coaching for early childhood educators: a review of empirical research literature, Professional Development in Education, 

45:4, 567-585, DOI: 10.1080/19415257.2018.1452781

2 As cited in 44 USC, 5 CFR Ch. 11 (1-1-99 Edition), 1320.3: Definitions: Facts obtained through direct observation by an employee or agent of the sponsoring agency or through nonstandardized oral communication in connection with such direct observations are considered an exclusion to the definition of “information” under PRA regulations (5 C.F.R. 1320.3(h)(3))

3 The Head Start PIR is approved under OMB #0970-0427.

4 Carson et al. (2010). Ecological momentary assessment: A research method for studying the daily lives of teachers. International Journal of Research and Method in Education, 33(2); Keller, et al. (2014). Teachers’ emotional experiences and exhaustion as predictors of emotional labor in the classroom: An experience sampling study. Frontiers in Psychology, 5, 1–10.