0990-OWH TIC TTA Evaluation OMB Supporting Statement A_revised

O MB Supporting Statement A: Justification

Cross-site Evaluation of the National Training Initiative on Trauma-Informed Care (TIC) for Community-Based Providers from Diverse Service Systems

HHSP23320095624WC

Final

October 15, 2015

Prepared for:
Dr. Adrienne Smith
OWH/OS
200 Independence Ave S.W.
Room 728F
Washington, DC 20201

Submitted by:
Abt Associates
55 Wheeler Street
Cambridge, MA 02138

In Partnership with:
Rutgers’ School of Criminal Justice

Background on the National Training Initiative on Trauma-Informed Practice for Community-Based Organizations in Diverse Service Sectors 2

1. Overview of the Training Initiative 2

2. Identification/Selection of Participants for the Training Initiative 3

3. Pre-training Assessment(s) 4

A. Justification 5

1. Circumstances Making the Collection of Information Necessary 5

2. Purpose and Use of Information Collection 6

3. Use of Improved Information Technology and Burden Reduction 8

4. Efforts to Identify Duplication and Use of Similar Information 9

5. Impact on Small Businesses or Other Small Entities 9

6. Consequences of Collecting the Information / Less Frequent Collection 9

7. Special Circumstances Relating to the Guidelines of CFR 1320.5 10

8. Comments in Response to the Federal Register Notice/Outside Consultation 10

9. Explanation of Any Payment or Gift to Respondents 10

10. Assurance of Confidentiality Provided to Respondents 10

11. Justification for Sensitive Questions 11

12. Estimates of Annualized Burden Hours and Costs to TTA Respondents 11

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 12

14. Annualized Cost to the Federal Government 12

15. Explanation of Program Changes or Adjustments 14

16. Plans for Tabulation and Publication and Project Time Schedule 14

17. Reason(s) Display of OMB Expiration Date is Inappropriate 17

18. Exceptions to Certification for Paperwork Reduction Act Submissions 18

Background on the National Training Initiative on Trauma-Informed Practice for Community-Based Organizations in Diverse Service Sectors

1. Overview of the Training Initiative

The U.S. Department of Health and Human Services Office on Women’s Health established the National Training Initiative on Trauma-Informed Practice for Community-Based Organizations in Diverse Service Sectors (Training Initiative).

The primary learning objectives of the Training Initiative were to:

• Educate service providers from diverse sectors to recognize signs of trauma;

• Assist service providers to recognize the impact of gender when assessing for trauma; and

• Build capacity of diverse providers to foster service delivery system changes to offer sensitive and appropriate trauma-informed practice.

The Training Initiative was implemented through three phases:

• Phase I: The planning and development of the Training Initiative including the creation and piloting of the training curriculum. This phase was completed in May 2013;

• Phase II: Providing an initial round of regional training and post-training technical assistance (TA) activities to community-based organizations (CBOs) in five regions. This phase was completed in September 2013; and

• Phase III: Providing a second round of regional training and post-training TA activities to CBOs in four regions, creating a Train-the-Trainer (TOT) model, and piloting with 4 new trainers. This phase was completed in September 2014.

Phase I of the training initiative included 84 participants from 22 CBOs across five OWH Regions. Over the following two years (Phases II and III) 219 individuals from 49 CBOs across nine OWH Regions received 2-day regional trainings using the training curriculum. Participants in each region included between four to six organizations represented by up to five staff from each organization. These individuals held diverse roles within the organization ranging from direct practice to leadership positions. The trainings promoted peer involvement and collaboration, and women with lived experience were encouraged to be a part of the training.

Table 1 provides an overview of contract activities provided across the three phases of the training initiative.

2. Identification/Selection of Participants for the Training Initiative

The TIC trainers and the Regional Women’s Health Coordinators (RWHCs) worked together to develop a list of potential community-based organizations to invite along with a set of criteria to identify participants for the training. These criteria included:

1. Identifying community-based organizations from the HHS Office on Women’s Health portfolio that were diverse geographically (i.e., rural, urban), by population of women served (i.e., minority women, mothers, etc.), and by type of services (social services, violence prevention services, etc.) provided. Ensuring that agencies and organizations recruited served a diverse range of women affected by trauma, including those serving:

   1. Individuals living with mental illness

   2. Female veterans

   3. Recent immigrants and refugees

   4. Minority women

   5. Survivors of sexual assault and domestic violence

   6. Formerly incarcerated women

2. Involving only agencies and community-based organizations with annual budgets of $10 million dollars or less.

Community-based organizations were provided an overview of the goals and objectives of the project and invited to complete the Request for Training Form in order to ensure their interest in the training, their staff’s availability for 2 days of training, and their commitment to post-training technical assistance. In addition, requesters were required to form a team of 4 to 5 staff with different levels of involvement in the agency or organization to be considered for the training. Specifically, agencies teams were to include an agency administrator, a program leader/executive, a clinical or program professional, and a staff person or consumer with lived experience of trauma. Within those parameters, agencies and organizations nominated their own (relevant) staff. The Request for Training Forms were then reviewed by regional OWH staff and a selection was made.

3. Pre-training Assessment(s)

The OWH evaluation contractor was not selected until Phase III training and technical assistance events were nearly complete. In addition, no other contractor was engaged earlier to conduct pre-tests or administer baseline measures to trainees. We further note that some agency staff may have been exposed to and/or been familiar with TIC principles prior to participating in the training initiative. The OWH evaluation contractors will probe the matter of prior exposure in the online survey and site visit interviews.

1. Justification

1. Circumstances Making the Collection of Information Necessary

This OMB package is submitted pursuant to Section 301 of the Public Health Service Act (42 U.S.C.241).

The long unmet needs of girls and women in our communities and the changing nature of health and human services delivery systems with an accelerated shift to home and community-based services, make it imperative to implement effective strategies to promote trauma-informed care (TIC) in these settings. People who experience adverse events—whether in childhood or adulthood, from family violence, crime or community violence, times of emergency or disaster, or political or war violence—share a common risk of traumatic effects from those events. Trauma can also have a significant impact on families across generations, communities, organizations, and workplaces. Policymakers and providers in many service sectors recognize the central role of trauma in causing or complicating physical and behavioral health conditions and the critical need for TIC systems. Because trauma impacts survivors in such profound ways, those who have trauma histories are found in multiple service sectors. Trauma drives gender-based health disparities, disproportionately affecting women and spawning serious physical and psychological health issues across the lifespan.¹ For girls and women impacted by trauma, using health and human services in provider organizations that are uninformed about trauma poses at a minimum risks that needs will not be adequately met. Absent knowledge about trauma as a possible underlying condition, a provider may misinterpret somatic and behavioral symptoms and miss an opportunity for intervention. At worst, there is a risk of re-traumatization because of coercive or insensitive practices employed in some settings across diverse service systems. By contrast, an informed provider engaged in the provision of TIC promotes recovery and fosters resiliency in the girls and women who turn to the organization or program for service.

The prevalence of traumatic life experience in children is high, with more than two thirds reporting at least one event by age 16, and of these 13.4 percent exhibit signs of post traumatic stress.² This prevalence is higher in females than males, due to higher rates of sexual assault and domestic violence, accounting for higher rates of PTSD and depression among girls and women, conditions that are associated with greater lifetime health risks.³ Reports from a nationally representative survey of 12- to 17-year-old youth cite an 8 percent lifetime prevalence of sexual assault, a 17 percent prevalence of physical assault, and a 39 percent prevalence of witnessing violence.⁴ Data from a review of research on children’s exposure to trauma indicate wide ranges for rates of PTSD, including: 20 percent to 63 percent in survivors of child maltreatment; 12 percent to 53 percent in the medically ill; and 5 percent to 95 percent in disaster survivors.⁵

Since it was established in 1991, the Office on Women’s Health (OWH) in the US Department of Health and Human Services (HHS) has promulgated policies and implemented programs designed to engage consumers, providers, and leaders to improve health status, eliminate health disparities, and reduce causes of illness, disability, and premature death for girls and women. OWH works to realize its vision that all women and girls achieve better health and improved wellbeing. Through staff in its national office and a network of Regional Women’s Health Coordinators (RWHCs) located in each of the 10 federal regions, OWH implements public health initiatives that are responsive to local, state, and regional needs and concerns about girl’s and women’s health. Among these public health initiatives, OWH has convened expert policy panels to inform its strategies, developed a TIC training curriculum, and tested and deployed it across regions with a plan to add sites and technical assistance (TA) support directed to a range of health and human services organizations in the 10 HHS regions. OWH requires a soundly-conceptualized process evaluation to provide ongoing evidence on barriers and facilitators to adoption and implementation TIC, so that systems of care (SOCs) can reach all targeted girls and women, regardless of location. OWH additionally requires evidence from an outcome evaluation to assess the impacts of TIC training and TA on the organization and delivery of services in participant programs. The contracted project team brings a strong and tested track record of incorporating theories of innovation and implementation into designs for evaluating program demonstration, training and TA initiatives focused on improving the effectiveness and appropriateness of services delivered to vulnerable children, women and families.

2. Purpose and Use of Information Collection

According to its recent report to Congress, OWH is in Phase 4 of the National Training Initiative on Trauma-Informed Care (TIC) for Community-Based Providers from Diverse Service Systems. The OWH goal for the training and TA is much more than simply increasing the knowledge of the individual participant. The goal is to achieve a high level of engagement and commitment through a relational approach to training, leading the participants to actions that will change the environment and operation of their programs and service delivery settings so that they are trauma-informed. An evaluation of such a goal goes beyond those typically implemented in testing knowledge gained, requiring an examination of the training and TA process and the outcomes produced in the form of impacts in the organization and delivery of care provided by the participants’ organizations or programs. As outlined, addressing trauma aligns with each of the OWH strategic priorities, and the investment in an evaluation of the TIC training and TA effort must be designed to produce actionable findings and recommendations to support the longer term efforts to build service delivery and systems capacity to meet the needs of girls and women.

Evaluating this seminal initiative will involve examination of the training and TA planning, development and administration processes, including participant satisfaction, as well as outcomes for individual participants, their organizations and their service recipients. OWH has contracted with a team of experts to assess aspects of the knowledge acquisition levels among individual participants, behavior change resulting from what was learned. The contractors will also examine measurable results, not only on participant behavior, but also on organizational plans and operations.

OWH proposes to collect information from participants in OWH’s National Training Initiative on Trauma-Informed Care (TIC) for Community-Based Providers from Diverse Service Systems for the purpose of assessing effectiveness of the training. The TIC Training and Technical Assistance effort has been carried out under a contract with Westat and Western Massachusetts Training Consortium, with these organizations possessing information about the curriculum development and implementation, the training phases and participants, and the products developed in the technical assistance effort. The design of the data collection is informed by the need to measure impact in alignment with the aims of the National Training Initiative (NTI). The three broad aims of this initiative are to:

• Educate service providers from diverse sectors to recognize the impact of trauma experience on their service recipients;

• Assist service providers to assess trauma, recognizing the impact of trauma on the health and wellbeing of women and girls; and

• Build the capacity of diverse providers to foster changes in the service delivery system to provide TIC practices to women, children, or families.

In designing the outcome evaluation to assess the success of the NTI in achieving these stated goals, the team also considered that there are two related levels involved in reaching those goals. These are: the individual participant level, where the knowledge and skills are gained, absorbed and used in his/her work; and the agency level where the training of individuals has an effect on changing the practice of the agency. These change processes have been described as the diffusion of a concept or practice that moves through stages in an individual of knowledge acquisition decision to act and transfer and implementation to their organization or agency of the knowledge and practice (Rogers, 1983). A number of factors enter into the successful transition through these stages, both factors that are contained within the trainee (demographics, prior knowledge and experience, setting from which they come) and those that are external to the individual (quality of curriculum and trainer, support of home agency, authority of participant to make change). Any plan has to include the collection of data on these internal and external variables factors for analysis of outcomes.

To successfully achieve its objectives, any training must:

• Increase the level of knowledge and understanding of the participants on the subject matter;

• Promote the utilization of that knowledge in the participants’ practice; and

• Result in positive changes to the organization or system in which the participants operate, including increased capacity and coordination of services.

These three levels of training effectiveness are both inter-related and cumulative.⁶ Participants must acquire the information or techniques offered in training before they can utilize them in their own practice; and, the ultimate goal of training groups of participants is to improve the way the organization or system in which they operate addresses this issue. For this reason, an evaluator cannot simply look at outcomes. The question of “What happened?” is central to answering the ultimate question “Was it successful?” If a training and TA program was not adequately implemented, a test of the effectiveness is not a fair one; that is, the training curriculum and TA protocol can be excellent, but not executed with fidelity.

The evaluation will assess the initiative’s success in achieving these stated goals, using a rigorous mixed-methods design that includes the collection and review of extant data sources, site visits, and the administration of a web survey to TIC training participants. The team’s multi-level approach to the evaluation will capture both knowledge gained and implementation impact achieved as a result of the TIC training and TA.

The evaluation includes both quantitative and qualitative data collection and analysis and will provide OWH a picture of who was involved in the training and TA, including individual participant characteristics, dynamics of the organizations they represent, and the composition of teams in which they participated for the training. Specifically, the project team is interested in changes that resulted at the individual participant and organizational level. The contractor’s mixed-methods approach includes several data collection activities and uses analytic tools designed to best fit the information gathered from a diverse set of sources. Using multiple data collection methods, the project team will review several types of data, including: administrative data on participants, their organizations, and the training and TA protocol they received; participant-written action plans; interview notes; program observations; and participant survey responses. These data will be gathered from a variety of sources, including: Westat, Western Massachusetts Training Consortium and participating NTI organizations; key informant interviews with TIC experts; selected on-site and virtual interviews with participating NTI organization representatives; and an online survey of all training and TA participants. Using these sources, the project team will evaluate:

• The level of knowledge gained by participants;

• The skills acquired by participants;

• Changes in the values and beliefs associated with the training and TA;

• The behavioral actions of participants taken as a result of the training and TA;

• The actions of each community organization’s staff team taken as a result of the training The changes in agency or institutional policy and/or practice as a result of the training and TA; and

• The impact of changes in the modification of the TA protocol (from Phase II to Phase III training sites).

The study will enhance the existing knowledge about factors contributing to effective TIC training, and how community-based providers can use TIC principles to enhance care delivery to their consumers. The analyses and findings will be used to further refine the TIC curriculum and training approach, and can help inform OWH and HHS in future policymaking efforts. Information collected will also help researchers and practitioners better understand the impact of adopting a trauma-informed approach on and the quality of care provided by community-based providers.

3. Use of Improved Information Technology and Burden Reduction

Data collection during the evaluative phase of the project will take place through an online survey and site visits.

Online Survey

The OWH contractor will deploy an online web survey to follow up with all training participants to assess, as noted earlier, their values and beliefs regarding trauma and the knowledge and skill acquired in the training and TA. The universe of training participants across all three phases of the training initiative (approximately 300 participants) will constitute the online survey target population. This survey was designed and tested for ease of use, readability, and brevity. The survey will take approximately 25-30 minutes of time. All participants who have been trained will constitute the online survey target population. Prior to accessing the survey, each participant will receive an orientation email introducing the project and providing a web address to which they can access the online survey.

The survey instrument will organize questions into three overarching assessment categories: values and beliefs, skills, and knowledge (for a full list of survey questions, see Attachment A – Participant Survey Scales). The survey will also gather brief information on certain characteristics of the participants (e.g., demographics, job responsibilities, experience, education and training). Programming and development of the analytic file for the web survey will be done by the contractor’s Client Technology group, which specializes in online survey creation, testing and execution. Survey data will be transformed into a common analytic file type (i.e. SAS/STATA) for analysis.

Site Visits

During the site visits, interviewers will take notes during interviews and will later summarize these notes into succinct site visit summaries employing a unified format for use in qualitative analysis. The project team will develop a preliminary NVivo "codebook" that defines overarching themes and curriculum components the team plans to discuss during site visits and team conference calls. Implementation data will be summarized by site and by groupings in narrative and, where appropriate, in simple descriptive tables. Specific comments may be coded in more than one theme and comments will be coded at the most specific theme. As visits are completed and notes are coded, two team members will review the content of all the themes to check for inconsistencies, redundancies and imprecision. All qualitative data will be coded using this theme structure.

For the purpose of the outcome evaluation, information from this phase may also be used to cluster training site characteristics and experiences to create contrasts for the purpose of understanding of what works for whom under what circumstances in regard to attitudes, knowledge and behavior at the individual level and organizational/environmental change at the organizational/community level. For a full description of site visit activities, see Attachment B – Site Visit Protocol.

4. Efforts to Identify Duplication and Use of Similar Information

The information collected through this project is not duplicative of another information collection. There have been no previous evaluations of OWH’s National Training Initiative on Trauma-Informed Care (TIC) for Community-Based Providers from Diverse Service Systems. This project is explicitly to measure the impact and effectiveness of the training.

5. Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this study.

6. Consequences of Collecting the Information / Less Frequent Collection

Site Visits

Data will be collected from site visit respondents one-time only.

Online Survey

Data will be collected from online survey respondents one-time only.

Note that project team anticipates that some of the training participants who will be sent the online survey will also be interviewed during the site visits. The data collected through the site visit interviews varies significantly, however, from the questions asked through the online survey. The online survey focuses on individual knowledge and skill acquisition, value and belief change. The site visit focuses on organization and practice change. Both the data gleaned from the site visits and the data collected through the online survey are essential in gauging the impact of the training.

There are no legal obstacles to reduce the burden.

7. Special Circumstances Relating to the Guidelines of CFR 1320.5

This request fully complies with the guidelines of 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice/Outside Consultation

A 60-day Notice was published in the Federal Register on Tuesday, October 28, 2014 (Vol. 79, No. 208, pp. 64194-64195No comments were received during the 60-day notice period.

9. Explanation of Any Payment or Gift to Respondents

Respondents will not receive any payment or gift for participating in this project.

10. Assurance of Confidentiality Provided to Respondents

Data will be kept private to the extent allowed by law.

Personal Identifiers

Personal identifiers will not be collected. During site visits, respondents will be assigned an interviewee ID. The online survey will automatically assign a separate respondent ID to be used to identify the respondent.

IRB Approval/Exemptions

The contractor’s IRB reviewed the evaluation plan and the site visit protocol (including the Informed Consent Form) for the OWH Trauma Informed Care: Cross-Site Evaluation of the Training and Technical Assistance Initiative and has determined that the evaluation is eligible for exemption under 45CFR46.101(b)(2):

(2) Research involving the use of educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures or observation of public behavior, unless: (i) information obtained is recorded in such a manner that human subjects can be identified, directly or through identifiers linked to the subjects; and (ii) any disclosure of the human subjects' responses outside the research could reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, or reputation.

Data Security

1. All data (including data collected through the Site Visits)

The OWH contractor shares the common goal of protecting the confidentiality, integrity and availability of all of The Office on Women’s Health’s data. The contractor has developed an Information System Security Plan (ISSP) and completed the Certification and Accreditation (C&A) process with several Federal agencies to receive authorization to operate a number of “low-” to “moderate-” risk category information systems. The contractor has developed their systems with a focus on security to meet the stringent requirements. The contractor complies with the Privacy Act of 1974, Health Insurance Portability and Accountability Act of 1996 (HIPAA), and the E-Government Act of 2002, including Title III: Federal Information Security Management Act (FISMA), which covers site security, security control documentation, access control, change management, incident response, and risk management. The contractor has also successfully complied with client requirements for a System of Records Notice (SORN) and a Privacy Impact Assessment (PIA).

The contractor has implemented full disk encryption (FIPS 140-2 compliant) software in our environment to protect the storage of data, as well as file transfer software (also FIPS 140-2 compliant) for the secure, encrypted transmission of sensitive data to and from clients and subcontractors. The contractor has also implemented anti-malware software in our environment and update definitions daily on each workstation. For boundary protection, the contractor has implemented Cisco ASA Firewalls.

The contractor has established an internal audit program to regularly review the success of our information security program. The contractor also retains a well-known external audit firm to conduct information technology audits. Lastly, the contractor’s Information Security regularly reviews the effectiveness of security controls.

Online Survey

The OWH contractor will be using FluidSurveys to manage the online survey data. FluidSurveys servers are located in data centers which provide biometric access controls, constant surveillance, redundant power feeds and generators, robust fire suppression, and carefully monitored climate control to protect the servers that store data. All accounts are password protected and all passwords are encrypted and never stored in clear text. Account logins also have brute-force login protection by preventing individuals/bots from attempting to guess a password too many times. FluidSurveys login (i.e. when a user logs in by typing their username/password) is protected using Secure Socket Layer Encryption (SSL). SSL will encrypt communications (256 bit) between the respondent's browser and our server. FluidSurveys servers are scanned regularly for threats and vulnerabilities by McAfee secure scan. The contractor’s surveys are section 508 compatible.

11. Justification for Sensitive Questions

Respondents will not be asked questions of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs to TTA Respondents

Burden for OWH TIC TTA participants engaged in the evaluation is estimated based on two core activities: participation in the site visits; and participation in the online survey. Estimation factors are detailed below and results are outlined in the following exhibits, Table 12A and Table 12B.

Site Visits

Approximately 144 staff will be selected to interview. One-on-one interviews with leadership and line staff will take between 30-34 minutes per interview.

Online Survey

Approximately 300 training participants will be sampled in order to preserve sample sizes at smaller units of analysis. The survey will take 25-30 minutes to complete.

There will be customary and usual business practices, such as introductory emails and phone calls to the site visit points of contact.

12A. Total Estimated Annualized Burden – Hours

Form Name	Type of Respondent	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours
Online Survey	Leadership and Line/Other Frontline Staff	300	1	25/60	125
Site Visits	Leadership and Line/Other Frontline Staff	144	1	40/60	96
Total					221

12B. Total Estimated Annualized Burden - Costs

Form Name	Type of Respondent (Form Name)	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Online Survey	Leadership and Line/Other Frontline Staff	125	$27.00	$3,397
Site Visit Protocol	Leadership and Line/Other Frontline Staff	96	$27.00	$2,592
Total		221		$5,989

Hourly wage rate estimated using http://www.bls.gov/oes/current/oes_nat.htm#19-0000, May 2014. National Occupational Employment and Wage Estimates for the United States. Estimates are calculated using the average of wages for Social and Community Service Managers ($32.56) and Community and Social Service Occupations (rounded down from $21.79). The above Hourly Wage Rate is the calculated average of the wages for Social and Community Service Managers and Community and Social Service Occupations (line staff).

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no costs to respondents other than their time, and no record-keeping is required.

14. Annualized Cost to the Federal Government

There are two elements of labor and cost burden to the Federal Government: those accrued by Federal Government Employees overseeing the OWH TIC TTA Evaluation; and those accrued by the Contractor, Abt Associates, in conducting the Evaluation.

Federal Employees

The OWH TIC TTA Evaluation COR estimated the annualized cost to the Federal Government for Federal Employee Oversight of the Evaluation. Estimation factors and results are detailed in the following exhibit, Table 14A.

14A. Total Estimated Annualized Burden to the Federal Government – Labor and Costs

Federal Employee Oversight
Period of Performance of Funding Request:	September 30, 2013 to September 29, 2015
Tasks	Labor Hours	Labor Costs	Total
Task 1: Project Management	30	$900	$900
Task 2: Design Study	200	$6,000	$6,000
Task 3: Development and Data Collection Testing	100	$3,000	$3,000
Task 4: OMB and IRB Clearance	40	$1,200	$1,200
Task 5: Data Collection	40	$1,200	$1,200
Task 6: Data Analysis	250	$7,500	$7,500
Task 7: Project Results Report	60	$1,800	$1,800
Total	720	$21,600	$21,600

Contracted Employees

Abt Associates estimated the annualized cost of the OWH TIC TTA Evaluation to the Federal Government for Abt staff. Burden estimates include Abt staff and subcontractor Labor Hours, Labor Costs, Operations, Direct, and Indirect Costs for Year 1 and Year 2. Estimation factors and results are detailed in the following exhibit, Table 14B.

14B. Total Estimated Annualized Burden to the Contractor Abt Associates – Labor and Costs

15. Explanation of Program Changes or Adjustments

This is a new data collection effort.

16. Plans for Tabulation and Publication and Project Time Schedule

Tabulation

As noted above, data will be collected through an online survey of participants and through site visits to participating programs/organizations. The individual survey results will be scored, aggregated, and tabulated for the final evaluation report, with quantitative presentation. The site survey data will be coded and analyzed for a qualitative presentation in the final evaluation report.

1. Process Evaluation Analysis

There are differences among the sites that participated in the OWH TTA initiative that must be accounted for in the evaluation. For example, OWH recognizes site variation in terms of the makeup of the trainee population, whether they were trained in person or remotely and whether they received TA in addition to the training curriculum. For this reason the process and implementation portion of the analysis will have both core data elements and site specific data elements. In addition the team will look across the 10 sites to see if there are common issues in implementing the training and/or in obtaining positive outcomes that can be used as guides to OWH in preparing for additional trainings.

The details of the site visit inquiry, including questions to be asked, observations to be made, and data to be requested, are contained in the Site Visit Protocol (see Appendix A).

Implementation data will be summarized by site and by groupings in narrative and, where appropriate, in simple descriptive tables. For example, the team can examine across sites recruitment and completion rates for participants, participant satisfaction, etc.

For the purpose of the outcome evaluation, information from this phase may also be used to cluster training site characteristics and experiences to create contrasts for the purpose of understanding of what works for whom under what circumstances in regard to attitudes, knowledge and behavior at the individual level and organizational/environmental change at the organizational/community level.

2. Outcome Evaluation Analysis

The project team anticipates at a minimum an 80 percent response rate for all new training sites and a 60 percent response rate for earlier training sites (given our expectation of staff turnover) to the online survey. It is our experience that a motivated, primarily professional respondent sample coupled with a brief and attractive survey format results in high rates of response.

1. Assessing Response Rates [online surveys for all participant trainees (n~300)]

The first stage of the analysis, particularly with the small numbers anticipated in each site, is an examination of response rates in the various sources of outcome data. These response rates will be further analyzed by possible sources of response variation (e.g., site, setting from which trainees come, mode of training administration). The project team will model “missingness⁷” using basic bivariate analytic techniques (e.g., chi square and t-tests) to determine what variables are associated with reduced response rates. The team will generate a report summarizing response rates, systematic variations and the potential effects of this nonresponse bias on the study results. The team will present this report to OWH in the initial analysis stages.

2. Analysis of Key Outcomes and Data Reduction (full sample)

There are multiple measures of each outcome in several domains (e.g., values and beliefs, skills, and knowledge). Basic descriptive analyses will be conducted first, followed by analyses designed to evaluate how these measures might be combined in indices, or unweighted or weighted scales. For example, factor/principal components analysis will be used to evaluate the degree to which multiple measures tend to “hang together.” In this case, weighted factor scores may be used to create single measures of each outcome. The final outcome indices/scales will be determined through the process of these analyses.

Publication

3. Draft Final Report Outline and Submission

Once data collection is complete, the contractor will submit a detailed final report outline to OWH for comment and feedback. The contractor will then prepare a draft report on the results of the project. In addition to the results, the report will include detailed background information on the objectives, scope, methodology, response rates, analysis, and data collection instruments used in the research. The report will also include a discussion of the implications of the findings for policy and practice, and recommendations for further research. An executive summary will be included as the beginning of the final report, and will serve as the foundation for brief reports and web postings. The report will be submitted to the OWH Task Order Officer (TOO) and the OWH OS Report Clearance Officer for review and approval, and the final version of the report will incorporate the feedback gathered through the OWH review process.

While the draft final report is under review, the project team will discuss to which agencies, organizations, and associations the team will offer brief reports to publish or post. For example, one target group may be the Women and Trauma Federal Partners’ Committee, collaborating on efforts to develop policies and services in support of women and girls who have been affected by exposure to trauma. In addition to HHS agencies, collaborating federal departments and agencies include the Departments of Labor, Justice, Defense, State, Education, Agriculture, Veterans Affairs, and Housing and Urban Development. The production of a list of candidates and final selections will be made in collaboration with the TOO.

4. Final Report Revision and Submission

After receiving comments from OWH, the contractor will revise the draft final OMB Supporting Statement and submit it to the OWH TOO. The revised OMB Supporting Statement will be responsive to feedback from the OWH TOO and OWH OS Report Clearance Officer. .

5. Journal Article Manuscript(s)

The contractor will use the material in the final report to draft one article suitable for publication in a peer-review journal. The OWH contractor, TOO, and program staff will collaborate to determine an appropriate journal or journals. The contractor will also collaborate to determine which project findings will serve as the topics of the journal article manuscript.

6. Presentations

The contractor will also design a PowerPoint presentation that provides background on the National Training Initiative on Trauma-Informed Care for Community-Based Providers from Diverse Service Systems, as well as the team’s evaluation methods, results, and their implications. The project team will collaborate with the TOO to determine the location and the audience for this presentation. Within one month of the project end, the team will also plan to offer a similar, in-person presentation at a national conference, to be determined through consultation with OWH staff. In addition, at least one webinar will be offered to allow broader audiences and those not able to attend conference or other in-person presentations to learn about the study, its findings, and its implications. An additional presentation will be offered as a webinar to be designed for the 10 study sites, to review the study findings and allow questions from research sites to be addressed by the project team or OWH staff.

Project Time Schedule

Below is a copy of the estimated time schedule.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

Site Visits

The OMB expiration date will be displayed on the top of the informed consent form (see Appendix C).

Online Survey

The OMB expiration date will be displayed on the entry screen for the online system (see Appendix D).

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

1 Felitti, V., Anda, R., Nordenberg, D. et al. (1998). Relationship of child abuse and household dysfunction to many of the leading cause of death in adults: The Adverse Childhood Experiences Study. American Journal of Preventive Medicine, 4, 245-58.

2 Copeland, W.E., Keeler, G., Angold, A., Costello, E.J. (2007). Traumatic events and posttraumatic stress in childhood. Archives of General Psychiatry, 64, 577-584.

3 Tolin, D.F. & Foa, E.B. (2006). Sex Differences in Trauma and Posttraumatic Stress Disorder: A Quantitative Review of 25 Years of Research. Psychological Bulletin, 132, 959–992.

4 Kilpatrick, D. G., & Saunders, B. E. (1999). Prevalence and consequences of child victimization: Results from the national survey of adolescents (Final Report No. 93-IJ-CX-0023). Charleston, SC: U.S. Department of Justice.

5 Gabbay, V., Oatis, M.D, Silva, R.R. & Hirsch, G. (2004). Epidemiological aspects of PTSD in children and adolescents. In Raul R. Silva (Ed.), Posttraumatic stress disorder in children and adolescents: Handbook. (1-17). New York: Norton.

6 Watkins, R., Leigh, D., Platt, W., & Kaufman, R. (1998). Needs assessment—a digest, review, and comparison of needs assessment literature. Performance Improvement, 37(7), 40-53.

7 The term “missingness” refers to the manner in which data are missing from a sample of a population.