Attachment A1 PII Research Matrix

ATTACHMENT A1: PII RESEARCH MATRIX FOR IMPLEMENTATION, KANSAS, WASHOE, ILLINOIS, RISE, COST, AND ADMINISTRATIVE DATA STUDIES

RESEARCH MATRIX FOR IMPLEMENTATION, KANSAS, WASHOE, ILLINOIS, RISE, COST, AND ADMINISTRATIVE DATA STUDIES
Instrument	Description	Frequency	Target Population	Annual Number of Respondents	Annual Burden Hours	Research Question
Implementation Study:
Survey of Organization/ System Readiness	Investigates the extent to which PII grantees are positioned through experience and organizational culture to make adjustments that support PII’s two main goals: To improve permanency outcomes for children who have the most serious barriers to permanency through the development of evidence-based, replicable child welfare service interventions To conduct rigorous evaluations to produce high-quality, credible evidence that the PII interventions are effective in decreasing the duration of stays in foster care and potentially replicable by future adopters. The survey will gauge whether grantee organizations support the PII goals by exploring individual respondents’ willingness to use and interest in using evidence-based interventions to address barriers to permanence for children and youth most at risk of LTFC. In addition, the questionnaire will explore respondents’ perceptions of organizational climate as it relates to readiness to change and individual and organizational interest in supporting rigorous evaluation.	Administered once to participants in all grantee sites.	Staff and partners employed by direct service providers for the PII initiative	60	18	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Implementa- tion Drivers Web Survey	Will track processes that sites use to implement interventions. While the intervention and size/scope of implementation varies across sites, there are implementation processes and strategies that are common across projects. Grounded in the National Implementation Research Network framework of organizational and systemic factors that facilitate successful implementation, these measures track progress building implementation infrastructure capacity. The driver assessment survey comprises eight sections that correspond to implementation driver domains that have been the focus of training and technical assistance provided to grantees to strengthen implementation capacity. These domains are: practitioner selection, training, supervision/coaching, performance assessment, decision support data systems, facilitative administration, systems intervention, and leadership.	Completed for the first time 6 months after the first clients to receive the full intervention are served by the first trained workers. Administered annually thereafter to determine the progress made toward the development of the drivers.	Grantees, partners, and stakeholders who are a part of the PII leadership and implementation structure	150	240	What organizational and systemic factors are associated with reaching and sustaining full implementation?
Grantee Case Study Protocol	The protocol allows for a qualitative examination of the key implementation activities; interim products and milestone events that occur during exploration, installation, and initial implementation; and the stages of implementation that set a foundation for achievement of full implementation. In addition, the implementation case study allows for examination of the context in which a PII intervention is being developed and implemented: How do external factors and broad systems issues influence implementation progress and success? In addition, the case study will explore the resources and organizational structure that support the implementation of PII. In essence, the implementation case study allows PII-ET to document the individualized story of each grantee’s implementation experience including enablers and barriers encountered during the implementation process.	Completed for years 1 and 2 and then updated on a yearly basis.	The Evaluation Team completes the case study using existing documentation. Once completed, the ET meets with grantee staff to address missing information and to ensure the case study accurately reflects site specific PII experiences.	30	240	What organizational and systemic factors are associated with reaching and sustaining full implementation? What costs were incurred in reaching and sustaining full implementation?
Fidelity Data (Implementation Quotient Tracker)	A reporting format used to collect data on the percent of caseworkers/practitioners that are implementing the prescribed intervention as intended and with model integrity. This tool captures the primary dependent variable: the proportion of caseworkers/practitioners at a given point in time that are meeting performance assessment criteria. For each practitioner allocated to the program the following will be answered: Is the practitioner in position; Has the practitioner been trained; Has the practitioner being coached as intended; Did the worker meet the performance assessment standards for the most recent assessment; Has the practitioner met the performance assessment standards over the last 4 quarters.	Administered for the first time 6 months after summative evaluation begins, then quarterly thereafter. Enables quantification of a site’s overall progress towards full implementation of the PII initiative.	Site’s evaluation liaison compiles the information for each allocated practitioner using data already available for site-specific evaluation and/or implementation requirements.	2	24	To what extent have grantees reached and sustained full implementation? How many calendar months were needed to reach and sustain full implementation?
Kansas Evaluation:
Family Assessment Battery	(1) Families are videotaped and experts rate them on dimensions such as problem solving, planning a fun activity, teaching, monitoring, and parenting processes; (2) parent-informant items assess social skills, problem behaviors, and academic competence of children and youth.	Completed three times: baseline for all cases, again when parents have completed the PMTO curriculum (6 months in most cases) or at 6 months for control cases, and finally at 12 months	Children age 3-16 with serious emotional disturbance and their parents	300 families	930 (including completion of Caregiver Initial Info Form	Do children age 3-16 in foster care who meet criteria for SED achieve more timely and stable permanence if their families receive early, intensive home-based parent management training compared to children in foster care whose families receive usual services?
North Carolina Family Assessment Scale (NCFAS)	Used by caseworkers to assess and rate family functioning (problems and strengths) in the domains of Environment, Parental Capabilities, Family Interactions, Family Safety, Child Well-being, Caregiver/Child Ambivalence, Social and Community Life, Self-Sufficiency, Family Health, and Readiness for Reunification. It is used at intake and again at case closure and the change score (the difference between the Intake and Closure ratings) indicates the amount of change achieved during the service period.			45 caseworkers respond for an average of 7 study cases each	945
Child and Adolescent Functional Assessment Scale (CAFAS)/ Preschool and Early Childhood Functional Assessment Scale (PECFAS)	Used to assess the day-to-day functioning (impairments and strengths) of children ages 5-19 in the domains of School/Work Role Performance, Home Role Performance, Community Role Performance, Behavior Toward Others, Moods/Emotions, Self-Harmful Behavior, Substance Use, Thinking, and Risk Behaviors. The PECFAS is a version used for younger children. The assessor completes the scale based on information collected during a typical clinical assessment.
Washoe Evaluation:
Overarching research question: What caregiver and child characteristics are associated with length of time in foster care?
Family Assessment Battery	Rates a child's problem behaviors and competencies; reports stresses related to parenting; reports measures of mental health symptoms in various domains; assesses parenting and childrearing attitudes of parents; assesses whether parents are at risk of child abuse or neglect; assesses caregiver’s readiness to accept change; assesses resiliency on eight subscales; examines how respondent’s social relationships provide social support; and provides indication of how stable a caregiver’s housing situation is. At baseline only (not at 6-month follow-up), assesses a caregiver’s history of abuse and trauma in childhood. At 6-month follow-up only, assesses the strength of the helping relationship between social workers and their clients.	Baseline and at 6-month intervals until the case is closed. All cases have an exit assessment at the time of case closure.	Two target populations: (1) new cases involving children aged 17 ½ or younger coming into the system, who are deemed unsafe, are living with a caregiver, and at risk of foster care placement; and (2) families with children who have been in foster care at least 12 months and who have one or more of the identified risk characteristics for long term foster care at time of placement (i.e., parental substance abuse, homelessness/inadequate housing, single parent households, or parental incarceration), a goal of adoption or guardianship, and an available caregiver.	175	525	Are there changes in the following over time, and are changes different for children whose caregivers receive the SAFE-FC intervention compared to those who receive business as usual? Caregiver parenting stress Caregiver mental health Caregiver attitudes about child-rearing Caregiver readiness to accept change Caregiver resiliency Caregiver housing stability. Does the relationship between a caregiver and their caseworker change over time and how does that relationship relate to permanency for the child? Are caregivers’ histories of abuse and trauma associated with children’s length of time in foster care?
Illinois DCFS Evaluation:
Youth Interview Battery	Abbreviated Dysregulation Inventory (ADI): Designed to assess three areas: (a) Emotional/Affective Dysregulation, (b) Behavioral Dysregulation, (c) Cognitive Dysregulation Youth Social Support Instrument (YSS): Designed to assess perceived social support received from familial adults, non-parental, and a peer Youth Emotional Permanency Instrument (YEP): Designed to examine the quality of the youth’s relationships with adults Parent Contact with Youth Instrument (PCY): Designed to assess the youth’s contact with his or her mother and father Trauma Symptom Checklist for Children (TSCC): Designed to assess distress and related symptoms after an acute or chronic trauma including Anxiety, Depression, Anger, Posttraumatic Stress, and Dissociation	Completed twice: baseline and then 6 months after	Foster youth ages 11-16 who are placed in traditional, relative, and specialized foster homes throughout the state of Illinois who, upon reaching the 2-year anniversary of entering foster care, are experiencing mental health symptoms and/or have had at least one placement change	864	461	Do foster youth ages 11-16 placed in traditional, relative, and specialized foster homes throughout the state who, upon reaching the 2-year anniversary of entering care, are experiencing mental health symptoms and/or have had at least one placement change, experience increased permanency rates within 3 years of entry if they receive TARGET services compared with similar youth who received services as usual?
Biological Parent Interview Battery	Abbreviated Dysregulation Inventory (ADI): Designed to assess three areas: (a) Emotional/Affective Dysregulation, (b) Behavioral Dysregulation, (c) Cognitive Dysregulation Social Provisions Scale (SPS): Designed to examine how respondent’s social relationships provide social support. Includes six constructs: (a) Attachment, (b) Social Integration, (c) Reassurance of Worth, (d) Reliable Alliance, (e) Guidance, and (f) Opportunity for Nurturance		Biological parents of the target foster youth, who fit the requirement that they be a viable resource for reunification	518	104
Foster Parent Interview Battery	Abbreviated Dysregulation Inventory (ADI)-Parent Version: Designed to assess three areas: (a) Emotional/Affective Dysregulation, (b) Behavioral Dysregulation, (c) Cognitive Dysregulation. The parent version, which is identical to the original instrument, instructs the parent to complete the instrument using the child as a reference. Parenting Practices Chicago Survey-Parent Version (PPCS): Designed to assess three constructs: Positive Parenting, Discipline Effectiveness, Discipline Avoidance, and Monitoring Social Provisions Scale (SPS): Designed to examine how respondent’s social relationships provide social support. Includes six constructs: (a) Attachment, (b) Social Integration, (c) Reassurance of Worth, (d) Reliable Alliance, (e) Guidance, and (f) Opportunity for Nurturance.		Foster parents of the target youth	864	461
RISE Grantee Impact Study:
Youth Interview	Self-Acceptance and Comfort (LGBTQ Status): Designed for RISE; examines youth’s personal comfort level with their sexual orientation, gender identity or their questioning of either sexual orientation or gender identity. Supporting/Rejecting Perceptions Survey: Designed for RISE; examines youth’s perceptions of how much their LGBTQ identity is supported or rejected by foster family, friends, persons in their school community, coworkers, neighbors, caseworkers, ministers, and anyone else the youth specifies. Permanent Connections Inventory: Designed for RISE; examines youth’s ability to identify adults in their lives that are committed to providing ongoing support and their perceptions of how the named adults support them. Debriefing Protocol: Designed for RISE; examines youth’s understanding of the sexual orientation and gender identity terminology included in the instruments.	Completed twice: baseline and 1 year later	LGBTQ and gender non-conforming youth ages 11-19 in out-of-home placements in Los Angeles County	22 youth: half treatment and half control	57	Do LGBTQ and gender-nonconforming children and youth in foster care achieve more timely and stable permanence if they and their families receive intensive wraparound services delivered by CCTs compared with LGBTQ children who receive usual services? What is the impact of RISE’s CCT component on proximal outcomes, including durable connections, enhanced well-being, and improved emotional permanency?
Youth Qualitative Interview	Qualitative Interview: Designed for RISE; examines youth’s perceptions of their experience receiving services from the RISE program and their foster care agency and explores the youth’s comfort in discussing sensitive topics with his or her caseworker, and their perceptions about the sensitivity of their foster care agency in working with LGBTQ people	Completed once: approximately 6 months after initiation of services	LGBTQ and gender non-conforming youth ages 11-19, in out-of-home placements in Los Angeles County	22 youth: half treatment and half control	26
CCT Facilitators administer to children ages 5-10	Permanent Connections Inventory (Child Version): Designed for RISE, identifies adults in the children’s lives that are committed to providing ongoing support and their perceptions of how the named adults support them.	Completed twice: baseline and 1 year later	LGBTQ and gender non-conforming children ages 5-10 in out-of-home placements in Los Angeles County, enrolled in CCT	2	2
CCT Facilitators complete	Emotional Permanency Instrument: Designed for RISE, examines the connections the youth has made with supportive adults who may provide permanency.	Completed twice: baseline and 1 year later	LGBTQ and gender non-conforming children and youth ages 5-19, in out-of-home placements in Los Angeles County, enrolled in CCT	2	8
CCT Facilitators complete	Child and Adolescent Functional Assessment Scale: Standardized and copyrighted instrument; assesses youth's day-to-day functioning across critical life subscales and determines whether a youth's functioning improves over time.	Completed twice: baseline and 1 year later	LGBTQ and gender non-conforming children and youth ages 5-19 in out-of-home placements in Los Angeles County, enrolled in CCT	2	4
Permanency Resource Interview	Permanent Connections Inventory (Adult Version): Designed for RISE; examines whether the permanency resource is committed to providing ongoing support, their experience in caring for youth, foster youth, and LGBTQ youth, and their comfort level with LGBTQ persons. Emotional Permanency Instrument: Designed for RISE, examines the connections the permanency resource has made with the youth. Supporting/Rejecting Attitudes Scale (MHI): Designed for RISE, examines the permanency resources’ attitudes towards, and level or acceptance or rejection of LGBTQ youth and adults, and examines characteristics that may influence support or rejection such as religiosity, exposure to LGBTQ persons in the media, and personal relationships with LGBTQ persons. Supporting/Rejecting Attitudes- Genderism and Transphobia Scale: Designed for RISE, examines the permanency resources’ attitudes towards gender and gender roles, and level or acceptance or rejection of persons that do and do not fit typical masculine and feminine archetypes.	Completed twice: baseline and 1 year later	Permanency Resources for LGBTQ and gender non-conforming youth ages 11-19, in out-of-home placements in Los Angeles County, enrolled in CCT	11	22
Current Caregiver Interview	Supporting/Rejecting Attitudes Scale (MHI): Designed for RISE, examines caregivers’ attitudes toward and level of acceptance or rejection of LGBTQ youth and adults, and examines characteristics that may influence support or rejection such as religiosity, exposure to LGBTQ persons in the media, and personal relationships with LGBTQ persons. Supporting/Rejecting Attitudes- Genderism and Transphobia Scale: Designed for RISE, examines the caregivers’ attitudes towards gender and gender roles, and level or acceptance or rejection of persons that do and do not fit typical masculine and feminine archetypes.	Completed twice: baseline and 1 year later	Current Caregivers for LGBTQ and gender non-conforming youth ages 11-19, in out-of-home placements in Los Angeles County, enrolled in CCT	11	13
Staff Pretest/Posttest	RISE ORB Staff Pre/Posttest: Designed for RISE and intended to measure change in knowledge after receiving ORB training. This instrument is a revision of the instrument approved by OMB in August 2013.	Completed twice: immediately before (pretest) and immediately after (posttest) ORB training	DCFS and private agency caseworkers	157	78*	For staff in DCFS offices and private agencies, is receiving the ORB training associated with greater competence in addressing issues of sexual orientation and gender nonconformity compared to their competence before receiving the ORB intervention?
Cost Study:
Cost Study Consent Form and Preparation Table	Designed for the PII Cost Study. As preparation for the focus group, participants review and comment on a preliminary list of project casework, supervisory, and management and administration activities.	Once	Caseworkers and supervisors with at least 60 days experience in PII services	9 caseworkers and supervisors	14	What is the cost-per-case of providing PII intervention services? What accounts for intra-intervention variation among cases? What is the cost-per-case of providing intervention services for different child and family subgroups? What accounts for the variations among child and family subgroups? What is the number of incidents and amount of person-time expended on each type of PII project activity, including casework, supervision, service delivery management, and project administration? What accounts for the variation in the mix of activities (types, incidence, and person-time expended)? Does cost-per-case differ across PII projects? If so, what accounts for the variation? What is the relationship between costs and outcome effect sizes for each PII project and across PII projects?
Cost Study Focus Group Guide	Designed for the PII Cost Study. The formal meeting involves caseworkers and supervisors breaking into small groups to discuss and come to consensus on names and definitions of individual and group activities and estimates of person-time for each activity. Participants will review weekly casework and supervision activity logs. In the post-meeting, participants complete a draft version of a weekly activity log and participate in a debrief call.	Once			36
Trial Administration of Activity Logs	Designed for the PII Cost Study. Caseworkers and supervisors will conduct a trial administration of the weekly case work and supervision activity logs developed in the focus group.	Once			14
Weekly Case Worker Activity Log	Designed for the PII Cost Study. For each case, caseworkers document via a web-based survey the PII project activities that were conducted during a specific week and the estimated total personnel time devoted to PII Project activities.	52	Caseworkers	123 caseworkers	2558
Weekly Supervisor Activity Log	Designed for the PII Cost Study. Supervisors document via a web-based survey PII project activities that were conducted during a specific week and the estimated personnel time devoted to PII Project activities.	52	Supervisors	39 supervisors	811
Monthly Management and Administration Activity Log	Designed for the PII Cost Study. Managers and administrators document via a web-based survey PII project activities that were conducted during a specific month and the estimated personnel time devoted to PII Project activities.	12	Managers and administrators	30 managers and administrators	180
Cross-site Administrative Data Study:
AFCARS	Adoption and Foster Care Analysis and Reporting System	4	Children in state foster care during 12-month reporting period	3 (with added fields), 2 (with no added fields)	12	Do the grantee interventions improve permanency outcomes (e.g., increase rates of permanency, or decrease time to permanency) for children in the foster-care system?
NCANDS	National Child Abuse and Neglect Data System		Children in alleged maltreatment cases disposed during the 12-month reporting period
SACWIS	Statewide Automated Child Welfare Information System		Children alleged maltreated or in state foster care

2