JCAMP 2_SSB_08-31-23_fnl

Judicial, Court, and Attorney Measures of Performance (JCAMP): Feedback and Implementation

OMB Information Collection Request

0970 - NEW

Supporting Statement Part B –

Statistical Methods

September 2023

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

1. Respondent Universe and Sampling Methods

The target populations for the Judicial, Court, and Attorney Measures of Performance (JCAMP) Feedback and Implementation data collection efforts include two different populations: one population for the JCAMP Feedback Survey, and another for the JCAMP data collection instruments.

The first population, for the JCAMP Feedback Survey, include all members of currently engaged state and tribal court teams implementing the JCAMP measures and receiving support from the JCAMP team under a current technical assistance (TA) grant (‘implementation teams’). Currently, 12 teams with 5–18 members each are implementing JCAMP. Team members include Court Improvement Program (CIP)¹ staff, judges, attorneys, Court Appointed Special Advocates, people with lived experience in child welfare, and other legal professionals. The JCAMP Feedback Survey is a census survey of all members of active JCAMP implementation teams, which is estimated to be around 125 members (although this is fluid as teams adjust members based on data needs). A census survey was selected because of the relatively small number of JCAMP implementation teams and the desire to capture the full range of variation in perspectives about implementation support that may be missed if a sample were used. Other survey administrations of similar length with CIP Administrators conducted by the study team have yielded 74 percent response rates. Given our announcement during a CIP All Call (see section B2 below), an endorsement from Scott Trowbridge (Children’s Bureau Child Welfare Program Specialist and federal program officer for the CIPs), current and historic working relationships with our study team, and extensive planned follow-up efforts (see below for details) we estimate a response rate of 80 percent for the JCAMP Feedback Survey. We expect to administer the survey one time to 100 individuals at the end of our implementation support each year.

The second population includes persons involved at the case level in child welfare. The JCAMP data collection measures include 7 surveys and 3 focus group guides that can be used to collect primary data from court participants in states and tribes to inform their selected JCAMP performance measures. Each of these instruments will be administered according to a process and schedule decided by each jurisdiction. Data collection will take place no more than once per year for each instrument in each jurisdiction. Estimates are based on conversations with the implementation sites about intentions and capacities and prior response rates to similar collections in the states/tribe.

2. Procedures for the Collection of Information

JCAMP Feedback Survey (Instrument 1)

At the end of the first year of JCAMP implementation, up to 100 members of JCAMP implementation teams will receive an email invitation (Appendix A) from the project team to complete the JCAMP Feedback Survey. To ensure quality control, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per respondent is submitted. This will also allow the study team to easily track response rates. At the end of week 2, the study team will individually email all non-responders (see Appendix B). At the end of week 4 the study team will call all non-responders and offer an additional week to respond, if needed (see Appendix C). If we cannot reach non-responders by phone, we will resend the email message with the deadline extended (see Appendix B). Currently the plan is for the survey will be open for 8 weeks, unless the respondent requests additional time to complete the survey. If needed, this timeline may be adjusted to ensure adequate responses are received.

JCAMP Data Collection Instruments

The JCAMP data collection instruments include the following:

• Parent Experience Survey (Instrument 2)

• Parent Experience Question Bank (Instrument 3)

• Parent Focus Group Guide (Instrument 4)

• Youth Post-Hearing Short Survey (Instrument 5)

• Youth Experience Survey (Instrument 6)

• Youth Court Experience Question Bank (Instrument 7)

• Youth Focus Group Guide (Instrument 8)

• Caregiver Experience Survey (Instrument 9)

• Stakeholder Survey (Instrument 10)

• Stakeholder Focus Group Guide (Instrument 11)

Specific procedures for administration of each instrument will be determined with individual jurisdictions implementing JCAMP. The practices described below for surveys and focus groups will be recommended and supported by the JCAMP study team. Examples of recommended email invitations, follow-up emails for non-responders, and phone scripts for non-responders for all surveys and focus groups are provided as appendices.

Surveys (Instruments 2, 3, 5, 6, 7, 9, and 10)

Respondents will receive an email invitation from the study team to complete the survey. To ensure quality control, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per respondent is submitted. This will also allow the study team to easily track response rates. At the end of week 2, the study team will individually email all non-responders. At the end of week 4 the study team will call all non-responders and offer an additional week to respond, if needed. If we cannot reach non-responders by phone, we will resend the email message with the deadline extended. The survey will be open for 8 weeks, unless the respondent requests additional time to complete the survey.

Focus Groups (Instruments 4, 8, and 11)

Participating jurisdictions will identify the stakeholders that should participate in focus groups. Focus groups may be conducted in-person or via the web using video software (e.g., Zoom, Microsoft Teams). Selected participants will be emailed a focus group request that asks them to respond within 2 weeks. At the end of week 2, the study team will call each invited respondent who has not yet responded to ask whether they want to participate. If we cannot reach the respondent by phone, we will resend the invitation email with the request to respond extended 1 week.

Focus groups will be scheduled at a mutually convenient time. All facilitators will complete training with the Co-Principal Investigators. One member of the study team will conduct the focus group following the guide and another will take notes. Permission will be requested to record the focus group to make sure that all responses are accurately captured. After each focus group, the notetaker will clean and save the notes, referring to the focus group recording as needed.

3. Methods to Maximize Response Rates and Deal with Nonresponse

As described above, we are planning extensive follow-up activities to ensure that respondents have every chance to participate. A list of the reasons why respondents decline to participate in the surveys will be maintained and regularly reviewed during data collection to determine barriers to participation. Since our data collection requires only a computer with internet connection, the biggest barrier to participation will likely be lack of time.

Data collected will not be used to make population estimates either for internal use or dissemination. For surveys, we will produce descriptive statistics to understand experiences with JCAMP implementation and to calculate the JCAMP performance measures. For focus groups, we will qualitatively analyze transcripts to identify themes and inform site-specific performance measurement data.

4. Test of Procedures or Methods to be Undertaken

The JCAMP Feedback Survey was reviewed by our Core Team, including researchers, attorneys, TA providers, and people with lived experience. The study team made needed revisions based on this feedback. The other instruments were reviewed by members of the JCAMP development team including people with lived experience, judges, attorneys, and other legal professionals.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for this study is being collected by Dr. Alicia Summers, Dr. Sophia Gatowski, and James Bell Associates on behalf of ACF. Co-Investigators Dr. Alicia Summers (775-686-8545) and Dr. Sophia Gatowski (604-807-4617) and team member Anne Fromknecht (703-247-2631) led development of the study design and data collection protocols and will oversee collection and data analysis.

The agency responsible for receiving and approving contract deliverables is:

Children’s Bureau (CB),

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services

The Federal project officer for this project is Scott Trowbridge. JD.

Attachments

Instrument 1_JCAMP Feedback Survey

Instrument 2_Parent Experience Survey

Instrument 3_ Parent Court Experience Question Bank

Instrument 4_Parent Experience Focus Group Guide

Instrument 5_Youth Post-Hearing Short Survey

Instrument 6_Youth Experience Survey

Instrument 7 Youth Court Experience Question Bank

Instrument 8_Youth Focus Group Guide

Instrument 9_Caregiver Experience Survey

Instrument 10_Stakeholder Survey

Instrument 11_Stakeholder Focus Group Guide

Appendix A_JCAMP Feedback Survey Invite

Appendix B_JCAMP Feedback Survey Non-Responder Email

Appendix C_JCAMP Feedback Survey Non-Responder Phone Script

Appendix D_Parent Experience Survey Email Invite

Appendix E_ Parent Experience Survey Non-Responder Email

Appendix F_ Parent Experience Survey Non-Responder Phone Script

Appendix G_Parent Focus Group Email Invite

Appendix H_Parent Focus Group Non-Responder Email

Appendix I_Parent Focus Group Non-Responder Phone Script

Appendix J_ Youth Survey Email Invite

Appendix K_Youth Survey Non-Responder Email

Appendix L_Youth Survey Non-Responder Phone Script

Appendix M_ Youth Focus Group Email Invite

Appendix N_ Youth Focus Group Non-Responder Email

Appendix O_Youth Focus Group Non-Responder Phone Script

Appendix P_Caregiver Survey Email Invite

Appendix Q_Caregiver Survey Non-Responder Email Script

Appendix R_Caregiver Survey Non-Responder Phone Script

Appendix S_Stakeholder Survey Email Invite

Appendix T_Stakeholder Survey Non-Responder Email

Appendix U_Stakeholder Survey Non-Responder Phone Script

Appendix V_Stakeholder Focus Group Email Invite

Appendix W_Stakeholder Focus Group Non-Responder Email

Appendix X_Stakeholder Focus Group Non-Responder Phone Script

1 CIP grants are awarded to the highest court systems in all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands.