Judicial, Court, and Attorney Measures of Performance
OMB Information Collection Request
0970 - NEW
Supporting Statement Part B –
Statistical Methods
April 2022
Submitted By:
Children’s Bureau
Administration for Children and Families
U.S. Department of Health and Human Services
Respondent Universe and Sampling Methods
Court improvement program (CIP) grants are awarded to the highest court systems in all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. The target population for our study is the CIP Administrators and their data staff from all 53 CIPs.
The Judicial, Court, and Attorney Measures of Performance (JCAMP) CIP Data Capacity Survey is a census survey of all 53 CIPs. A census survey was selected because of the relatively small number of CIPs and the desire to capture the full range of variation in data capacity across the country that may be missed if a sample were used. Other survey administrations of similar length with CIP Administrators conducted by the study team have yielded 74 percent response rates. Given our announcement during a CIP All Call (see section B2 below), an endorsement from Scott Trowbridge (Children’s Bureau Child Welfare Program Specialist and federal program officer for the CIPs), current and historic working relationships with our study team, and extensive follow-up efforts we estimate a response rate of 80 percent for the JCAMP CIP Data Capacity Survey. We expect to administer the survey to 106 individuals (2 respondents per CIP).
The JCAMP Pilot Site Debrief Form will only be administered to up to 6 CIPs and court staff who participate in piloting the performance measures. The responses are not designed to produce statistically generalizable findings. We expect to administer the survey to 24 individuals.
Procedures for the Collection of Information
JCAMP CIP Data Capacity Survey
Each month, all CIP Administrators attend a mandatory call (“All Call”) hosted by the Capacity Building Center for Courts (CBCC). During one of these calls Scott Trowbridge (Children’s Bureau Child Welfare Program Specialist and federal program officer for the CIPs) will let the CIP Administrators know about the survey by reading the recruitment script (see Appendix A). The script will notify Administrators that they will receive an email invitation (see Appendix B) from the project team to participate in the JCAMP CIP Data Capacity Survey. To ensure quality control, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per respondent is submitted. This will also allow the study team to easily track response rates. A letter of support from Scott Trowbridge, Children’s Bureau Child Welfare Program Specialist, encouraging participation will be attached to the email invitation (see Appendix C). Mr. Trowbridge is the federal program officer for the CIP and is well known and respected by the CIP community.
At the end of week 2, the study team will individually email all non-responders (see Appendix D). At the end of week 4 the study team will call all non-responders and offer an additional week to respond, if needed (see Appendix E). If we cannot reach non-responders by phone, we will resend the email message with the deadline extended (see Appendix D). The survey will be open for 4 weeks, unless the respondent requests additional time to complete the survey.
JCAMP Pilot Debrief Form
Following participation in the pilot process, up to 24 respondents will receive an email invitation (Appendix F) from the project team to complete the JCAMP Pilot Debrief Form. To ensure quality control, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per respondent is submitted. This will also allow the study team to easily track response rates. At the end of week 2, the study team will individually email all non-responders (see Appendix G). At the end of week 4 the study team will call all non-responders and offer an additional week to respond, if needed (see Appendix H). If we cannot reach non-responders by phone, we will resend the email message with the deadline extended (see Appendix G). The survey will be open for 4 weeks, unless the respondent requests additional time to complete the survey.
Methods to Maximize Response Rates and Deal with Nonresponse
As described above, we are planning extensive follow-up activities to ensure that respondents have every chance to participate. A list of the reasons why respondents decline to participate in the surveys will be maintained and regularly reviewed during data collection to determine barriers to participation. Since our data collection requires only a computer with internet connection, the biggest barrier to participation will likely be lack of time, particularly given the challenges presented by the COVID-19 pandemic. To mitigate this barrier we will consider extending data collection and allow the surveys to be completed by a CIP staff member designated by the CIP Administrator.
Data collected will not be used to make population estimates either for internal use or dissemination. For the JCAMP CIP Data Capacity Survey, we will produce descriptive statistics to understand the variation in data capacity to inform development of the performance measure implementation guide. Data from the JCAMP Pilot Debrief Form will be used to inform the final performance measures document, implementation guide, and future technical assistance.
Test of Procedures or Methods to be Undertaken
Both instruments were reviewed by a former CIP Administrator; Christine Kiesel. Ms. Kiesel reviewed written copies of the surveys and provided written feedback regarding readability, time to complete the survey, and other thoughts about how to improve the survey. The study team made needed revisions based on this feedback.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The information for this study is being collected by Dr. Alicia Summers, Dr. Sophia Gatowski, and James Bell Associates on behalf of ACF. Co-Investigators Dr. Alicia Summers (775-686-8545) and Dr. Sophia Gatowski (604-807-4617) and team member Anne Fromknecht (703-247-2631) led development of the study design and data collection protocols and will oversee collection and data analysis.
The agency responsible for receiving and approving contract deliverables is:
Children’s Bureau (CB),
Administration for Children and Families (ACF)
U.S. Department of Health and Human Services
The Federal project officer for this project is Scott Trowbridge.
Attachments
Instrument 1_CIP Data Capacity Survey
Instrument 2_Pilot Debrief Form
Appendix A_All Call Script
Appendix B_Data Capacity Survey, Email Invite
Appendix C_Scott Trowbridge Letter
Appendix D_Data Capacity Survey, Non-Responder Email
Appendix E_Data Capacity Survey, Non-Responder Phone Script
Appendix F_Pilot Debrief, Email Invite
Appendix G_Pilot Debrief, Non-Responder Email
Appendix H_Pilot Debrief, Non-Responder Phone Script
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Anne Fromknecht |
| File Modified | 0000-00-00 |
| File Created | 2022-04-05 |