U.S. Department of Health and Human Services Administration for Children and Families |
Procedural Justice Informed Alternatives to Contempt Demonstration (PJAC) |
Office of Child Support Enforcement (OCSE)
|
New Collection |
330 C Street, SW Fifth Floor Washington, DC 20201 |
OMB No. 0970-NEW
|
Project Officer: Elaine Sorensen |
Supporting Statement for Implementation, Cost, and Impact Studies |
|
Part B: Statistical Methods |
|
July 2017 |
Respondent Universe and Sampling Methods
In September 2016, the Office of Child Support Enforcement (OCSE) within the Administration for Children and Families (ACF) issued grants to six state child support agencies to provide procedural justice-informed services to noncustodial parents who are far enough behind in their child support payments that they are facing contempt proceedings. The overall objective of the PJAC evaluation is to document and evaluate the effectiveness of the approaches taken by these six PJAC grantees. All six grantees will be included in the evaluation’s three key components: (1) the implementation study, (2) the impact study, and (3) the benefit-cost study.
In this Information Collection Request (ICR), clearance is sought for five instruments. The sampling approaches for these five instruments are:
Staff data entry for random assignment. The random assignment tool, embedded in the study MIS, will be used by program staff to conduct random assignment for all noncustodial parents enrolled into the PJAC evaluation.
Study MIS to track program participation. The study MIS will be used by program staff to document services received by all PJAC program participants during the grant period.
Staff interview topic guide. Interviews will be conducted with child support staff at the grantee agencies, as well as staff at their community partner agencies that will be providing additional supportive services by referral. Respondents will be selected purposively using organizational charts and information on each employee’s role at the host organization and its partner organizations. At the grantee level, we will interview the grantee director as well as key managers and coordinators. Some grantees are implementing PJAC in multiple counties or communities. For those grantees, we will also interview the grantee’s lead staff member in each community. All grantees are required to partner with other organizations to provide services, including employment, parenting, education, mental health, and substance abuse. We will interview the lead staff person responsible for grant activities at each partner agency. In addition, we will interview frontline case workers. Purposeful selection is appropriate for staff selection because insights and information can only come from individuals with particular roles or knowledge.
Participant interview topic guide. The evaluation team will conduct individual interviews with participants from both the program and control groups in each of the six grantee sites. Interviews will be conducted with unique individuals during each of two rounds of implementation site visits. Participants will be randomly selected from among those who have engaged with the child support agency in some way, either through participation in PJAC services among program group members or responsiveness to contempt proceedings among control group members. The goal will be to interview 15 participants in each site during each visit. Anticipating that participants will be difficult to recruit, the evaluation team will oversample, selecting approximately 30 participants in each site in the hopes of completing 15 interviews.
Participant survey tracking letter. The survey will be administered to a subsample of 3,000 participants, divided evenly between the six grantee sites. While specific plans are still being determined for how subsampling will be conducted, the evaluation team plans to use available information to oversample among those most likely to engage with the child support agency since these are the participants for whom the survey content will be most relevant and for whom obtaining survey completes will be most feasible, as many noncustodial parents in the study may never be located. However, by including others in the subsample as well, the evaluation team hopes to be able to weight back to a more representative sample, optimizing the utility of the survey data. All participants selected for the subsample will be sent the survey tracking letter based on the best address information available for them.
Additionally, all six PJAC grantees will be part of the impact study. Noncustodial parents will be eligible for PJAC if they are about to be sent to contempt for failure to meet their child support obligations. Grantees are expected to randomly assign 1,000 NCPs who are eligible for PJAC each year for three years, yielding a total of 3,000 sample members within each site and 18,000 sample members across all six sites. Sixty-five percent of these NCPs will be randomly assigned to the treatment group and will be offered PJAC services; the remaining 35 percent will be randomly assigned to the control group and will not be offered these services.
Because there is likely to be some variation in the PJAC programs offered by the six grantees, each site will be analyzed separately, although pooled analysis will also be conducted if such analysis is deemed appropriate once more is known about the similarity of the six PJAC programs. To support site-level analysis, relatively large samples are required within each site to detect policy-relevant program impacts.
Procedures for the Collection of Information
Data collection procedures for the five instruments being submitted under this ICR are described below:
Staff data entry for random assignment (IC #1) and study MIS to track program participation (IC #2). In the six grantee sites, program staff will identify noncustodial parents who are eligible for PJAC services. When intake workers are ready to enroll a noncustodial parent into the PJAC sample, program staff will use the study MIS to enter a few key pieces of information (pulled from their internal databases) and conduct random assignment. The study MIS will also be used in each site to document service use by program participants. The web-based system will allow program staff to document all service outreach and contact with participants, enhanced investigation and location efforts, case management, referrals to other supportive services in the community, etc.
Staff and community partner interview topic guide (IC #3). Interviews will be conducted with child support staff and local service provider staff during site visits conducted in Fall 2018 and Winter 2020. Interviews will be one-on-one or in small groups, depending on the staffing structure, roles, and number of staff in each role. Topics for the first round of interviews will include documenting the service model, implementation system, and inputs to implementation; assessment of early program operations and participant responsiveness; experiences of staff, participants, and community partners; operational challenges and solutions. Topics for the second round of interviews, once service delivery had reached a steady state, will include documentation of program operations, assessment of participant experiences and responsiveness, community partnerships, staff experiences, adaptations to the service model, and cost data collection. Both rounds will place particular emphasis on understanding how procedural justice-informed approaches were incorporated into service delivery.
Participant interview topic guide (IC #4). Interviews will be conducted with individual study participants during site visits. For program group members, the objective of these interviews will be to explore participants’ perspectives on the availability, quality, content, and value of PJAC services. The interviews will be used to learn about participants’ experiences with each of the core services offered and their perceptions of fairness and transparency in their interactions with the system. For control group members, questions will focus on their experiences in the contempt process and enforcement actions, any supportive services they may have received, and their perceptions of fairness and transparency in interacting with the system from their perspective.
Participant survey tracking letter (IC #5). Tracking letters will be mailed to the 3,000 members of the survey subsample at 2-3 different points in time between the point at which they are enrolled into the study and 12 months later, when the evaluation survey firm will reach out to them to attempt to conduct a survey interview. Tracking letters will request confirmation that current contact information on record is still accurate and/or updates to contact information. Participants will have the option of providing this updated information via three different modes: (1) postage-paid return envelope, (2) call-in to the survey firm, and (3) secure web submission.
Methods to Maximize Response Rates and Deal with Nonresponse
Expected Response Rates
The only instrument in this ICR for which response rate calculation are relevant is the participant survey tracking letter. Based on previous tracking efforts for surveys targeting a relatively similar population (low-income noncustodial parents in the Enhanced Transitional Jobs Demonstration and the Subsidized and Transitional Employment Demonstration), the evaluation team believes the response rate of sample members providing updated contact information will be low, hovering around 5 to 10 percent for each mailing. However, given the expected difficulty of surveying the PJAC population and the relatively low cost of these mailings, the evaluation team believes tracking to be a worthwhile endeavor despite the anticipated low response rate. Additionally, by allowing sample members to return updated contact information via multiple modes (mail, phone, web), the evaluation team hopes to maximize the potential return of the tracking effort.
Dealing with Nonresponse and Nonresponse Bias
The evaluation team anticipates difficulties with nonresponse in the 12-month follow-up survey, an instrument that will be submitted as part of the next OMB submission. In an effort to mitigate these concerns, we have included participant survey tracking letters with $2 pre-paid incentives as part of the overall survey effort, in hopes of maintaining better contact information for survey sample members and lessening non-response. Supporting Statement A includes additional information regarding the choice to use pre-paid incentives based on recent literature.
Additionally, we plan to offer a $15 gift card to participants as a gift in appreciation of their participation in individual participant interviews during site visits and to offset their transportation and other costs.
Maximizing Response Rates
To maximize response rates and data reliability for the study MIS, the following steps will be taken:
Develop a user-friendly, flexible study MIS. The study MIS will be specifically designed for use by grantee site staff. As such, it will be extremely user-friendly and flexible, so as to meet the needs of each site. Making the system simple and easy to use will improve the quality of the data collected. In addition, by providing sites with this system, we standardize the information being collected from each site and improve the reliability of our implementation measures.
Include data quality checks in the study MIS. The study MIS will also ensure data reliability by instituting automatic data quality checks. For example, if grantee staff enter odd or unlikely values in a particular field, the system will prompt users to check the value. For some fields, the response values will be restricted; for others, grantee staff site will be able to override the check.
Provide extensive training to grantee site staff. To increase data quality, the evaluation team will provide extensive training to system users prior to initial use. Initial training will be on site; follow-up training will be conducted using web and telephone conferences. Following training, evaluation team members will conduct follow-up site visits to ensure compliance with procedures and be available by phone and email to assist users.
Monitor data quality. The evaluation team will also monitor the data entered by grantee sites and provide feedback to grantees on their data quality. Initially, data will be monitored on a weekly basis, and will be tapered gradually to monthly monitoring as agencies demonstrate their ability to use the system correctly.
Test of Procedures or Methods to be Undertaken
All functions of the MIS system will be rigorously tested and evaluated by the development team to ensure proper functionality. Additionally, the evaluation team will consult with practitioners on the usability of the system and engage these practitioners in the testing phase. The evaluation team will troubleshoot and correct any errors or problems detected during the testing phase, focusing on continuous improvement until the system operates smoothly for all users. Furthermore, all PJAC caseworkers will be trained on proper data entry procedures for random assignment and program participation tracking. The evaluation team will closely monitor data entry and provide feedback and oversight to grantees.
The other information collection activities included in this request do not lend themselves to testing. However, the participant 12-month survey, staff survey, and staff time study which will be included in a future ICR will involve pre-testing strategies that will be described in the next submission.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Preliminary input on statistical methods was received from staff in OCSE as well as staff at the University of Wisconsin and MDRC, including the following individuals:
Dr. Elaine Sorensen
Senior Advisor, U.S. Department of Health and Human Services, Administration for Children and Families, Office of Child Support Enforcement
330 C Street SW
Washington, DC 20201
Dr. Dan Meyer
Professor
University of Wisconsin-Madison, School of Social Work
1350 University Ave.
Madison, WI 53706
Dr. Cynthia Miller
Senior Fellow
MDRC
16 E. 34th Street Fl. 19
New York, NY 10016
Dr. Richard Hendra
Senior Research Associate
MDRC
16 E. 34th Street Fl. 19
New York, NY 10016
Dr. Erin Valentine
Research Associate
MDRC
16 E. 34th Street Fl. 19
New York, NY 10016
In the future, further input on analytic approaches may be sought from additional staff at these organizations.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Melanie Skemer |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |