SSB - Formative Research GenIC: Family Partners for Research Study

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Family Partners for Research Study

Formative Data Collections for ACF Research

0970 – 0356

Supporting Statement

Part B

June 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Christine Fortunato and Laura Hoard

Office of Planning, Research, and Evaluation

Part B

B1. Objectives

Study Objectives

The Family Partners for Research Study is intended to inform the design of future National Survey of Child and Adolescent Well-Being (NSCAW) data collections. The main objectives of this information collection are to:

1. Conduct an initial exploration of the assessments under consideration in potential future NSCAW data collections, including assessing the feasibility and utility of making available alternative modes by which to administer child well-being assessments (i.e., in-person, remotely¹), and with different reporters (i.e., parent, interviewer);

2. Gain a comprehensive understanding of respondent (i.e., parent) experience with the set of child well-being assessments through in-person semi-structured feedback interviews and online feedback questionnaires.

The purpose of this information collection is to provide information for consideration about potential future NSCAW data collections. Moreover, the aim is to gain a better understanding of the extent to which remote (i.e., online, telephone) and in-person data collection modes and associated experiences of burden, challenges, and preferences (as reported by parents) are comparable by engaging individuals with lived experience in the child welfare system (CWS)².

Generalizability of Results

These activities are intended to explore the feasibility of administering child well-being assessments in-person and remotely and to collect qualitative information on respondent experiences, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The information collection plans described here were designed to gather information that will inform planning and preparations for future NSCAW data collections by exploring the value and feasibility of multi-mode administration for a select set of child well-being assessments. Study activities were designed to explore the administration of assessments in different modes (i.e., remotely and in-person) and with different reporters (i.e., parent and interviewer). In addition, activities were designed to allow parents who have former lived experience with the CWS to provide feedback on various aspects of their participation (e.g., administration modes, instrumentation, perceptions of burden). These activities align with the goal of providing individuals who have lived experience with the CWS an opportunity for meaningful and appropriate involvement throughout the project. Information gathered through this effort will be used to inform future NSCAW design components and data collections.

Information obtained in the study is not intended to be representative of general populations. Study limitations, including not varying the order in which respondents receive remote and in-person instrumentation (i.e., all respondents will receive the remote assessments first, followed by the in-person) as well as a relatively small sample size, will be disclosed in written study products.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target population will be comprised of parents who previously had contact with the CWS (i.e., cases are closed) and their child aged 2-5 years. The sample of parents to be recruited currently serve as professional family mentors to families entering the CWS or as peer mentors.

Sampling

To identify potential respondents, non-probability, purposive sampling will be used to recruit up to 70 parents and their child aged of 2 to 5 years. Participating parents and children will not be representative of all family mentors, children, and families currently in contact with the CWS, or of peer mentors. Instead, the goal is to gather preliminary information on the feasibility, value, and relevance of novel (i.e., not used in prior NSCAW cohorts) instrumentation and alternative methodologies and to increase awareness of those issues of greatest relevance to children and families involved with the CWS.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Remote and In-Person Child Well-being Assessment

To develop study Instruments 2, 4, and 5, priority constructs and domains were identified, focusing on child social-emotional well-being. A selection of standardized, norm-referenced assessments³ that measure child social-emotional well-being including aspects of development; functioning; mental, emotional, and behavioral development; life satisfaction and happiness; and parenting were identified by the following criteria:

1. Are considered as possible assessments for potential future NSCAW data collections, and were not administered in prior NSCAW cohorts

2. Are of short duration and/or include characteristics that are intended to reduce respondent burden and increase engagement

3. Have been administered across multiple modes in previous research (e.g., in-person and online)

4. Report strong psychometrics

Using these criteria, items from the following child well-being assessments were selected:

• Developmental Assessment of Young Children, Second Edition (DAYC-2)⁴ to assess aspects of child development including early childhood communication skills, cognition, and social emotional skills.

• Vineland Adaptive Behavior Scales, Third Edition (Vineland-3)⁵ to assess aspects of child functioning including communication and daily living

• The Behavior Assessment System for Children, Third Edition⁶ (BASC-3), Behavioral and Emotional Screening System (BESS), Parent Form to assess mental, emotional and behavioral development.

• Parental Assistance with Child Emotional Regulation (PACER)⁷ to assess aspects of parenting including co-regulation, warmth, and responsiveness.

Feedback on the Online and In-Person Child Well-being Assessment

In addition to the selected standardized, norm-referenced assessments described above, feedback will be gathered from parent respondents via a 13-item online questionnaire (Instrument 3) to be administered immediately following the online assessment. These items were developed by the project team as a means of gathering input on survey characteristics that are hypothesized to have the potential to increase respondent burden for potential future NSCAW data collection(s) (e.g., contacting protocols, informed consent and assent process, ease of use of technology, engagement in the assessment, appropriateness of instrument/assessment timing, and factors that may foster participant engagement).

Finally, parent respondents will participate in in-person semi-structured feedback interviews (Instrument 6) designed to help the project team better understand parents’ experiences with remote and in-person assessments, e.g., challenges they experienced, recommendations they had to improve the experience. In addition, these interviews will ask respondents to envision how they believe a typical NSCAW respondent may experience the remote and in-person assessments that they (and their eligible child) were given. These items were developed by the project team with a goal of gaining insight around what a likely NSCAW respondent may experience, should these online and in-person assessments be administered. Parental burden, challenges, and preferences are critical to study design considerations and cannot be answered through publicly available data sources or by federal partners.

Respondents will not be asked to read materials or prepare information prior to remote or in-person assessments.

Exhibit 1 presents each of the six instruments, along with the corresponding study objective for each.

Exhibit 1: Data Collection Instruments

Data Collection Instruments	Related Study Objective
Instrument 1: Online Parent Pre-Screener and Screening Call	Objective 1: Conduct an initial exploration of the assessments under consideration in potential future NSCAW design components and data collections, including assessing the feasibility, utility of making available alternative modes by which to administer child well-being assessments (i.e., in-person, remotely), and with different reporters (i.e., parent, interviewer).
Instrument 2: Remote Child Well-Being Assessment by Parent Report
Instrument 4: Parent In-Person Interviewer-Administered Child Well-Being Assessment
Instrument 5: Child In-Person Interviewer Administered Play Session
Instrument 3: Online Parent Feedback Questionnaire	Objective 2: Gain a comprehensive understanding of respondent (i.e., parent) experience with the set of child well-being assessments through in-person semi-structured feedback interviews and online feedback questionnaires.
Instrument 6: In-Person Parent Feedback Interview

B4. Collection of Data and Quality Control

The project team will collect all data, first remotely (i.e., online, telephone) and then in-person.

Respondent Recruitment

To recruit a final sample of 60 parents (from the pool of 70 parents who complete the screener) and their eligible child aged 2-5, the project team will collaborate with three entities, the first two of which work directly with family mentor organizations:

1. The Kent School of Social Work, which is the designated evaluator for Kentucky and other states’ Sobriety and Recovery Teams (START)

   1. START organizations usually have staff family mentors embedded within child welfare agencies.

2. The National Resource Center (NRC) for the Infant-Toddler Court Program (ITCP) at ZERO TO THREE.

   1. The NRC works with family mentor organizations by supporting families involved with Infant-Toddler courts.

3. The Healing Place (Kentucky), an addiction recovery center network whose nationally recognized program provides the opportunity for clients who successfully complete the program to serve as peer mentors to individuals who are newer to the program.

RTI has an established working relationship with all 3 organizations.

These organizations will be sent an email with a fact sheet to introduce the study design and goals, and explain the request being asked (Appendix B: Outreach Materials for Organizations), with a request to disseminate an email to their staff of family/peer mentors to introduce the study (Appendix C: Outreach Materials for Family Mentors). The outreach materials sent to the family mentors will include an advance letter, fact sheet, parental consent form for their review, and a secure website link to a pre-screener website (Instrument 1⁸ – Online Parent Pre-Screener and Screening Call), wherein interested parents can choose to opt into the study by answering the questions, including providing contact information and child age, and agreeing to be contacted by RTI staff (while also including a few dates/times they are available for the next contact). Following this, an interviewer will contact the parent by phone (Instrument 1 – Online Parent Pre-Screener and Screening Call Script), during which time they will confirm information the parent had entered into the pre-screener (e.g., that they have a child 2-5 years old), provide information about the study, confirm parents have the necessary equipment (i.e., tablet and internet connection), provide guidance on accessing the online instrument, and respond to any questions they may have, including any questions on the consent form. Should the parent not have a tablet and/or internet connection, the interviewer will have one or both shipped to their home. The parent respondent will access the online assessments through the secure link included in the outreach materials at their convenience. Prior to beginning the online assessment, a pre-programmed study introduction will initiate the session and will include the text of the consent form they were sent in the outreach email (Appendix A: Parent Consent Form; Appendix C: Outreach Materials for Family Mentors).

Data Collection Activities and Monitoring for Quality and Consistency

For the standardized, norm-referenced assessments that will be administered remotely parents will receive a phone call from an interviewer at a prescheduled time to administer the remote telephone administration of the DAYC-2. The parent will then receive a link to access this online part of the assessment, followed by the 13-item feedback questionnaire. Two weeks later, study interviewers will contact respondents via email to schedule the in-person standardized, norm-referenced assessments. After this has been completed, in the same visit, parents will participate in the in-person semi-structured feedback interview, during which time the interviewer will take notes and ask for clarification where necessary, aiming to capture the information as close to verbatim as possible. Interviewers will also record the responses (with respondent permission) on their laptop.

Recordings and notes from these interviews will be saved on RTI’s secure servers and will only be accessible by members of the project team and will be destroyed upon study completion. A report summarizing the analysis comparing administration modes for assessments as well as information gathered from the online feedback questionnaire and the in-person semi-structured feedback interviews will be provided to ACF.

Interviewers will be trained in the administration of each of the instruments containing standardized, norm-referenced assessments (i.e., Instruments 2, 4, and 5). Specifically, to the extent possible, interviewers who were hired for NSCAW III will be brought on to engage respondents, prompt them to complete the online survey, and visit their home for the in-person interviews. The NSCAW III interviewers will be expected to meet at least one of the following criteria to engage in training and certification for the DAYC-2:

• Experience administering assessments of young children; or

• Clinical experience with psychological and/or behavioral assessments, and/or an advanced degree or certificate in a related field.

The primary trainer will review credentials prior to training to ensure each interviewer is a good fit. A training protocol for interviewers will be developed by a licensed psychologist and expert in assessment administrations. Following training, trainees will be required to complete at least five assessments with various age groups and respondent types (e.g., parent or child) prior to being certified to begin working.

Data Quality

Assessments (Instrument 2, Instrument 4) will be programmed in Tangerine, an open-source electronic data collection platform built to support researchers collecting data in multiple modalities. Tangerine was selected for this study because of the flexibility and security it provides when conducting research in-person, remotely, or online. Quality assurance and quality control measures will be implemented throughout the development lifecycle, including testing of assessments and confirmation of scoring in conjunction with subject matter experts in the project team.

For online assessments, the parent will receive a link provided by Tangerine which they will use to complete various types of instrumentation depending on their child’s age. The “Get Help” feature will send a notification to an interviewer. If the interviewer is online, they will connect with the parent upon receipt of the message or will follow up to assist the parent within 1 business day of receiving the alert.

The project team will also monitor data collection efforts and report data collection progress on a bi-weekly basis to ACF. These updates will include number of respondents recruited and enrolled, sample demographics, missing data observations, and any noteworthy events.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The remote and in-person assessments, as well as the online feedback questionnaire and in-person semi-structured feedback interviews with parents, are not designed to produce statistically generalizable findings. Additionally, participation is at the respondent’s discretion. Response rates will not be calculated or reported, although intended sample numbers and actual sample numbers will be reported.

Non-Response

As respondents will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Other than child gender and child age, respondent demographics are not being collected. As such, demographics information will be limited in any reports or dissemination products.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination products. Policy decisions will not be made off this data, as it is not representative.

B7. Data Handling and Analysis

Data Handling

Data will be collected through remote and in-person assessments; the protocol will vary depending on the age of the child. The online assessment will be administered in Computer-Assisted Self Interview (CASI) software. The in-person assessment is administered in Computer-Assisted Interview (CAI) software. The use of CASI and CAI software will allow the project team to program assessments with start and end points based on the child’s developmental level, requiring the programming basal and ceiling points as well as use of skip logic across assessments.

After data has been collected, the project team will conduct quality checks to ensure it is appropriate for analysis, including logic checks among responses, checks for outliers, and visual exploratory data analysis to examine response patterns as well as patterns in missing data.

Data Analysis – Quantitative

Quantitative analysis will compare administration modes’ equivalence of child well-being assessments (Instrument 2, 4, and 5). This analysis corresponds to Objective 1 in B.1⁹. Planned analyses include:

1. Descriptive (e.g., mean, standard deviation, frequency, standard errors) for simple comparison estimates of completion and scores of each assessment and all assessments by data collection efforts.

2. Test-retest correlation between the two data collection efforts (e.g., in-person, online) will be similar to the reported test-retest correlation of the standardized assessment, as reported in the instruments’ manuals. If the test-retest reliability of the in-person and online is close enough to the published value, further analysis will test whether there are mean differences across the two modes that bias results up or down, which would require applying a bias adjustment.

3. Bivariate (Chi square) of proportions of children in the clinical range for standardized assessments by data collection effort to determine overlap across instruments and within modes.

With N=60 and two modes of administration, differences in test-retest can be detected, as can bias of Cohen’s d=.54 or larger. That is a medium effect indicating that the average score could be half of a standard deviation difference between the two modes of administration. Noteworthy, having a high correlation between scores is more important than whether there are mean differences that can be adjusted. For stable outcomes, this correlation is expected to be high. Using the non-inferiority margin, the distance between the observed test-retest correlation (ρ0) is from the published test-retest correlation (ρ1) would be examined prior to concluding that the two data collection models yielded different results. For example, the test-retest for DAYC-2 was ρ1=0.70, and with and N of 60, the observed test-retest could drop to ρ0=0.46 (upper output) before it could be concluded that there was a deterioration in test-retest.

Analyses will also determine whether the proposed assessments will support the integration of information across data collection modes (e.g., in-person, remote) for a future NSCAW.

Data Analysis – Qualitative

Qualitative analyses of the online feedback questionnaire (Instrument 3) and the in-person semi-structured feedback interviews (Instrument 6) will correspond to Objective 2 in B.2¹⁰. Analyses will identify key themes related to the following:

1. Mode-related challenges

2. Technology usability problems (online only)

3. Potential burden

4. Any foreseen problems that may potentially influence future study respondents (specifically parents) to engage and participate on future NSCAW data collection efforts

5. Recommendations to promote engagement and participation of future respondents

From this analysis, key findings will be identified, and recommendations offered as part of the report to ACF.

Data Use

Results of these activities will be prepared for internal use by ACF, as a means of informing future NSCAW data collection efforts. Any external release by ACF will be in the form of presentations, reports, briefs, and/or manuscripts, and will not include any generalizable information, but will rather present findings, themes, and recommendations derived from this data collection that will be beneficial to the public (researchers, federal agencies, other interest-holders) as it will disseminate knowledge related to the feasibility of administering assessments in-person and remotely and provide details on high-level challenges that may influence response rates in other national studies. The project team and ACF will make it clear within these dissemination products that the information is not generalizable.

B8. Contact Persons

Name	Title/Affiliation	Email
Melissa Dolan	Project Director, RTI International	[email protected]
Dalia Khoury	Associate Project Director, RTI International	[email protected]
Heather Ringeisen	Design Options Task Lead, RTI International	[email protected]
Christine Fortunato	Contracting Officer’s Representative, ACF	[email protected]
Laura Hoard	Alternate Contracting Officer’s Representative, ACF	[email protected]

Attachments

Instruments

Instrument 1: Online Parent Pre-Screener and Screening Call

Instrument 2: Remote Child-Well Being Assessment by Parent Report

Instrument 3: Online Parent Feedback Questionnaire

Instrument 4: Parent – In-Person Interviewer-Administered Child Well-Being Assessment

Instrument 5: Child – In-Person Interviewer-Administered Play Session

Instrument 6: In-Person Parent Feedback Interview

Appendices

Appendix A: Parent Consent Form

Appendix B: Outreach Materials for Organizations

Appendix C: Outreach Materials for Family Mentors

1 Remote administration is the umbrella term underneath which online and telephone administration line.

2 As described in Section B.2, parent respondents will have lived experience with the CWS.

3 These assessments make up Instruments 2 and 4.

4 Voress, J., & Maddox, T. (2013). Developmental assessment of young children. 2nd ed. PRO-ED.

5 Sparrow, S. S., Cicchetti, D. V., & Saulnier, C. A. (2016). Vineland adaptive behavior scales: Third edition (Vineland-3) Pearson.

6 Reynolds, C. R., & Kamphaus, R. W. (2015). Behavior assessment system for children (3rd ed.). Bloomington: NCS Pearson, Inc.

(BASC–3).

7 Cohodes, E. M., Preece, D. A., McCauley, S., Rogers, M. K., Gross, J. J., & Gee, D. G. (2022b). Development and validation of the

parental assistance with child emotion regulation (PACER) questionnaire. Research on Child and Adolescent Psychopathology,

50(2), 133-148.

8 Instrument 1, the Parent Screener, consists of an Online Pre-Screener to determine eligibility and a Pre-Screening Call for those confirmed to be eligible through the Online Pre-Screener.

9 Objective 1: Conduct an initial exploration of the assessments under consideration in potential future NSCAW data collections, including assessing the feasibility and utility of making available alternative modes by which to administer child well-being assessments (i.e., in-person, remotely), and with different reporters (i.e., parent, interviewer).

10 Objective 2: To gain a comprehensive understanding of respondent (i.e., parents) experience with the set of child well-being assessments through in-person semi-structured feedback interviews and online feedback questionnaires.

8