SSA - SAS-HV Pretesting GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting and Strengthening the Home Visiting Workforce (SAS-HV)

Pre-testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part A

August 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Nicole Denmark

Shirley Adelstein

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (0970-0355).

Description of Request: This is a request to survey and conduct a group interpretation meeting with home visiting practitioners and researchers as part of a concept mapping study design to inform the development of a measure of reflective supervision for home visiting. The data collected from these activities will help identify what end users of the measure view as key elements of reflective supervision in the home visiting context. Results will be used to assemble a pool of items for a new measure of reflective supervision.

We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Maternal Infant Early Childhood Home Visiting (MIECHV) legislation mandates federally funded home visiting programs maintain high-quality supervision practices, and reflective supervision is endorsed in the most recent MIECHV formula funding guidance. Despite strong theoretical support for reflective supervision, there is limited understanding of how it is implemented in practice and limited evidence of effectiveness. This is due, in part, to a lack of valid reliable measures of reflective supervision. Valid, reliable measures of reflective supervision are important for advancing research on the role of reflective supervision in supporting home visitors’ work with families.

The purpose of the Supporting and Strengthening the Home Visiting Workforce (SAS-HV) project is to advance understanding of how to support and strengthen the early childhood home visiting workforce. A prior phase of the SAS-HV project developed a conceptual model of reflective supervision and reviewed current research, measures, and practice to identify gaps in knowledge. This next phase will address a key gap identified by developing and testing a measure of reflective supervision that is practice-relevant and useful for research. This collection is a necessary step to develop a measure of reflective supervision for home visiting that will be primarily used for research and informed by practitioner experience and perspectives, with promising secondary use for practice.

There are no legal or administrative requirements that necessitate the collection. The Administration for Children and Families (ACF) is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of this information collection is to learn what end users of the measure view as key elements of reflective supervision for the home visiting context. This information will contribute to the development of a valid and reliable measure of reflective supervision in home visiting. As a first step in the measure development process, the ACF Office of Planning, Research, and Evaluation (OPRE) is seeking approval to survey and have discussions with home visitors, supervisors, and researchers to identify what these end users of the measure view as key elements of reflective supervision for the home visiting context. Our process for screening potential participants according to established inclusion criteria will help ensure participants reflect the characteristics of potential end users of the measure.

This proposed information collection meets a primary goal of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

We plan to use pretesting information collection to seek input from a broad, diverse array of interested parties on key elements of reflective supervision, which will be analyzed to inform the development of a new measure. The research question we are seeking to address is as follows:

• What do home visiting practitioners and researchers view as elements of reflective supervision for home visiting?

Study Design

The Study Team will use a concept mapping methodology to identify elements of reflective supervision that are deemed most important for reflective supervision and relevant for home visiting from the perspectives of practitioners and researchers with relevant expertise. Concept mapping provides a structured approach for analyzing and visually organizing participant views on a specific topic to identify emerging themes and examine relationships between themes.

In earlier project phases, the Study Team identified a list of reflective supervision elements. The process used to develop this list is described below and in SSB 2. The proposed concept mapping data collection involves structuring, rating, and interpretation tasks, using this list of elements.

The information collection is designed to have three main information collection activities; Table 1 provides a summary overview of these activities:

1. Screening questionnaire: In the questionnaire, participants will be asked to provide information about their race, ethnicity, primary language, role with respect to reflective supervision, and years of experience with reflective supervision. They will also be asked to provide information about the location of the home visiting program they work for, number of families served by the home visiting program, race and ethnicity of families served, primary language of families served, and program model(s) implemented. Finally, participants will be asked to describe their experience with reflective supervision in the home visiting context.

2. Web-based survey: On the survey, participants will be asked to 1) sort a list of elements into similar conceptual groups, and 2) rate elements according to their importance and relevance for the home visiting context. The survey will also ask for basic demographic information about participants, such as participant role in home visiting (supervisor, supervisee, home visitor, researcher, evaluator), home visiting model(s) implemented, and race/ethnicity.

3. Group interpretation meeting: Following analysis, participants from the concept mapping activity will be asked to participate in a virtual meeting to review and engage in interpretation of findings.

Table 1. Data Collection Activities

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
Screening for participation	Screening questionnaire	Respondents: Practitioners (home visitors and supervisors). Content: Potential participant demographics and experience with home visiting; information about the home visiting programs with which they are associated. Purpose: Information collected will be used to determine which individuals meet inclusion criteria and will be invited to participate.	Mode: Online/web-based survey Duration: 5 minutes
Web-based concept mapping activity	Web-based survey	Respondents: Practitioners (home visitors and supervisors) and researchers. Content: Structuring and rating of key elements of reflective supervision. Purpose: Survey results will be analyzed to identify emerging clusters and themes, which will guide the interpretation phase of the study.	Mode: Online/web-based survey Duration: 1 hour
Group interpretation meeting	Group interpretation meeting guide	Respondents: Practitioners (home visitors and supervisors) and researchers. Content: Results from web-based concept mapping activity described above will be presented to participants. The goal of this meeting is for facilitators and participants to examine and name emerging clusters and themes and to determine which are most salient. Purpose: Results will be used to define categories of reflective supervision and to assemble a pool of items to be used on a new measure of reflective supervision.	Mode: Virtual meeting Duration: 1.5 hours

Other Data Sources and Uses of Information

Prior project work informed the development of a list of reflective supervision elements that will be used in the concept mapping activities for this study.

The Study Team reviewed initial activities completed as part of this project, including extraction of elements of reflective supervision from recent literature and grouping them conceptually into categories. The Study team also identified strategies, resources, and existing measures related to reflective supervision.

Building upon the information gathered through initial activities, the Study Team developed a more comprehensive list of reflective supervision elements to bring into the concept mapping phase. To further inform the list, the Study Team conducted a review and crosswalk of existing reflective supervision measures as well as home visiting model, state, and Tribal expectations, and guidelines.

A3. Use of Information Technology to Reduce Burden

Participants will be recruited via an announcement sent through the Home Visiting Applied Research Collaborative listserv. If they are interested in participation, they will have the option to complete a screening questionnaire to determine eligibility.

Surveys will be completed using a secure web platform (GroupWisdom) to reduce participant burden. The proposed survey content was designed to capture the concept mapping activities in this format. Additionally, group interpretation meetings will be completed virtually to reduce participant burden. These meetings will be recorded (upon participant agreement) so that participants do not need to repeat responses or wait for the facilitator to document detailed responses.

Direct person-to-person communication is required. Though the group interpretation meeting format may be more burdensome than an alternative form of computerized assisted data collection (such as a survey or questionnaire), this phase of concept mapping is essential, as it provides a structured approach to analyzing and visually organizing study participant views on reflective supervision elements to identify emerging themes and examining relationships between the themes.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

Information to achieve purposes stated in A.2 does not exist for the field of home visiting. There currently is not information available to understand the key elements of reflective supervision needed to develop a relevant, feasible, acceptable, culturally responsive, and useful measure for home visiting.

A5. Impact on Small Businesses

The web-based survey and group interpretation meetings will include individual staff at state and territory local implementing agencies (LIAs) and Tribal Home Visiting programs, which may be small businesses, as well as researchers who may be employed by small businesses. The requested information is the absolute minimum necessary for the intended use of the data.

The Study Team will minimize the burden on individuals by keeping the survey and interpretation meeting as short as possible. We will also schedule the meetings at a time most convenient for participants and will not request written responses.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published on January 5, 2021; Volume 86, Number 2, page 308, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announcing the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published on May 21, 2021; Volume 86, Number 97, page 27624, and provided a thirty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

The Study Team consulted with a small group of research and evaluation experts, practitioners, and those who support practitioners (supervisors) on the initial list of reflective supervision elements¹. This consultation helped refine and pare down the list of elements that will be used for concept mapping activities, thus reducing burden to study participants. Technical workgroup members include:

Sherryl Scott Heller

Jon Korfmacher

Dawn Nixon

David Schultz

Angela Tomlin

Edward Watkins

Maria Elena Oliveri

A9. Tokens of Appreciation

The study will not include tokens of appreciation.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

For the purposes of study recruitment, we will collect participant name, state, and email. For enrolled participants we will retain records of participant name, state, and email. Group interpretation meetings will include recordings of participants voice and face. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

As specified in the contract for this project, the Study Team will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Study Team will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

At the beginning of the group interpretation meeting, the facilitator will verbally announce, “We would like to record this discussion for notetaking purposes. If you do not wish to be recorded, you may choose to leave the meeting.” This message will also be displayed on screen.

Data Security and Monitoring

The Study Team will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Study Team has developed a Data Security Plan that assesses all protections of respondents’ personally identifiable information. The Study Team will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the evaluator’s contract, the Study Team will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Study Team will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Study Team will: ensure that this standard is incorporated into the Study Team’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations.

A11. Sensitive Information ²

There are no sensitive questions in this data collection.

A12. Burden

Explanation of Burden Estimates

Screening questionnaire: We anticipate up to 60 participants will complete an electronic screening questionnaire. The questionnaire will take about 5 minutes to complete. We do not anticipate variance in response time by respondent type.

Web-based survey: Up to 40 participants will be asked to complete a web-based survey. The survey will take about 60 minutes to complete. We do not anticipate variance in response time by respondent type.

Group interpretation meeting: Following analysis, participants from the concept mapping activity will be asked to participate in a 90-minute virtual meeting to review and engage in interpretation of findings. Up to 40 participants may participate in the interpretation meeting. Multiple interpretation group discussion meetings may be held concurrently to allow for manageable meeting sizes, equal opportunity for participant feedback, and to ensure meetings adhere to the 90-minute meeting duration.

Estimated Annualized Cost to Respondents

The estimated total annual cost to respondents is approximately $3,728.55 (see burden table below). This cost to respondents is based on the average wage of community and social service staff persons (occupation code 21-1000), social and community services managers (occupation code 11-9151), and social scientist and related workers (occupation code 19-3000) for the state network participants. Estimates come from the 2021 Bureau of Labor Statistics report on Wage Estimates (retrieved from https://www.bls.gov/oes/current/oes_nat.htm).

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Screening questionnaire	60	1	.09	5	$35.51	$191.75
Web-based survey	40	1	1	40	$35.51	$1,420.40
Group interpretation3 meeting	40	1	1.5	60	$35.51	$2,130.60
Total	60			105	$35.51	$3,728.55

A13. Costs

The Study Team proposes providing up to $85.00 in honoraria, in the form of a gift cards, to be provided to participants for their time spent providing their expert guidance given their professional expertise. We propose to provide participants with a $35.00 gift card for completing the 60-minute web-based survey and a $50.00 gift card for participating in a 90-minute group interpretation meeting. The Presidential Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policy Making, as well as the ACF Evaluation Policy discuss community engagement and inclusion in research. Consistent with these guidance documents, and to ensure involvement from a variety of people with diverse professional experiences and expertise in the home visiting field, we plan to offer all participants an honorarium. The honorarium is intended to offset costs of providing expert guidance to the project, such as staff time away from other necessary work, or other expenses that might otherwise prevent individuals from participating in the study. The amount for the honorarium was determined by averaging the average hourly wage rates for staff that might participate in the questionnaire, web-based survey, and group interpretation meeting and using this average to calculate an average wage rate for 2.5 hours.

A14. Estimated Annualized Costs to the Federal Government

The estimated annualized costs are based upon full-time equivalent time, operational expenses (such as equipment, overhead, printing, and staff support), and other expenses which would not have been incurred without this collection of information.

Activity	Estimated Cost
Survey administration and monitoring Group interpretation meetings	$18,000.00
Analysis	$19,4000.00
Reporting and documentation	$7,500.00
Total/Annual costs over the request period	$44,900.00

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella clearance for pre-testing (0970-0355).

A16. Timeline

Data collection activities will occur within a 5-month period after OMB approval. Data analysis will occur within 4 months of completing data collection.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix 1: Home Visiting Applied Research Collaborative Recruitment Announcement

Appendix 2: Recruitment Email for Researchers

Instrument 1: Screening Questionnaire

Instrument 2: Web-Based Survey

Instrument 3: Group Interpretation Meeting Guide

1 Since input was collected from fewer than ten individuals, these activities were not subject to the Paperwork Reduction Act.

2 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

10