SSA - Formative Generic Program Support - THV CQICs Process Study

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Tribal Continuous Quality Improvement Collaboratives Process Study

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part A

June 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Nicole Denmark, Office of Planning, Research, and Evaluation

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531)

• Description of Request:

This request is for information collection related to a process study of the Tribal Home Visiting Continuous Quality Improvement Collaboratives (THV CQICs). The Early Language and Literacy and Family Engagement THV CQICs bring together 19 grantees in a structured and facilitated process for testing evidence-based strategies to make practice improvements towards each collaborative’s aim. The process study will document how the THV CQICs are implemented and examine its feasibility and appropriateness for tribal communities. Information collection will include a survey of THV CQIC participants. This information will help ACF THV assess whether to include this type of CQIC approach in the ongoing learning agenda and to plan for future technical assistance needs.

We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

ACF’s Office of Planning, Research, and Evaluation (OPRE) at the U.S. Department of Health and Human Services (HHS) seeks approval under the umbrella generic: Formative Data Collections for Program Support (OMB #0970-0531) to collect information from all 19 Tribal Maternal, Infant, and Early Childhood Home Visiting (MIECHV) grantees participating in the Early Language and Literacy and Family Engagement Tribal Home Visiting Continuous Quality Improvement Collaboratives (THV CQICs). This information is necessary to document how the THV CQICs are implemented and assess the feasibility and appropriateness of THV CQICs for tribal grantees.

Continuous quality improvement has been a focus and area of innovation in the MIECHV program since its inception. In 2013, the Home Visiting Collaborative Improvement and Innovation Network (HV CoIIN) was implemented using the Breakthrough Series (BTS) model with MIECHV grantees and local home visiting agencies. While the HV CoIIN demonstrated that the BTS model could be used to improve services and outcomes within home visiting (Education Development Center, 2020), the THV CQICs are the first attempt to adapt the BTS model to improve tribal early childhood services. Likewise, while the BTS model has been shown to be effective in other health and human services programs, the approach is time and resource intensive. ACF is interested in this CQIC approach and the information collected through this proposed generic information collection (GenIC) would allow ACF to learn how feasible and appropriate the model is for tribal home visiting contexts. ACF has contracted with James Bell Associates (JBA) to complete this information collection.

There are no legal or administrative requirements that necessitate or authorize this information collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Planning for provision of programmatic or evaluation-related training or technical assistance (T/TA).

• Development of learning agendas and research priorities.

The purpose of this process study is to document how the THV CQICs, an adapted version of the Institute for Healthcare Improvement’s BTS Collaborative Model, is implemented within ACF’s THV program. The THV CQICs bring together 19 grantees for a 16-month long structured and facilitated process for testing evidence-based as well as innovative strategies to make improvement toward collaborative-level aims related to early language and literary and family engagement. The collaborative aims are quantifiable desired outcomes of the collaboratives (e.g., the collaborative aim for the Early Language and Literacy CQIC is to increase the percentage of families who read, sing, or tell stories to their children every day). Throughout the collaboratives, grantee teams select strategies to implement and then complete rapid Plan-Do-Study-Act (PDSAs) cycles to test, assess results, and scale change within their own programs. Grantees meet virtually to share learnings from these cycles and review collaborative-level progress on a set of standard measures related to the topic. ACF’s goals for the collaboratives include: 1) improve home visiting practice; and 2) build grantee capacity for CQI.

This request describes the larger descriptive process study of the THV CQICs for context, but this GenIC request is for approval of one specific survey (see Study Design for additional information). The information from the process study will help inform ACF THV about whether to include this CQIC approach in their ongoing learning agenda and to plan for future technical assistance needs. ACF is the primary stakeholder of the study findings. Secondary stakeholders include THV grantees, TA providers, and other federal agencies providing grants to tribal communities.

Process Study Aims and Questions

The two aims of the process study are to: 1) describe how the CQI Collaboratives were implemented; and 2) describe results and lessons learned from implementing the CQI Collaboratives. The evaluation questions, listed below, have been organized according to each of these two aims. Questions 1.1 through 1.5 below address THV CQIC implementation, resources expended, TA supports, grantee engagement, and changes tested by grantees. Questions 2.a.1 through 2.c.4 address outcomes of the quality improvement work, and the appropriateness and feasibility of the approach.

Aim 1. Describe how the CQI Collaboratives were Implemented

1.1. How were the CQI Collaboratives implemented within the Tribal Home Visiting context?

1.2. What kinds of resources at the federal level (including contracted TA providers) were utilized/put in place to implement the CQI Collaboratives?

1.3. What kinds of TA supports were utilized by grantees?

1.4. To what extent did grantees engage in the collaboratives?

1. 5. What kinds of changes did grantees test through rapid cycle PDSAs?

Aim 2a. Describe Results and Lessons Learned

Aim 2a. Outcomes of the Quality Improvement Work

2a.1. What kinds of improvements to home visiting practice were made by individual grantee teams?

2a.2. What kinds of improvements were seen by the collaboratives as a whole?

2a.3. What knowledge was gained by the changes that tribal grantees tested?

2a.4. What capacities were built as a result of participating in the collaboratives?

Aim 2b. Appropriateness of the approach

2b.1. In what ways were the collaboratives useful to grantees in reaching their own goals and objectives?

2b.2. How compatible were the collaboratives with grantees’ practices and values?

2b.3. Would Tribal Home Visiting grantees recommend that this approach be implemented again in Tribal Home Visiting or other federally-funded grant programs and if so, what are their recommendations for improvement?

Aim 2c. Feasibility of the approach

2c.1. To what extent were grantees able to experience shared learning with other grantees in the collaborative?

2c.2. What were some of the challenges experienced by grantees through participation in the collaboratives?

2c.3. What did teams perceive as the most valuable supports?

2c.4. What resources, strengths, and capacities are required to participate in the collaboratives?

Study Design

The information from the process study of the THV CQICs is meant to contribute to the body of knowledge on ACF programs. It may be relevant to other ACF programs that fund services in tribal communities, but the results are not generalizable outside of the THV grantees participating in the collaboratives. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

The THV CQIC process study will utilize both primary and secondary data. Primary data will be collected to examine grantee perspectives on the collaborative. Primary data collection methods include a survey of staff at all 19 THV grantees participating in the collaboratives (Instrument 1: Survey of THV CQIC participants) and telephone interviews with individual staff members from a subset of 8 grantees¹. Home visitors, program managers, program directors, evaluators and other local staff are participating in the collaborative and will be invited to participate in the survey. Follow-up interviews will be conducted with one staff member from 8 purposively chosen grantee teams (8 interviews total). We will select grantees that represent a range in prior CQI experience, home visiting model implemented, program size, and setting (i.e., rural reservation or urban). Interviewing one staff member from 8 grantee teams will allow us to sample for these characteristics while limiting interviews to fewer than half of participating grantees. Secondary data analyses will entail review of existing documents and data associated with implementing the CQICs (Tribal Continuous Quality Improvement Collaboratives GenIC. OMB #: 0970-0531, approved 2/19/2020). Secondary data will be analyzed to describe implementation and results of the collaborative. Data will be extracted from documents and reports submitted by participating grantees including Collaborative Charters, monthly PDSA Planning Tools and data reports. Survey data collection will occur following OMB approval over about a one-month period (currently estimated to start in December of 2021) and interviews with the subset of grantees will follow.

Methods and Data Collection Activities

Questions within Aims 1 and 2a, which focus on describing how the THV CQICs were implemented and the results of the collaboratives, will be approached primarily through qualitative methods. Aims 2b and 2c, which focus on the appropriateness and feasibility of the approach, will primarily be explored through mixed methods. Table 1 specifies the methodological approach for each evaluation question. References to the survey of THV CQIC participants covered by this GenIC are bolded.

Survey of grantee CQIC participants. A web-based survey will be administered to staff toward the end of the THV CQICs’ third Action Period (December 2021). The survey consists of 50 short questions that capture grantee staff members’ assessment of: capacities built as a result of participation, collaborative topic alignment with program goals and objectives, recommendations for improvement and future implementation, perceived peer sharing and shared learning, barriers to participation, and helpfulness of THV CQIC supports available to grantees. Table 2 summarizes information about the survey.

Table 1. Methods by Evaluation Question

Evaluation question	Analytical approach	Data sources
Aim 1. Describe how the THV CQICs were implemented
1.1 How were the CQI Collaboratives implemented within the Tribal Home Visiting context?	Qualitative analysis	Secondary data
1.2 What kinds of resources at the federal (including contracted TA providers) were utilized/put in place to implement the CQI Collaboratives?	Qualitative analysis	Secondary data
1.3 What kinds of TA supports were utilized by grantees?	Qualitative analysis	Secondary data
1.4 To what extent did grantees engage in the collaboratives?	Quantitative analysis	Secondary data
1.5 What kinds of changes did grantees test through rapid cycle PDSAs?	Qualitative analysis	Secondary data
Aim 2a. Outcomes of the quality improvement work
2a.1. What kinds of improvements to home visiting practice were made by individual grantee teams (i.e., improvements from running individual team PDSAs)?	Qualitative analysis	Telephone interviews with 8 grantee staff from CQI teams; Secondary data
2.a.2 What kinds of improvements were seen by the collaboratives as a whole?	Quantitative analysis	Secondary data
2a.3 What knowledge was gained by the changes that tribal grantees tested?	Qualitative analysis	Telephone interviews with 8 grantee staff from CQI teams; Secondary data
2a.4 What capacities were built as a result of participating in the collaboratives?	Mixed methods- explanatory sequential	Survey of THV CQIC participants; Telephone interviews with 8 grantee staff from CQI teams
Aim 2b. Appropriateness of the approach (i.e., the perceived fit, relevance, or compatibility of the approach for Tribal Home Visiting grantees)
2b.1 In what ways were the collaboratives useful to grantees in reaching their own goals and objectives?	Mixed methods- explanatory sequential	Survey of THV CQIC participants; Telephone interviews with 8 grantee staff from CQI teams
2b.2 How compatible were the collaboratives with grantees’ practices and values?	Mixed methods- explanatory sequential	Survey of THV CQIC participants; Telephone interviews with 8 grantee staff from CQI teams
2b.3 Would Tribal Home Visiting grantees recommend that this approach be implemented again in Tribal Home Visiting or other federally-funded grant programs and if so, what are their recommendations for improvement?	Quantitative/qualitative analysis	Survey of THV CQIC participants
Aim 2c. Feasibility of the approach (i.e., the extent to which a new treatment, or an innovation, can be successfully used or carried out within a given setting)
2c.1 To what extent were grantees able to experience shared learning with other grantees in the collaborative?	Mixed methods- explanatory sequential	Survey of THV CQIC participants; Telephone interviews with 8 grantee staff from CQI teams;
2c.2 What were some of the challenges experienced by grantees through participation in the collaboratives?	Mixed methods- triangulation	Survey of THV CQIC participants; Secondary data
2c.3 What did teams perceive as the most valuable supports?	Mixed methods- triangulation	Survey of THV CQIC participants; Telephone interviews with 8 grantee staff from CQI teams;
2c.4 What resources, strengths, and capacities are required to participate in the collaboratives?	Qualitative analysis	Telephone interviews with 8 grantee staff from CQI teams;

Table 2. Data Collection Activity Covered Under this GenIC

Data Collection Activity	Instrument	Respondent, Content, Purpose of Collection	Mode and Duration
Survey of THV CQIC Participants	Instrument 1: Survey of THV CQIC participants	Respondents: Grantee staff participating in the THV CQICs Content: Capacities built, role in reaching grantees’ goals and objectives, compatibility with grantees’ practices and values, recommendations for implementation in future grant programs, suggestions for improvement, opportunities for shared learning, challenges experienced, value of supports received General Purpose: This instrument will assess grantees’ experiences in the collaborative	Mode: Web Duration: 10 minutes

Other Data Sources and Uses of Information

Interviews with a subset of grantee staff. Telephone interviews with a total of 8 grantee staff will be conducted to enable elaboration of survey findings. Interviews will allow for an in-depth exploration of grantees’ experiences participating in the THV CQICs. The interviews will be conducted using a purposive sample of staff from four grantee teams from each collaborative (individuals representing a total of 8 grantee CQI teams will be interviewed). One member of a grantee’s local CQI team will be invited to participate in a 90-minute interview. The interview protocol will include questions about similar topics as the survey including capacities built as a result of participation, collaborative topic alignment with program goals, perceived peer sharing and shared learning, and helpfulness of THV CQIC supports available to grantees. Unique topics explored include improvements made from running PDSAs; lessons learned from the change strategies adopted; change strategies that were unsuccessful; program goals and objectives reached through participation; challenges experienced; resources, strengths, and capacities needed to participate in the collaboratives; and impacts of the COVID-19 health pandemic on grantees’ ability to participate in CQI activities and meet the collaborative requirements.

Secondary data analyses. Secondary data will be analyzed to assess the implementation, outcomes and feasibility of the THV CQICs. The use of secondary data will reduce grantee burden and leverage existing information generated for the purpose of THV CQIC implementation. The study team will abstract data from existing documents and administrative datasets using a standardized process. The secondary data sources and the information they will provide are described in Table 3 below. Two of these secondary data sources, the PDSA Planning Tools and Monthly Reporting Templates, are covered under the following GenIC:

• Tribal MIECHV Continuous Quality Improvement Collaboratives. (OMB #: 0970-0531, approved 2/19/2020)

Table 3. Secondary Data Sources

Data Source	Information Provided
Existing Documents
Collaborative Charters	Collaborative background; purpose; goals; aims; and roles and responsibilities of grantee teams, TA providers, faculty, and ACF
THV CQIC infographic	Major activities to be completed and the overall THV CQIC timeline
PDSA Planning Tools	Number and types of PDSA topics, changes tested and adopted, grantee insights on findings from the Study stage
Monthly Reporting Templates	Measurable outcomes of the collaboratives, challenges experienced during PDSA cycle, and training and TA needs
Administrative Data Sources
THV CQIC Implementation Tracking Spreadsheet	Date, purpose and content of activities occurring throughout the duration of the two collaboratives (e.g.., Learning Sessions and content, webinars, Leadership calls, Action Period calls, etc.)
Federal and TA Provider THV CQIC Activity Log	Activities completed by ACF and TA providers to plan and implement the THV CQICs, resources expended to implement the THV CQICs
TA Logs and Reports	Types of TA supports provided and used by grantees

A3. Use of Information Technology to Reduce Burden

Information will be collected using Qualtrics web-based survey software. We anticipate this format will provide the lowest burden to the respondent.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team has developed a plan to leverage existing data for this process study in order to reduce duplication, minimize burden, and increase utility and government efficiency. Please see section A2 above for information about the use of existing data.

A5. Impact on Small Businesses

The project will include tribal human service agencies. We will only request information required for the intended use.

A6. Consequences of Less Frequent Collection

Without collecting information on grantees’ experiences with the THV CQIC, ACF will not have relevant information needed to make decisions about future CQIC efforts. This information will be collected once.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published two notices in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request. The first notice was published on October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. ACF did not receive any substantive comments.

Consultation with Experts Outside of the Study

We consulted with ACF program office staff, contracted TA providers, and grantee representatives. We also consulted with evaluators from 4 of the THV programs. These consultations focused on the evaluation aims and questions, data collection protocol and instruments.

Table 4. Outside Experts Consulting on the Process Study

Name	Affiliation
Samantha Martin	Wellington Group Consulting
Kyle Noble	Lake County Tribal Health Consortium
Ramona Danielson	North Dakota State University
Debra Heath	University of New Mexico
Sophia Taula Lieras	Zero to Three
Petra Smith	Zero to Three

A9. Tokens of Appreciation

This information collection will not include tokens of appreciation for participation.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

This effort does not include the collection of sensitive personally identifiable information (PII). Staff email addresses will be used to administer the survey, but email addresses will not be connected to survey data files. The survey will collect respondents’ role within their program team, the length of time respondents participated in the THV CQIC, and the name of the program they work for. Access to staff email addresses is restricted to only those working on the THV CQIC Process Study project.

Assurances of Privacy

We will inform respondents of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As the contract specifies, the contractor will comply with all federal and departmental regulations for private information. The contractor shall ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

Data Security and Monitoring

JBA has an established firm-wide System Security Plan that assesses all data security measures and monitoring procedures to ensure secure storage and transmittal of information. This plan is updated at least annually.

Primary data collected for the THV CQIC Process Study will be stored on a secure OneDrive site. Secondary data will be stored on a secure SharePoint site. JBA is a subscriber to the FedRAMP ATO-holding Microsoft Online 365 Service with both Business and Enterprise licenses. JBA maintains multiple SharePoint and OneDrive sites to separate data between projects and access requirements within those projects. In addition to operating with Microsoft best practices for security, SharePoint and OneDrive will use the following additional controls that fall within JBA responsibilities for management. Additional documentation pertaining to the security of SharePoint and OneDrive can be found within Microsoft’s approved FedRAMP package.

Access Controls

• JBA SharePoint and OneDrive requires users to authenticate using multi-factor authentication for all users.

• JBA SharePoint and OneDrive uses role-based access permissions to limit access to sensitive data and separate access based on assigned roles.

• Only Administrators have access to modify the security policies, sharing permissions or role-based access permissions.

• Permissions granted to a user account are based on the principal of least privilege so that users are not afforded access to the system greater than their minimum requirements.

• Passwords used by user and administrative accounts require a minimum of 16 characters and must be complex, meaning that they must contain at least one number, one capital letter, and one symbol.

Remote Access

• JBA only permits users with a valid account access to the JBA managed SharePoint sites. Anonymous or sharing links are prohibited.

• JBA SharePoint Online is hosted on the FedRamp approved Microsoft Office 365 SharePoint online service. Microsoft controls remote access to the SharePoint Platform. JBA controls user access into the JBA owned and operated sites. Security related to the transmission to and from SharePoint online is documented in the Microsoft FedRamp package available to the government at https://www.fedramp.gov.

A11. Sensitive Information ²

The information collection does not include sensitive information. The project team will seek approvals or exemptions from JBA’s IRB. JBA will also consult with participating tribal grantees to determine whether review by local tribal IRBs is needed. JBA will engage local tribal IRBs after OMB approval is received in order to avoid asking tribal IRBs to re-review study materials if there are changes required by OMB.

A12. Burden

Explanation of Burden Estimates

Survey respondents include all members of the THV grantee teams participating in the CQICs. We have estimated that up to 130 grantee staff will be invited to take the survey. To estimate this number, we reviewed data on the number of THV staff members employed at each grantee agency, as well as the number of staff vacancies submitted in the THV Quarterly Form 4 (OMB-0970-0525). The web-based survey instrument is expected to take 10 minutes to complete per response.

Table 5. Estimated Annualized Burden and Costs to Respondents

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 1: Survey of THV CQIC Participants	130	1	.17	22	$36.76	$808.72

Explanation of Cost Estimates

Costs were estimated based on the job code is 21-1093, Social and Human Service Assistant. Wage data from 2020 is $18.38 per hours. To account for fringe benefits and overhead the rate is multiplied by two which is $36.76.

https://www.bls.gov/oes/current/oes211093.htm

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Table 6. Estimated Annualized Costs to the Federal Government

Cost Category	Estimated Costs
Stakeholder Engagement	$1,704
Survey Development	$14,200
Obtaining OMB Approval	$12,070
Obtaining IRB Approval	$40,328
Survey Administration and Data Analysis	$4,260
Developing Report of Findings	$11,360
Total costs over the request period	$83,922

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

Data collection will take place following IRB (if needed) and OMB approval for one month. The web-based survey will be administered in December of 2021.

Table 7. Timeline

	Begins	Complete
Administer Survey	December 2021	December 2021
Analysis of Survey Data	January 2022	February 2022
Collection and Analysis of Secondary Data	Currently in process	April 2022
Developing Report of Findings	June 2022	August 2022

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument 1: Survey of THV CQIC Participants

Appendix A: THV CQIC Survey Invitation Emails

References

Education Development Center (2020). Home Visiting Collaborative Improvement and Innovation Network 2.0 (HV CoIIN 2.0). https://hv-coiin.edc.org/sites/hv-coiin.edc.org/files/General%20CoIIN%20Fact%20Sheet%201.2020.pdf.

1 These interviews are not subject to PRA due to the number of respondents (fewer than 10).

2

14