Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Tribal Continuous Quality Improvement Collaboratives Process Study
Formative Data Collections for Program Support
0970 - 0531
Supporting Statement
Part A
June 2021
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Nicole Denmark, Office of Planning, Research, and Evaluation
Part A
Executive Summary
Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531)
Description of Request:
This request is for information collection related to a process study of the Tribal Home Visiting Continuous Quality Improvement Collaboratives (THV CQICs). The Early Language and Literacy and Family Engagement THV CQICs bring together 19 grantees in a structured and facilitated process for testing evidence-based strategies to make practice improvements towards each collaborative’s aim. The process study will document how the THV CQICs are implemented and examine its feasibility and appropriateness for tribal communities. Information collection will include a survey of THV CQIC participants. This information will help ACF THV assess whether to include this type of CQIC approach in the ongoing learning agenda and to plan for future technical assistance needs.
We do not intend for this information to be used as the principal basis for public policy decisions.
A1. Necessity for Collection
ACF’s Office of Planning, Research, and Evaluation (OPRE) at the U.S. Department of Health and Human Services (HHS) seeks approval under the umbrella generic: Formative Data Collections for Program Support (OMB #0970-0531) to collect information from all 19 Tribal Maternal, Infant, and Early Childhood Home Visiting (MIECHV) grantees participating in the Early Language and Literacy and Family Engagement Tribal Home Visiting Continuous Quality Improvement Collaboratives (THV CQICs). This information is necessary to document how the THV CQICs are implemented and assess the feasibility and appropriateness of THV CQICs for tribal grantees.
Continuous quality improvement has been a focus and area of innovation in the MIECHV program since its inception. In 2013, the Home Visiting Collaborative Improvement and Innovation Network (HV CoIIN) was implemented using the Breakthrough Series (BTS) model with MIECHV grantees and local home visiting agencies. While the HV CoIIN demonstrated that the BTS model could be used to improve services and outcomes within home visiting (Education Development Center, 2020), the THV CQICs are the first attempt to adapt the BTS model to improve tribal early childhood services. Likewise, while the BTS model has been shown to be effective in other health and human services programs, the approach is time and resource intensive. ACF is interested in this CQIC approach and the information collected through this proposed generic information collection (GenIC) would allow ACF to learn how feasible and appropriate the model is for tribal home visiting contexts. ACF has contracted with James Bell Associates (JBA) to complete this information collection.
There are no legal or administrative requirements that necessitate or authorize this information collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose
Purpose and Use
This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):
Planning for provision of programmatic or evaluation-related training or technical assistance (T/TA).
Development of learning agendas and research priorities.
The purpose of this process study is to document how the THV CQICs, an adapted version of the Institute for Healthcare Improvement’s BTS Collaborative Model, is implemented within ACF’s THV program. The THV CQICs bring together 19 grantees for a 16-month long structured and facilitated process for testing evidence-based as well as innovative strategies to make improvement toward collaborative-level aims related to early language and literary and family engagement. The collaborative aims are quantifiable desired outcomes of the collaboratives (e.g., the collaborative aim for the Early Language and Literacy CQIC is to increase the percentage of families who read, sing, or tell stories to their children every day). Throughout the collaboratives, grantee teams select strategies to implement and then complete rapid Plan-Do-Study-Act (PDSAs) cycles to test, assess results, and scale change within their own programs. Grantees meet virtually to share learnings from these cycles and review collaborative-level progress on a set of standard measures related to the topic. ACF’s goals for the collaboratives include: 1) improve home visiting practice; and 2) build grantee capacity for CQI.
This request describes the larger descriptive process study of the THV CQICs for context, but this GenIC request is for approval of one specific survey (see Study Design for additional information). The information from the process study will help inform ACF THV about whether to include this CQIC approach in their ongoing learning agenda and to plan for future technical assistance needs. ACF is the primary stakeholder of the study findings. Secondary stakeholders include THV grantees, TA providers, and other federal agencies providing grants to tribal communities.
Process Study Aims and Questions
The two aims of the process study are to: 1) describe how the CQI Collaboratives were implemented; and 2) describe results and lessons learned from implementing the CQI Collaboratives. The evaluation questions, listed below, have been organized according to each of these two aims. Questions 1.1 through 1.5 below address THV CQIC implementation, resources expended, TA supports, grantee engagement, and changes tested by grantees. Questions 2.a.1 through 2.c.4 address outcomes of the quality improvement work, and the appropriateness and feasibility of the approach.
1.1. How were the CQI Collaboratives implemented within the Tribal Home Visiting context?
1.2. What kinds of resources at the federal level (including contracted TA providers) were utilized/put in place to implement the CQI Collaboratives?
1.3. What kinds of TA supports were utilized by grantees?
1.4. To what extent did grantees engage in the collaboratives?
1. 5. What kinds of changes did grantees test through rapid cycle PDSAs?
2a.1. What kinds of improvements to home visiting practice were made by individual grantee teams?
2a.2. What kinds of improvements were seen by the collaboratives as a whole?
2a.3. What knowledge was gained by the changes that tribal grantees tested?
2a.4. What capacities were built as a result of participating in the collaboratives?
2b.1. In what ways were the collaboratives useful to grantees in reaching their own goals and objectives?
2b.2. How compatible were the collaboratives with grantees’ practices and values?
2b.3. Would Tribal Home Visiting grantees recommend that this approach be implemented again in Tribal Home Visiting or other federally-funded grant programs and if so, what are their recommendations for improvement?
2c.1. To what extent were grantees able to experience shared learning with other grantees in the collaborative?
2c.2. What were some of the challenges experienced by grantees through participation in the collaboratives?
2c.3. What did teams perceive as the most valuable supports?
2c.4. What resources, strengths, and capacities are required to participate in the collaboratives?
The information from the process study of the THV CQICs is meant to contribute to the body of knowledge on ACF programs. It may be relevant to other ACF programs that fund services in tribal communities, but the results are not generalizable outside of the THV grantees participating in the collaboratives. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.
The THV CQIC process study will utilize both primary and secondary data. Primary data will be collected to examine grantee perspectives on the collaborative. Primary data collection methods include a survey of staff at all 19 THV grantees participating in the collaboratives (Instrument 1: Survey of THV CQIC participants) and telephone interviews with individual staff members from a subset of 8 grantees1. Home visitors, program managers, program directors, evaluators and other local staff are participating in the collaborative and will be invited to participate in the survey. Follow-up interviews will be conducted with one staff member from 8 purposively chosen grantee teams (8 interviews total). We will select grantees that represent a range in prior CQI experience, home visiting model implemented, program size, and setting (i.e., rural reservation or urban). Interviewing one staff member from 8 grantee teams will allow us to sample for these characteristics while limiting interviews to fewer than half of participating grantees. Secondary data analyses will entail review of existing documents and data associated with implementing the CQICs (Tribal Continuous Quality Improvement Collaboratives GenIC. OMB #: 0970-0531, approved 2/19/2020). Secondary data will be analyzed to describe implementation and results of the collaborative. Data will be extracted from documents and reports submitted by participating grantees including Collaborative Charters, monthly PDSA Planning Tools and data reports. Survey data collection will occur following OMB approval over about a one-month period (currently estimated to start in December of 2021) and interviews with the subset of grantees will follow.
Questions within Aims 1 and 2a, which focus on describing how the THV CQICs were implemented and the results of the collaboratives, will be approached primarily through qualitative methods. Aims 2b and 2c, which focus on the appropriateness and feasibility of the approach, will primarily be explored through mixed methods. Table 1 specifies the methodological approach for each evaluation question. References to the survey of THV CQIC participants covered by this GenIC are bolded.
Survey of grantee CQIC participants. A web-based survey will be administered to staff toward the end of the THV CQICs’ third Action Period (December 2021). The survey consists of 50 short questions that capture grantee staff members’ assessment of: capacities built as a result of participation, collaborative topic alignment with program goals and objectives, recommendations for improvement and future implementation, perceived peer sharing and shared learning, barriers to participation, and helpfulness of THV CQIC supports available to grantees. Table 2 summarizes information about the survey.
Table 1. Methods by Evaluation Question
Table 2. Data Collection Activity Covered Under this GenIC
Data Collection Activity |
Instrument |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Survey of THV CQIC Participants |
Instrument 1: Survey of THV CQIC participants |
Respondents: Grantee staff participating in the THV CQICs
Content: Capacities built, role in reaching grantees’ goals and objectives, compatibility with grantees’ practices and values, recommendations for implementation in future grant programs, suggestions for improvement, opportunities for shared learning, challenges experienced, value of supports received
General Purpose: This instrument will assess grantees’ experiences in the collaborative
|
Mode: Web
Duration: 10 minutes |
Other Data Sources and Uses of Information
Interviews with a subset of grantee staff. Telephone interviews with a total of 8 grantee staff will be conducted to enable elaboration of survey findings. Interviews will allow for an in-depth exploration of grantees’ experiences participating in the THV CQICs. The interviews will be conducted using a purposive sample of staff from four grantee teams from each collaborative (individuals representing a total of 8 grantee CQI teams will be interviewed). One member of a grantee’s local CQI team will be invited to participate in a 90-minute interview. The interview protocol will include questions about similar topics as the survey including capacities built as a result of participation, collaborative topic alignment with program goals, perceived peer sharing and shared learning, and helpfulness of THV CQIC supports available to grantees. Unique topics explored include improvements made from running PDSAs; lessons learned from the change strategies adopted; change strategies that were unsuccessful; program goals and objectives reached through participation; challenges experienced; resources, strengths, and capacities needed to participate in the collaboratives; and impacts of the COVID-19 health pandemic on grantees’ ability to participate in CQI activities and meet the collaborative requirements.
Secondary data analyses. Secondary data will be analyzed to assess the implementation, outcomes and feasibility of the THV CQICs. The use of secondary data will reduce grantee burden and leverage existing information generated for the purpose of THV CQIC implementation. The study team will abstract data from existing documents and administrative datasets using a standardized process. The secondary data sources and the information they will provide are described in Table 3 below. Two of these secondary data sources, the PDSA Planning Tools and Monthly Reporting Templates, are covered under the following GenIC:
Tribal MIECHV Continuous Quality Improvement Collaboratives. (OMB #: 0970-0531, approved 2/19/2020)
Table 3. Secondary Data Sources
Data Source |
Information Provided |
Existing Documents |
|
Collaborative Charters |
Collaborative background; purpose; goals; aims; and roles and responsibilities of grantee teams, TA providers, faculty, and ACF |
THV CQIC infographic |
Major activities to be completed and the overall THV CQIC timeline |
PDSA Planning Tools |
Number and types of PDSA topics, changes tested and adopted, grantee insights on findings from the Study stage |
Monthly Reporting Templates |
Measurable outcomes of the collaboratives, challenges experienced during PDSA cycle, and training and TA needs |
Administrative Data Sources |
|
THV CQIC Implementation Tracking Spreadsheet |
Date, purpose and content of activities occurring throughout the duration of the two collaboratives (e.g.., Learning Sessions and content, webinars, Leadership calls, Action Period calls, etc.) |
Federal and TA Provider THV CQIC Activity Log |
Activities completed by ACF and TA providers to plan and implement the THV CQICs, resources expended to implement the THV CQICs |
TA Logs and Reports |
Types of TA supports provided and used by grantees |
A3. Use of Information Technology to Reduce Burden
Information will be collected using Qualtrics web-based survey software. We anticipate this format will provide the lowest burden to the respondent.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
The study team has developed a plan to leverage existing data for this process study in order to reduce duplication, minimize burden, and increase utility and government efficiency. Please see section A2 above for information about the use of existing data.
A5. Impact on Small Businesses
The project will include tribal human service agencies. We will only request information required for the intended use.
A6. Consequences of Less Frequent Collection
Without collecting information on grantees’ experiences with the THV CQIC, ACF will not have relevant information needed to make decisions about future CQIC efforts. This information will be collected once.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published two notices in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request. The first notice was published on October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. ACF did not receive any substantive comments.
We consulted with ACF program office staff, contracted TA providers, and grantee representatives. We also consulted with evaluators from 4 of the THV programs. These consultations focused on the evaluation aims and questions, data collection protocol and instruments.
Table 4. Outside Experts Consulting on the Process Study
Name |
Affiliation |
Samantha Martin |
Wellington Group Consulting |
Kyle Noble |
Lake County Tribal Health Consortium |
Ramona Danielson |
North Dakota State University |
Debra Heath |
University of New Mexico |
Sophia Taula Lieras |
Zero to Three |
Petra Smith |
Zero to Three |
A9. Tokens of Appreciation
This information collection will not include tokens of appreciation for participation.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
This effort does not include the collection of sensitive personally identifiable information (PII). Staff email addresses will be used to administer the survey, but email addresses will not be connected to survey data files. The survey will collect respondents’ role within their program team, the length of time respondents participated in the THV CQIC, and the name of the program they work for. Access to staff email addresses is restricted to only those working on the THV CQIC Process Study project.
Assurances of Privacy
We will inform respondents of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As the contract specifies, the contractor will comply with all federal and departmental regulations for private information. The contractor shall ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
Data Security and Monitoring
JBA has an established firm-wide System Security Plan that assesses all data security measures and monitoring procedures to ensure secure storage and transmittal of information. This plan is updated at least annually.
Primary data collected for the THV CQIC Process Study will be stored
on a secure OneDrive site. Secondary data will be stored on a secure
SharePoint site. JBA is a subscriber to the FedRAMP ATO-holding
Microsoft Online 365 Service with both Business and Enterprise
licenses. JBA maintains multiple SharePoint and OneDrive sites to
separate data between projects and access requirements within those
projects. In addition to operating with Microsoft best practices for
security, SharePoint and OneDrive will use the following additional
controls that fall within JBA responsibilities for management.
Additional documentation pertaining to the security of SharePoint and
OneDrive can be found within Microsoft’s approved FedRAMP
package.
Access Controls
JBA SharePoint and OneDrive requires users to authenticate using multi-factor authentication for all users.
JBA SharePoint and OneDrive uses role-based access permissions to limit access to sensitive data and separate access based on assigned roles.
Only Administrators have access to modify the security policies, sharing permissions or role-based access permissions.
Permissions granted to a user account are based on the principal of least privilege so that users are not afforded access to the system greater than their minimum requirements.
Passwords used by user and administrative accounts require a minimum of 16 characters and must be complex, meaning that they must contain at least one number, one capital letter, and one symbol.
JBA only permits users with a valid account access to the JBA managed SharePoint sites. Anonymous or sharing links are prohibited.
JBA SharePoint Online is hosted on the FedRamp approved Microsoft Office 365 SharePoint online service. Microsoft controls remote access to the SharePoint Platform. JBA controls user access into the JBA owned and operated sites. Security related to the transmission to and from SharePoint online is documented in the Microsoft FedRamp package available to the government at https://www.fedramp.gov.
A11. Sensitive Information 2
The information collection does not include sensitive information. The project team will seek approvals or exemptions from JBA’s IRB. JBA will also consult with participating tribal grantees to determine whether review by local tribal IRBs is needed. JBA will engage local tribal IRBs after OMB approval is received in order to avoid asking tribal IRBs to re-review study materials if there are changes required by OMB.
A12. Burden
Explanation of Burden Estimates
Survey respondents include all members of the THV grantee teams participating in the CQICs. We have estimated that up to 130 grantee staff will be invited to take the survey. To estimate this number, we reviewed data on the number of THV staff members employed at each grantee agency, as well as the number of staff vacancies submitted in the THV Quarterly Form 4 (OMB-0970-0525). The web-based survey instrument is expected to take 10 minutes to complete per response.
Table 5. Estimated Annualized Burden and Costs to Respondents
Instrument |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
Instrument 1: Survey of THV CQIC Participants |
130 |
1 |
.17 |
22 |
$36.76 |
$808.72 |
Explanation of Cost Estimates
Costs were estimated based on the job code is 21-1093, Social and Human Service Assistant. Wage data from 2020 is $18.38 per hours. To account for fringe benefits and overhead the rate is multiplied by two which is $36.76.
https://www.bls.gov/oes/current/oes211093.htm
A13. Costs
There are no additional costs to respondents.
A14. Estimated Annualized Costs to the Federal Government
Table 6. Estimated Annualized Costs to the Federal Government
Cost Category |
Estimated Costs |
Stakeholder Engagement |
$1,704 |
Survey Development |
$14,200 |
Obtaining OMB Approval |
$12,070 |
Obtaining IRB Approval |
$40,328 |
Survey Administration and Data Analysis |
$4,260 |
Developing Report of Findings |
$11,360 |
Total costs over the request period |
$83,922 |
A15. Reasons for changes in burden
This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).
A16. Timeline
Data collection will take place following IRB (if needed) and OMB approval for one month. The web-based survey will be administered in December of 2021.
Table 7. Timeline
|
Begins |
Complete |
Administer Survey |
December 2021 |
December 2021 |
Analysis of Survey Data |
January 2022 |
February 2022 |
Collection and Analysis of Secondary Data |
Currently in process |
April 2022 |
Developing Report of Findings |
June 2022 |
August 2022 |
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instrument 1: Survey of THV CQIC Participants
Appendix A: THV CQIC Survey Invitation Emails
References
Education Development Center (2020). Home Visiting Collaborative Improvement and Innovation Network 2.0 (HV CoIIN 2.0). https://hv-coiin.edc.org/sites/hv-coiin.edc.org/files/General%20CoIIN%20Fact%20Sheet%201.2020.pdf.
1 These interviews are not subject to PRA due to the number of respondents (fewer than 10).
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Denmark, Nicole (ACF) |
File Modified | 0000-00-00 |
File Created | 2022-02-23 |