SSB - Formative Generic Program Support - THV CQICs Process Study

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Tribal Continuous Quality Improvement Collaboratives Process Study

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

June 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Nicole Denmark, OPRE

Part B

B1. Objectives

Project Objectives

The purpose of this generic information collection (GenIC) is to help assess the feasibility and appropriateness of the Tribal Home Visiting Continuous Quality Improvement Collaboratives (THV CQICs), an adaptation of the Breakthrough Series Collaborative model, as implemented in ACF’s Tribal Home Visiting Program (THV). This GenIC is part of a larger process study that will help ACF’s THV program office assess whether to include this type of CQIC approach in the ongoing learning agenda and to plan for future technical assistance needs.

Generalizability of Results

The process study is intended to present an internally-valid description of the implementation of the THV CQICs not to promote statistical generalization to other sites or service populations. Results are not intended to be generalizable beyond the group of 19 grantees participating in the THV CQICs as part of their grant activities.

Appropriateness of Study Design and Methods for Planned Uses

Collaborative participants will be invited to participate in a web-based survey on their experience with and perspectives on the THV CQICs. This survey will produce mostly quantitative data that can be analyzed for each collaborative and staff role. Data from the study is not intended to be representative of the population outside of individuals participating in the THV CQICs. The descriptive process study will provide data on participant perspectives but will not provide information on participant outcomes and is not intended to be an impact evaluation. The survey relies on self-report instead of objective measurement; therefore, survey findings may be subject to social desirability bias. Supporting Statement A provides details on the study design for the overarching process study and describes how the survey, which is the focus of this GenIC, fits into the broader study and other data collection activities. The limitations of the data will be described in any resulting materials.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The target population includes the cohort of 19 Tribal MIECHV grantees who are participating in the Tribal CQICs as part of their grant activities. Grantee staff members from the 19 participating grantees will be invited to complete the survey.

Sampling

The findings from this GenIC and the larger process study are not intended to be generalizable, therefore probability sampling is not a priority. We will not sample from amongst the target population but rather invite all staff who were members of their CQI teams during the collaboratives to take the survey. We will invite staff from the entire cohort of grantees to complete the survey because we expect grantee teams to have different experiences due to their unique community and program contexts. Additionally, all program staff from each site will be asked to participate because we anticipate that staff across grantee CQI teams who are in different roles (i.e., program manager, home visitor, evaluator) will have differing perceptions of the feasibility and applicability of the THV CQICs to their work. We also expect that staff will have various levels of longevity being a participant in the collaborative and therefore may have a broad range of experiences with the collaborative methods (e.g., meeting in-person versus virtually due to the pandemic). Allowing all staff the opportunity to complete the survey will support the notion that everyone’s perspectives and viewpoints are valuable to the study and will contribute to the findings.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument

The survey instrument was developed based on a thorough review of existing Breakthrough Series Collaborative and Home Visiting Collaborative Improvement and Innovation Network (CoIIN) literature. Survey items from previous evaluations were reviewed for relevance and compatibility with the study aims and evaluation questions. Selected survey items were tailored to fit the THV CQICs context using terms familiar to THV grantees. Additional survey items were developed to measure constructs not previously assessed in the literature.

The survey domains and themes were reviewed, and feedback was gathered from THV grantee representatives, ACF program office staff, and contracted TA providers. The instrument was pilot tested with members from the contractor’s team who were not involved in the CQICs for readability and structure.

B4. Collection of Data and Quality Control

James Bell Associates (JBA) (i.e., contractor) will administer the survey electronically through Qualtrics web-based survey software. Each staff person will receive an invitation to participate in the survey via email sent directly from Qualtrics (Appendix A). Email invitations will be addressed to each staff person by name and each individual will receive a unique survey link. If required by IRB, informed consent will be obtained electronically before the respondent proceeds to the survey questions. Respondents can complete the survey at a time that is convenient for them, including over multiple sittings, if needed. By using unique survey links, we can troubleshoot any challenges that individual respondents may have accessing the survey and schedule email reminders to only those staff who have not completed the survey. We will send up to 2 email reminders (Appendix A) to staff who have not started or completed the survey.

We will monitor data collection for quality and consistency through weekly data quality check reports. These reports identify duplicates, missing values, validations, date inconsistencies and valid ranges. Team members will follow up on any quality issues identified in the reports with the respondent, if appropriate.

B5. Response Rates and Potential Nonresponse Bias

The survey is not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Staff from all Tribal Home Visiting grantees participating in the THV CQICs will be invited to participate in the survey.

Response Rates

Although findings are not intended to be representative, we will assess response rates qualitatively. We will document respondent type (e.g., program director, home visitor) in written materials associated with the data collection. We will calculate the response rate by dividing the number of survey responses received by the number of survey invitations sent to unique individuals. Only staff members currently employed by the grantee will be included in the calculation. We will send two reminder emails to staff who have not responded to the survey or who started the survey but have not finished. The survey invitation will communicate the value of respondent input and the role feedback will play in informing future efforts.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Once the data have been received in Qualtrics, we will utilize SAS to process and clean the data. This involves deleting fake/test data, renaming variables, converting character variables to numeric, cleaning dates and cleaning data entry errors. Next, we will remove duplicates, recode missing variables, create clean variables, assign labels and recode write-ins. Finally, we will create any needed analysis variables.

Data Analysis

We will calculate frequency distributions to summarize patterns across survey items and to examine variability in the data. We will produce descriptive statistics to summarize variances and means for relevant items and groups of items. We will examine the survey data across all 19 grantees, as well as by collaborative, to learn more about grantee perceptions from the two collaboratives. Data will be analyzed by staff role as well.

Data Use

We will not create a public-use file based on the information collected.

Information collected may be incorporated into documents or presentations that are made public such as through conference presentations, websites, or social media. The following are some examples of ways in which we may share information resulting from these data collections: technical assistance plans, presentations, infographics, project specific reports, or other documents relevant to stakeholders such as federal leadership and staff, grantees, and/or T/TA providers. In sharing findings, we will describe the project methods and limitations with regard to generalizability and as a basis for policy.

B8. Contact Person(s)

JBA is conducting this information collection and developed the plans for data collection in collaboration with ACF. For questions around how data will be collected and analyzed, please contact Julie Morales, [email protected].

Attachments

Instrument 1: THV CQICs Process Study Survey

Appendix A: Email Invitation and Reminder Text

4