Supporting Statement B - AMCS Study - Formative generic

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Assessing Models of Coordinated Services for Low-Income Children and Their Families

OMB Generic Information Collection Request: Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

July 2019

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Laura Hoard and Ivelisse Martinez-Beck

Part B

B1. Objectives

Study Objectives

The purpose of the AMCS project is to understand how states and local communities are coordinating services related to early care and education (ECE), family economic security, and/or other health and human services across sectors to most efficiently and effectively serve low-income children and their families. As noted in section A2, information gathered on existing coordinated services approaches through these generic clearance activities will inform future field work, in addition to informing technical assistance for Preschool Development Grant Birth to Five (PDG B-5) grantees. Coordinated services approaches are of keen interest to ACF leadership, as well as the broader field of human services. As such, this study seeks to provide timely description of state and local coordinated services approaches that are currently operating in the United States. These generic clearance activities will address the following research questions through staff verification and completion of state and local model profiles, their provision of an organizational chart and staffing information, and state and local telephone interviews:

• RQ2: How do coordinated services approaches intend to reduce barriers and road blocks for families to access [early care and education, family economic security, and/or other health and human] services? Are there federal barriers to implementing such approaches?

• RQ4: What have we learned from efforts to integrate enrollment and eligibility processes for health and human services?

• RQ5: Are states and/or localities examining service delivery dynamics across ECE programs to assess availability of care slots and services to meet the needs of eligible families? How are they using data to understand service delivery dynamics?

• RQ6: How is public and private ECE funding targeted to meet the needs of at-risk children and families? Are there differences in the families that are able to access services?

The full set of study research questions is listed in Table 1 of section A2.

Generalizability of Results

This data collection is intended to present an internally valid description of coordinated services approaches at the state and local level in chosen sites. Data on program effectiveness will not be collected, and the information will not enable statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The goal of this data collection is to produce accurate summaries of current coordinated services approaches, in a uniform format. The descriptive information collected under this ICR will inform site selection for in-depth follow up and provide timely information about current coordinated services approaches for ACF and its grantees. Qualitative data serve the purposes of the study well, because this is an initial effort to understand coordinated services approaches and identify how far states and localities are in their efforts to coordinate.

B2. Methods and Design

Target Population

The target population includes coordinated services approaches operating at the state or local levels. To be included in the AMCS study team’s national scan, coordinated services approaches must coordinate early care and education services with family economic security and/or other health and human services. Under this ICR, staff operating and working on a purposive sample of coordinated services approaches identified in the national scan will be invited to engage in data collection activities.

Sampling and Site Selection

While the national scan relied on publicly available information and is therefore not subject to the Paperwork Reduction Act, we have included a description to aid understanding of the data collection activities included in this generic information collection package.

Sources for identifying state and local coordinated services approaches through the national scan included web searches; information from related projects; relevant federal and state initiatives; relevant technical assistance (TA) initiatives; and private sector initiatives. All coordinated services approaches that met the following criteria were included in the national scan:

• Currently operates in the United States;

• Has a public website or public documents available for review;

• Serves at least some low-income families;

• Directly provides early care and education services for children age 5 and younger;

• Provides family economic security and/or family-focused health and human services in addition to early care and education; and

• Intentionally coordinates multiple health and human services.

From the universe of state and local coordinated services approaches identified using the six criteria listed above, we will identify purposive samples for each of the data collection efforts as follows:

• State and local model profile confirmation and completion. We will create state and local model profiles using the criteria above for a purposively selected set of state and local coordinated services approaches using publicly available data. This purposive sample will be selected to reflect the variation and features across state and local coordinated services approaches, while at the same time minimizing duplicative information from multiple coordinated services approaches that are using the same, or a similar, approach. We will then invite coordinated services approaches to confirm and complete the state or local model profile, as relevant, for their coordinated services approach. The budget for this information collection allows us to create and confirm/complete model profiles for up to 76 state and local coordinated services approaches. As necessary, we will follow-up with phone calls to prompt staff to confirm and complete the model profile data. The budget allows us to follow up by phone with up to 30 state and local coordinated services approaches. If more than 30 model profiles have missing information, we will target follow up phone calls based on ACF’s interest in diversity of features represented in coordinated services approaches.

• Request for staffing information and state and local telephone interviews. From the set of state and local coordinated services approaches included in the state and local model profile confirmation and completion activity, we will select a purposive sample of 20 coordinated services approaches from which to collect staffing information and the approach organizational chart and to invite staff to participate in a group telephone interview. We will ask for an organizational chart of the approach (if one exists) and staffing information from approaches who are invited to participate in a telephone interview. These 20 state and local coordinated services approaches will be selected based on what we learn through the publicly obtained data in the national scan, as well as during the model profile confirmation and completion. The AMCS study team will aim to maximize variation in features of the coordinated services models (for example, geographic location, population served, funding type(s), coordination structure, services provided) while also considering what coordinated services approaches might be best able to provide information to answer the study’s research questions (see section B1). We will use the following criteria to help determine the sample of 20 coordinated services approaches:

  • The responsiveness of state and local coordinated services approaches to the model profile confirmation and completion;

  • State and local coordinated services approaches that seem to be “working” (that is, services are being coordinated and people are using those coordinated services);

  • The degree to which state and local coordinated services approaches have any evidence or data about outcomes;

  • The degree to which state and local coordinated services approaches include features that are of particular interest to learn more about in eventual site visits (for example, involvement of the private sector or having well-integrated eligibility processes); and

  • Variation in geographic locations and populations served across all coordinated services approaches selected.

To account for the possibility that not all 20 state and local coordinated services approaches identified will be willing to participate, we will create a list of 10 additional state and local coordinated services approaches that will serve as replacements. If possible, we will choose replacements with similar characteristics to the state or local coordinated services approach that they are replacing. If the list of 10 replacement approaches is exhausted, we will determine whether additional replacement sites are useful and feasible to pursue.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

The AMCS study team has modeled the state and local model profile templates, staffing information template and state and local telephone interview protocols on instruments used in other studies with similar populations¹. We tailored the instruments to state or local coordinated services approaches so that questions would be relevant to efforts operating at the respective level. Our approach to pre-populate the state and local model profile templates with publicly available information on coordinated services approaches will help to streamline the data collection effort and only request information that cannot be found elsewhere. We solicited feedback on the profile templates from stakeholders with expertise in coordinated services approaches at the state and local levels and from ACF colleagues who work with similar populations. Their feedback was used to refine the state and local model profile templates. We do not plan to further pretest the templates or protocols.

B4. Collection of Data and Quality Control

The AMCS study team will collect data for each component of this information collection request in the following way. All referenced outreach materials are included in the relevant instrument documents:

Confirmation of state and local model profile data (Instruments 1 & 2). After the AMCS study team has compiled draft state and local model profiles using publicly available information, the AMCS study team will email a contact person from the state or local coordinated services approach (using email addresses identified through public sources). The draft profile will be attached to the email. We will invite state and local coordinated services approach staff to add any information that was not found during the review of publicly available information and to correct any information in the state or local model profile that is inaccurate. We estimate that the time to confirm and complete the information would be up to two hours.

If the contact person at the state or local coordinated services approach does not respond to the initial email request to confirm and complete the state or local model profile, we will send up to two additional email follow-up requests. For a subset of approaches (up to 30) that are under consideration for engaging in a telephone interview, the AMCS study team will follow up with a phone call. In that phone call, AMCS study team members will describe the request for state or local model profile confirmation and inquire about the appropriate person to consult. We will ask select questions from the model profile while on the call and then re-send the state or local model profile and ask that staff at the state or local coordinated services approach confirm the information (or complete missing information) and return it to our AMCS study team. We estimate that the follow-up phone call will be approximately 30 minutes. We will make several attempts to reach the right person through the phone call.

Request for staffing information and state and local telephone interviews (Instruments 3, 4 & 5).

Once the sample for the state and local telephone interviews has been identified, the AMCS study team will first send an email to state and local coordinated services approaches describing the telephone interview and describing the staffing and organizational chart information request. The AMCS study team will follow the email with a phone call to the director of the state or local coordinated services approach (or the person who confirmed the state or local model profile if that is not the director). In the phone call, the AMCS study team will describe the goals of the telephone interview and coordinate a time for the telephone interview should the state or local coordinated services approach accept the invitation to participate. State and local coordinated services approaches will be asked to send the AMCS study team an organizational chart of their coordinated services approach (if they have one) and to identify two to five staff they think would be appropriate to include in the telephone interview. The person identifying these staff will be asked to provide their names, job titles, and tenure with the organization or agency on a form to be returned (Instrument 3) along with the organizational chart.

State and local telephone interviews will be scheduled for groups at a time that is convenient for the state or local coordinated service approach staff and will last approximately one hour. We anticipate one group interview per coordinated services approach. The AMCS study team will use the state or local telephone interview protocol (Instrument 4 or Instrument 5, respectively) to guide the semi-structured small group telephone interview. All state and local telephone interviews will be conducted by two AMCS study team members, one asking questions and a second typing notes, capturing key quotes and responses. With permission from respondents, the AMCS study team members will use an audio recorder to record telephone interviews in order to later confirm direct quotes and other details from the telephone interviews.

Procedures to protect the privacy of information are described in section A10.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Confirmation of state and local model profile data

We expect to send 76 state and local model profiles to state and local coordinated services approaches for confirmation and completion. It is important to note that we are using purposive sampling that is not intended to be representative and we will not be making statistical inferences from the data. We will pursue approaches using the methods described in section B4 to try to maximize response rates.

Request for staffing information and state and local telephone interviews

We have grouped these data collection activities together because we will only request staffing information from state and local coordinated services approaches that agree to participate in a telephone interview. The 20 approaches we contact to participate in the telephone interview will be purposively selected from the group of 76 approaches. If approaches refuse to participate in the telephone interview we will contact approaches from the replacement group (as described in B4).

Non-Response

The AMCS study team will use a mix of strategies to ensure high response rates for data collection. A mix of email and telephone outreach will provide different ways for state and local coordinated services approaches to respond to requests to confirm state and local model profile data, share staffing information and participate in state and local telephone interviews. AMCS study team members will explain the goals of AMCS and our interest in learning more about the potential participants’ coordinated services approach.

Several other specific strategies include:

• State and local model profile templates include “not applicable” and “I don’t have this information” response options to mitigate nonresponse to questions that are not relevant to respondents.

• The scheduling of telephone interviews will be flexible, to account for the availability of key state and local coordinated services approach staff.

During the state or local telephone interview itself (based on our experience asking similar questions with similar populations) the AMCS study team does not anticipate significant item nonresponse to interview questions. Interview questions have been tailored to coordinated services approaches operating at the state or local levels, respectively. Interviewers will be trained to use appropriate probes and prompts to encourage responses to all questions. Additionally, we do not expect questions to be sensitive for participants; thus, we expect they will feel comfortable answering all questions.

B6. Production of Estimates and Projections

No estimates or projections will be produced based on this work.

B7. Data Handling and Analysis

Data Handling

Item nonresponse: State and local model profile templates include “don’t know” and “not applicable” response options to aid in the interpretation of any blank cells in the profile. The AMCS study team will review the state and local model profiles after they have been confirmed. For the group of approaches invited for telephone calls, if any fields on their profiles are left blank, we will follow up during the telephone interview to ask them to provide the information needed to complete those fields.

We do not anticipate that state and local coordinated services approaches will have time to respond to all of the telephone interview questions included in the protocol within the hour long interview. Questions are listed in the order of priority, such that interviewers can follow the protocol linearly and stop when the interview time is up.

Data entry

State and local model profiles will be reviewed upon receipt and any typographical errors will be corrected. If needed, information provided by respondents will be formatted (e.g., we will add bullets or group into paragraphs) to facilitate clarity; no substantive information provided by respondents will be altered. Profiles will be incorporated into the model scan summary document and will not be entered into a database.

Telephone interviews (20) will be audio recorded and these recordings will be used as needed to supplement, correct, or clarify interviewers typed notes. Typed notes will then be reviewed by senior project staff to identify any missing information or areas lacking clarity. After this review, notes will be imported into a qualitative coding software (NVivo) for analysis and use in the follow-up study, to be approved in a future ICR.

Data Analysis

Analysis Plans

Using the completed model profiles, will respond to the study’s research questions by developing cross-case summary information that includes counts and descriptive information about the range of state and local coordinated services approaches currently in operation and features that are commonly present based on the national scan.

Once interview notes have been organized, we will do a thematic coding of responses based on the AMCS research questions. Trained AMCS team members will review the information gathered during the interviews and identify common themes or features across approaches. Themes will be determined through meetings with the research team. Themes will focus on program implementation and might include topics such as efficiencies realized through coordination, barriers to coordination, or ways in which staffing and service delivery operate within a coordinated services approach. The AMCS study team will code information from the state and local telephone interviews using a qualitative coding software. During coding, a senior member of the team will review the coding to ensure consistency in how the codes are applied to the data. We will provide descriptive information about the number of times interview participants describe particular coordinated services approach features and then summarize the qualitative information, organizing the summary by topic. We will summarize both the primary themes for each state or local coordinated services approaches’ interview and themes across interviews.

We will use the state and local model profiles and state and local telephone interview information from specific state or local coordinated services approaches to identify approaches to recommend for future site visits. Coordinated services approaches will be recommended for site visits with the goal of identifying sites most likely to provide the most in depth answers to the study research questions (for example, approaches that coordinate many services or that have been in existence long enough to know whether their approaches have worked). Priority will also be given to coordinated services approaches operating at the state-level as this will have the most direct applicability to technical assistances efforts for PDG B-5 grantees.

We will include both the summary of themes and the site-visit recommendations in an internal document.

Please see Section A.16 for information on the schedule for tabulation and reporting based on these findings.

Data Use

Findings from the state and local model profile confirmation and state and local telephone interviews will be used to provide technical assistance and inform subsequent field work covered under a forthcoming full information collection request.

B8. Contact Person(s)

OPRE

Laura Hoard, Ph.D.

[email protected]

BSC

Tracy Carter Clopet, Ph.D.

[email protected]

Mathematica Policy Resesarch

Elizabeth W. Cavadel, Ph.D. (Project Director)
[email protected]

Attachments

INSTRUMENT 1: Confirmation of State Model Profile

INSTRUMENT 2: Confirmation of Local Model Profile

INSTRUMENT 3: Request for Staffing Information

INSTRUMENT 4: State Telephone Interview Protocol

INSTRUMENT 5: Local Telephone Interview Protocol

5