PART B 4.25.19 clean

p art B

Assessing Evidence of Effectiveness in Adult Education: OMB Data Collection Package

April 25, 2019

Submitted to:

Institute of Education Sciences
550 12th Street, SW

Room 4105
Washington, DC 20004

Project Officer: Melanie Ali
Contract Number: 91990018C0057

Submitted by:

Mathematica Policy Research

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director: Peter Schochet
Reference Number: 50728.BY.T03.000.000

CONTENTS

PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 1

B.1. Respondent universe and sampling methods 1

B.2. Procedures for the collection of information 3

B.3. Methods to maximize response rates and minimize nonresponse 4

B.4. Tests of procedures or methods to be undertaken 4

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data 5

appendix a: feasibility study screener and interview protocol

appendix B: MATHEMATICA confidentiality statement

appendix C: advance mail

TABLES

B.1. Key members of the study team 5

B.2. Individuals who will oversee the collection and analysis of data for the study Assessing Evidence of Effectiveness in Adult Education 5

PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This Office of Management and Budget (OMB) package requests clearance for data collection to support the study Assessing Effectiveness in Evaluation of Adult Education. The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) has contracted with Mathematica Policy Research and its subcontractors, Manhattan Strategy Group and Social Policy Research Associates (SPR), to conduct this study (Contract 91990018C0057).

The funding of adult education programs through the Workforce Innovation and Opportunity Act (WIOA) of 2014 is a key component of federal efforts to help low-skilled adults succeed in the workforce and become productive and engaged citizens. However, existing research on adult education provides little guidance on effective approaches for adult learners. As part of the congressionally mandated National Assessment of Adult Education, ED is implementing this feasibility and design study, which is intended to help fill this knowledge gap. The study begins with a feasibility assessment to determine whether specific adult education approaches could be rigorously evaluated at this time. If evaluations that could provide policy- and practice-relevant knowledge are determined to be feasible, ED may decide to conduct up to two evaluations in a subsequent phase of the National Assessment of Adult Education.

The study’s initial feasibility assessment, the focus of this clearance package, will determine whether specific adult education approaches prioritized by ED and aligned with WIOA priorities can be evaluated using rigorous study designs. The assessment will draw on interviews with directors of adult education programs that currently implement, or that are planning to implement, one of the prioritized approaches. If ED determines that impact studies are feasible, revised clearance packages will be submitted for data collections not covered under this request.

B.1. Respondent universe and sampling methods

The universe of potential respondents for this feasibility study includes adult education program directors at sites that are either using or planning on implementing one of the adult education program models, activities and/or services that we are considering for an impact study.¹ There are eight approaches that are being considered for an impact study: (1) integrated education and training (IET) through co-teaching, (2) IET through co-planning and alternate teaching, (3) on-ramp and bridge classes to prepare students for occupational skills training, (4) bridge classes that focus on college readiness, (5) career navigation supports, (6) distance learning, (7) blended learning, and (8) mobile or online learning tools to supplement instruction.

Since the feasibility study will not statistically sample adult education programs, we will not make statements that generalize beyond the programs in the study. The study team will identify 60 adult education programs that are thought to be currently implementing or would be interested in and able to implement one of the selected approaches to participate in an interview. We expect that 10 of these program directors will be screened out during the screening portion of the interview (see Sections A1 and B1 of the interview protocol in Appendix A) either because they are not implementing the approach as defined for the study or they are not interested in or able to implement the approach. The feasibility interview will be conducted with 50 program directors. The approach for selecting the programs will involve three key steps:

Identify adult education programs that are implementing or planning to implement an approach being considered. To identify these programs:

• We will begin by creating a list of potential respondents drawing on information collected through our team’s technical assistance work with adult education providers.

• To refine this list, we will solicit recommendations from our technical working group (TWG) members. We will ask for suggestions of providers who are implementing any of the approaches or are planning to implement one of the approaches and could potentially do so within the timeframe of the study.

• Once we compile a list of suggested programs, we will gather and record preliminary information available on program websites and state adult education office websites to include:

  • Director name and contact information

  • Whether identified approach is implemented (if available)

  • Whether any other approaches of interest are implemented (if available)

  • Location of provider

  • Type of organization

Identify program directors to screen for eligibility. Using the list of programs we compile, we will prioritize 60 providers to contact for additional screening to determine if they are implementing or planning to implement one of the priority approaches. We will prioritize providers based on information collected in the previous step, considering the following factors when selecting programs:

• Similarity in focus and structure between current offerings and the prioritized approaches (if information is available from sources above)

• We will include programs in medium to large cities (likely the types of sites we would target for a future random assignment or regression discontinuity evaluation that relies on oversubscription), though programs in rural areas may be included as well.

• We will seek to include a range of organization types, so that we can understand how implementation and challenges vary across settings.

In this group of 60 programs, we will include 35 programs that are implementing at least one of the 8 approaches being considered for an impact study, and 25 programs that are planning to implement a prioritized approach. The list will include at least 4 programs implementing and 3 programs that are potential implementers for each approach being considered for an impact study. A list of 15 alternates will be created to replace any program directors that decline to participate in an interview.

Begin each call with the program director with a 5-minute phone screening. Each phone call will begin with a short set of questions (Sections A1 and B1 of the interview protocol in Appendix A) to confirm that one of the approaches is offered for sites identified as current implementers, or for programs planning to implement an approach, that they have thought enough about implementation to answer questions about the feasibility of studying the approach. We anticipate that the study team will need to conduct a screening call with up to 60 program directors. Assuming that 10 will be screened out and thus not complete the feasibility interview, we expect to complete 50 feasibility interviews.

B.2. Procedures for the collection of information

The study team will conduct the screening and feasibility interviews with program directors in summer 2019. The interviews will help to narrow the range of adult education approaches for possible inclusion in impact evaluations, shape the study design options for each approach (e.g. random assignment, regression discontinuity), including data collection that will be requested in later clearance packages, and gauge program directors’ interest in participating in a possible study. This information will be included in the study design report.

We will interview the program directors by phone. This mode allows the interviewers to work around respondents’ schedules and to complete the interview over multiple sessions if necessary. The interview protocol is included in Appendix A.

a. Statistical methodology, estimation, and degree of accuracy

The data collected from interviews will be qualitative information related to the feasibility of conducting rigorous evaluations of adult education approaches. The study team will conduct the screening and feasibility interviews with purposefully identified program directors; no sampling will be used to create the sample of respondents.

Similarly, no statistical methods of estimation will be needed in the analysis of the interview data. The qualitative data collected will be analyzed and reported in a memorandum of findings prepared by the study team for ED. Because of the purposive selection of programs, the findings from the interviews will not be generalizable to all adult education programs.

b. Unusual problems requiring specialized sampling procedures

The study will not use a probability sample for either the selection of adult education programs or the selection of respondents within the selected programs. There are no unusual problems requiring specialized sampling procedures.

c. Periodic data collection cycles to reduce burden

The study team will conduct the screening and feasibility interviews with each selected program director only once. To reduce burden on respondents, the team will review program websites prior to the interviews to collect any publicly available information about the program. The team’s experience in conducting interviews and in developing protocols will ensure that data are collected during the interviews such that the burden on program staff will be minimal.

B.3. Methods to maximize response rates and minimize nonresponse

This section describes our methods for maximizing response rates and minimizing nonresponse in our interviews of adult education program directors.

Notifications and establishing legitimacy. To encourage participation by adult education programs, the study team will conduct informal outreach to state adult education directors encouraging them to reach out to providers in our sample that are located in their states to encourage participation and establish legitimacy of the study. Next, the study team will contact program directors by telephone informing them about the evaluation and requesting their participation in this feasibility study. When email addresses are available, we will send an informative email (Appendix C) to all potential respondents in which we will explain the importance of this study, the expected burden for the interview process, and the value of their participation.

Flexibility in schedule. We will then conduct outreach via telephone calls or emails to schedule a telephone meeting with program directors at their convenience. To ensure full cooperation from the program directors, the study team will be flexible in scheduling interviews to accommodate the needs of respondents. There might, however, be instances when a respondent is unable to meet; when this happens, a member of the study team will ask to meet with the respondent’s designee or schedule a follow-up call at a more convenient time.

Reminders and continuing outreach. We will conduct reminder calls and send brief reminder emails to program directors from whom we do not receive a timely response. These calls and emails will stress the importance of being interviewed by the study team, provide concrete steps for doing so, and suggest available times and dates for them to select.

Site coordination. All the adult education programs selected for the feasibility study will be assigned a liaison to the study team. The liaison will be responsible for conducting all outreach efforts for a particular site, thus simplifying the participation and messaging for each site.

B.4. Tests of procedures or methods to be undertaken

The interview protocol to be used for the feasibility study has been reviewed by content and methodological experts to ensure clarity and optimal ordering of the questions.

Additionally, wherever possible, the interview protocol was developed from existing, proven items from other similar studies. Questions from similar feasibility studies were tailored to the adult education context and the specific needs of the current feasibility study. The topics covered by these questions include features of the programs, target populations, assessments, current scope of the programs, and interest in participating in a rigorous evaluation.

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data

Key members of the study team are listed in Table B.1.

Table B.1. Key members of the study team

Name and role	Affiliation
Melanie Ali, Contract Officer Representative Marsha Silverberg, Associate Commissioner	Institute of Education Sciences Institute of Education Sciences
Peter Schochet, Project Director Alina Martinez, Deputy Project Director Julie Bruch, Feasibility Task Lead Lisbeth Goble, Survey Director	Mathematica Policy Research Mathematica Policy Research Mathematica Policy Research Mathematica Policy Research
Alisa Belzer, Lead Adult Education Content Expert	Rutgers University
Laura Lanier, Co-Principal Investigator	Manhattan Strategy Group

Staff responsible for overseeing the collection and analysis of data are listed in Table B.2.

Table B.2. Individuals who will oversee the collection and analysis of data for the study Assessing Evidence of Effectiveness in Adult Education

Name and role	Affiliation
Peter Schochet, Project Director Alina Martinez, Deputy Project Director Julie Bruch, Feasibility Task Lead	Mathematica Policy Research Mathematica Policy Research Mathematica Policy Research
Laura Lanier, Co-Principal Investigator	Manhattan Strategy Group

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Seattle, WA ■ TUCSON, AZ ■ Washington, DC ■ Woodlawn, MD

1 Hereafter, the adult education program models, activities and/or services that the feasibility study will explore are referred to as approaches.