Att_ESL OMB Clearance Part A -not tracked Revised 7.23.08

This is the second of two clearance requests submitted to OMB to carry out data collection activities for the Adult ESL Literacy Impact Study, formerly known as the Adult ESL Explicit Literacy Impact Study. OMB has approved the following data collection instrument, which is described in the study’s first OMB submission for the screening phase of the study (OMB No. 1850-0811):

ESL Site Screening Protocol

This second OMB submission requests approval for the following collections during the full impact study:

Teacher Data Form (2008);
Teacher Data Form (2009);
Program Intake Form;
Student Background Interview; and
Daily Student Attendance Sheets.

A.1 Explanation of Circumstances that Make Collection of Data Necessary

Study Background

Recent statistics from the U.S. Department of Education (ED) show that more than 40 percent of the over 3 million students in federally funded adult education program are English as a second language (ESL) students (ED, 2004). These students—most of whom are immigrants and refugees—represent a wide range of nationalities and cultural backgrounds. Although a considerable number of these ESL students come from Mexico and other Hispanic regions in the Americas, fairly recent research shows that significant numbers also come from Africa, India, the Philippines, China, Vietnam, and Eastern Europe (Wrigley, Richer, Martinson, Kubo and Strawn, 2003).

Many of these ESL students have had limited educational opportunities or come from preliterate cultures, and consequently have not developed basic foundational reading and writing skills in their native language. They struggle with the dual challenge of acquiring literacy skills as they endeavor to develop communicative competence in English.

The Adult ESL Literacy Impact Study is a national evaluation designed to test the effectiveness of a promising literacy workbook (Sam and Pat) in improving the English reading and speaking skills of adult English as a second language (ESL) literacy level learners. Sam and Pat uses a structured reading approach based on the principles of the Wilson Reading System and adapted for adult ESL literacy learners. The Wilson Reading System was evaluated through a randomized controlled trial with third and fifth graders and found to have significant positive impacts on measures of phonemic decoding skills.^¹ Based upon this study, the What Works Clearinghouse determined the Wilson Reading System to have potentially positive effects on alphabetics among elementary students.

The study is the first rigorous evaluation of a literacy workbook designed for adult ESL literacy-level learners, who lack English language skills and who have limited literacy skills in their native language. The study is sponsored by the Institute of Education Sciences within the U.S. Department of Education and conducted by the American Institutes for Research (AIR) and its partners, The Lewin Group, Berkeley Policy Associates (BPA), Mathematica Policy Research (MPR), and Educational Testing Service (ETS).

The proposed study will focus on three key research questions:

How effective is instruction based on the Sam and Pat workbook in improving the English reading and speaking skills of low-literate adult ESL learners?
Is Sam and Pat more effective for certain groups of students (e.g., native Spanish speakers)?
Do Sam and Pat teachers engage in more structured literacy instruction than control teachers (e.g., is there a “service contrast”), and are impacts greater where the group contrast is greater?

The study will provide practitioners with valuable information on the effectiveness of a literacy workbook designed to improve the skills of this challenging population.

Study Overview

The Study Design

The research team, in consultation with experts in the fields of research design, adult ESL, and literacy instruction, has designed a randomized field trial to examine the impact of literacy instruction based on the workbook Sam and Pat, along with the professional development needed for the teachers to implement the instruction. The final design of the impact study will be contingent on the nature of the sites participating, but the initial design calls for approximately:

10 adult education programs;
40 teachers (4 in each program); and
1,800 low-literate adult ESL learners (180 in each program).

Within each program, teachers and students will be randomly assigned to one of two conditions:

The Sam and Pat condition, which will include a minimum of 60 hours of Sam and Pat-based instruction per term, with any remaining class time being spent on the types of instruction usually provided by the program.
The control condition, which will consist of the instruction usually provided by the program.

Teachers/classes within each program site will be randomly assigned in pairs, so that each pair of experimental and control classes meets at the same time, in the same building, and for the same number of hours.

Across the study sites, the total number of class hours will vary, and may range from approximately 60 to 225 total hours, depending on the programs’ course schedules.

The study will take place during the fall term of 2008 and the winter term of 2009, and will include a different cohort of students each term (approximately 900 students per term).

The Literacy Instruction Materials

A critical component of this study is the use of instructional materials appropriate for literacy instruction in the literacy-level adult ESL classroom, with corresponding support to monitor and train teachers to implement the study-related instruction faithfully. Few examples of literacy texts for the study population exist to choose from, however. To identify the intervention to be tested, study staff first held a competition open to the public, and solicited proposals from publishers, ESL programs, and other developers. When no bids were received, study staff began reviewing “off the shelf” ESL texts and evaluated them for appropriateness for the study population. Ultimately, study staff solicited a bid from the developers of a text called Sam and Pat, which was designed for the target population and used extensively in Massachusetts. Instruction based on the Sam and Pat literacy workbook published by Thomson Heinle is systematic, direct, sequential, and multi-sensory. It also includes multiple opportunities for practice with feedback. It was written by three teachers with extensive experience with adult ESL literacy students—Jo Anne Hartel, Betsy Lowry and Whit Hendon—who will also provide the treatment teachers with three days of intensive training on implementation of Sam and Pat prior to the fall 2008 term. This training was piloted with six teachers in the spring of 2008. The teachers found the information learned during the training to be very useful, and gave it positive evaluations. Minimal changes are planned for the full study training, based on teacher suggestions. In addition, there will be follow-up classroom observations of the treatment teachers conducted by the developers, with feedback on implementation provided to the teachers. There will also be a two to three hour refresher Webinar training held for all treatment teachers before the spring term begins.

Data Collections

The data collections for which we seek clearance in this package include:

Teacher Data Form (2008): A brief data form to collect background information from teachers;
Teacher Data Form (2009): A brief data follow-up form to collect information on teachers’ professional development and instructional experiences;
Program Intake Form: A brief web-based student data form filled out by program staff;
Student Background Interview: Interview with students to verify background information; and
Daily Student Attendance Sheets: Daily attendance (yes/no) of students to estimate their exposure to instruction in the treatment and control groups.

More information on these collections is provided in Section A.2.

Additional data collections associated with the study include:

Classroom observations conducted by study staff to document treatment fidelity and describe the instruction that occurs in treatment and control classrooms;
Native Language Literacy Locator: A brief native language literacy assessment used by program staff to determine students’ native language literacy level and place them into a class;
Standardized pre-testing of English reading ability (measured with the Woodcock-Johnson III® Tests of Achievement (WJ III® ) or similar assessment), and speaking/listening ability (measured with the WJ III® and Oral and Written Language Scales (OWLS) listening subtest or similar assessment), administered by study staff; and
Standardized post-testing of English reading (measured again with the WJ III® or similar assessment) and speaking ability (measured with the Receptive One-Word Picture Vocabulary Test (ROWPVT) or similar assessment), also administered by study staff.

Flow of Study Activities

The Adult ESL Literacy Impact Study activities are summarized in Exhibit A.1. Before student intake and random assignment occurs, teachers selected by the programs to participate in the study will complete the Teacher Data Form (2008 version; see Appendix A) and be randomly assigned to condition. Teachers assigned to teach Sam and Pat will attend a three-day training delivered by the workbook authors in the summer of 2008.

Exhibit A.1. Study Activities Flow Chart

Prior to the beginning of the fall 2008 and winter 2009 terms, students (approximately 900 students each term) will apply for the program as they normally do, following the program’s usual intake procedures and assessment to determine appropriate placement. Since we will include only programs that have a managed enrollment policy or that have open enrollment that is time limited, we expect the student intake to begin about 2–4 weeks prior to the start of class and end no later than 2 weeks after classes start. Program staff will then enter students’ intake information into the study’s secure intake and random assignment Web-based database (see Appendix B).

If the program staff determine during intake that a student belongs in a literacy level class, and the student is at least 18 years old, the student will be considered eligible for the study. The intake staff will explain the study to the student and obtain informed consent (Appendix C) translated into the student’s native language. We will collect no further data on students who choose not to participate in the study, but they will be allowed to take the class(es) for which they are registered by the program. In programs with literacy level classes that are not participating in the study, the program will assign the non-consenting students to non-study classes. In programs where all classes at a site are participating in the study, non-consenting students will be assigned to a study class by the program, according to the programs’ normal procedures for assigning students to classrooms. In these instances, the programs will be asked to ensure that treatment and control classrooms remain roughly equal in size. Non-consenting students will not be included in the study’s data collections.

Eligible consenting students will be randomly assigned to either the Sam and Pat condition or the control condition on the first day that they report to the program for class. During the first week of attending class, each student will be asked to participate in a background interview in their native language (see Appendix D) and in a battery of standardized English literacy and speaking/listening pre-tests. Teachers will take daily attendance throughout the study period (see Appendix E).

During each instructional term, study staff will observe each class one to two times. The purpose of the observations is to document fidelity of implementation in the treatment classes and contamination of the control classes, as well as to collect descriptive data about the instruction provided to both groups.

At the conclusion of each instructional term, both groups of students will be assessed on standardized measures of English reading and speaking. Teachers will also be asked to fill out an additional teacher data form (2009 version; Appendices F.1 and F.2 for the Sam and Pat and control group versions) to give the study team feedback on their training and instructional experiences during the study.

A.2 How The Information Will Be Collected, by Whom, and For What Purpose

The data collections for which we seek clearance include the following:

Teacher Data Form (2008);
Teacher Data Form (2009);
Program Intake Form;
Student Background Interview; and
Daily Student Attendance Sheets.

Teacher Data Form (2008)

The 2008 Teacher Data Form (Appendix A) will be utilized to collect background information about the study teachers, including: teacher credentials, educational background, years of overall and ESL teaching experience, and demographics. The form will be administered via an e-mail link to the study’s Web-based database during summer 2008. Teachers’ background information will be used for descriptive purposes, and to ensure that the teachers assigned to the treatment and control classes are equivalent across these characteristics.

Teacher Data Form (2009)

The 2009 Teacher Data Form (Appendices F.1 and F.2) will be utilized to collect information on study teachers’ professional development experiences from summer 2008 through spring 2009 and other information related to implementation of literacy instruction within the classroom. The form will be administered via an e-mail link to the study’s Web-based database at the conclusion of the winter/spring 2009 term. There are two versions of the data form—one specific to Sam and Pat teachers, and one for the control group teachers.

Program Intake Form

The Program Intake Form (Appendix B) collects basic background information about students (names, contact information, years of prior education, etc.), and represents the kind of information typically collected by programs. For the purposes of the study, intake staff will be asked to enter this information into the study’s Web-based database.

Student Background Interview

The Student Background Interview (Appendix D) is designed to verify information about students’ educational, demographic, and language background. It will be administered in each student’s native language by study staff using a face-to-face format. The data will be used to describe the students participating in the study.

Daily Student Attendance Sheets

Daily student attendance sheets (Appendix E) will be filled out by teachers for each class period in order to provide the study with a measure of instructional “dosage”. It will also be used to track class entry/exit and any potential “crossover” of students between treatment and control classes.

Exhibit A.2 illustrates the data collection schedule for the study’s full set of data collections. Please note that while the same teachers will participate in both the summer/fall 2008 and winter 2009 data collections, there are two separate student cohorts (fall 2008 and winter 2009). Data collections involving these students will only occur once per cohort.

A.3 Use of Improved Information Technology to Reduce Burden

To take advantage of the fact that all programs have computers with internet connections that are accessible to teachers, we will administer the teacher data form via a simple e-mail link to a secure web-based data form. In addition, the Program Intake Form will be filled out by program staff by accessing the program-staff section of the study’s secure Web-based database. The student collections will be face-to-face administrations by necessity, given the literacy level of the students involved.

Exhibit A.2. Data Collection Schedule

Data Collection^²	Respondent	Summer 2008	Fall 2008	Winter 2009	Type of Data
Teacher Data Form (2008)*	Teachers	X			Descriptive information about teacher’s background
Teacher Data Form (2009)*	Teachers			X	Descriptive information about teachers’ professional development experiences and literacy instruction implementation
Native Language Literacy Locator	Students: Fall and Winter Cohorts^³		X	X	Assessment of native language literacy level to determine literacy level status
Program Intake Form*	Program Staff on Behalf of Students: Fall and Winter Cohorts		X	X	Student background information
Student Background Interview*	Students: Fall and Winter Cohorts		X	X	Descriptive information about student’s educational, language, and demographic background
Daily Student Attendance Sheets*	Teachers		X	X	Dosage/exposure to instruction and to track entry/exit from class
Classroom Observations	Study Staff		X	X	Implementation fidelity, control group contamination, descriptive information about instruction in both groups
English Literacy and Speaking/Listening Pre-Tests	Students: Fall and Winter Cohorts		X	X	Pre-test covariate data
English Literacy and Speaking/Listening Post-Tests	Students: Fall and Winter Cohorts		X	X	Outcomes data

A.4 Efforts to Identify and Avoid Duplication

There are currently no systematic collections of the types of data required by the study.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses or entities will be involved as respondents. Every effort will be made to minimize the burden on programs, students, and teachers participating in the study.

A.6 Consequences of Less-Frequent Data Collection

Each type of data collected in the Adult ESL Literacy Impact Study will only be collected as needed (usually once), and serves a vital role in the conduct of the study.

The need for research that will provide answers to the growing concerns about what works in adult ESL education is critical to the implementation of effective instruction. Understanding the usefulness of ESL literacy instruction for low-literate adult ESL learners is fundamental, not only for instructional practice, but also for the field of adult ESL education. Without this study, states and ESL programs will have a limited basis on which to address the growing need for scientifically based instructional methodologies. The data collection for this study will therefore fill an important gap in the adult ESL research literature.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

No special circumstances apply to this study.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

The 60 day federal register notice was published on February 11, 2008 by the Office of the Federal Register, National Archives and Records Administration (NARA). See Volume 73, No. 28, page 7721 of the Federal Register for more specific information. No public comments have been received regarding this submission.

The names and affiliation of the study’s technical working group are listed in Exhibit A.3.

The data collection instruments were developed by AIR, in collaboration with its subcontractors at Lewin Group, BPA, ETS, and World Education.

A.9 Payment to Respondents

The Adult ESL Impact Study involves the random assignment of teachers and students to treatment or control conditions. With a random assignment design, it is critical to maintain the integrity of the treatment and control groups and ensure equivalence between the two groups. This study’s ability to detect effects of the treatment will be compromised to the extent there is attrition of either the treatment or control group students, and especially if there is differential attrition. If a significant portion of either the treatment or control group teachers declines to participate in the study’s data collections, it will not be possible to conduct meaningful analyses based on “intent to treat,” since it is not possible to add new members to either group. To the extent that members of the treatment or control group are lost from the study’s data collections, the findings are biased, and study funds are wasted.

Exhibit A.3. Technical Working Group Members

Name and Title	Affiliation
Judith Gueron, Scholar in Residence	MDRC
Russell Gersten, Director, Instructional Research Group	RMC Research
Daphne Greenberg, Associate Director of the Center for the Study of Adult Literacy	Department of Educational Psychology & Special Education, Georgia State University
Larry V. Hedges, Board of Trustees Professor of Statistics and Social Policy, Faculty Fellow	Institute for Policy Research, Northwestern University
Robinson G. Hollister, Jr., Professor of Econometrics, Labor and Social Economics, and Health Economics Swarthmore College	Swarthmore College
Dr. Robin Scarcella, Director of the ESL Program & Lecturer of the ESL Program, Linguistics Department	University of California, Irvine
Dr. Anne Whiteside, Practitioner	City College of San Francisco

Achieving high responses rates for the current study will be a challenge. The study involves a lengthy and intense period of data collection. Relatively high student mobility among the adult ESL population and a largely part-time teaching force that is not compensated for time spent outside regular teaching hours will further impede the study’s ability to achieve a high response rate on all data collections.

To increase the study’s chances of achieving high response rates, we propose the incentive payments summarized in Exhibit A.4 for the teacher surveys and student pre- and post-tests. These payments were determined in accordance with the guidelines for incentives for NCEE evaluation studies (Guidelines for Incentives for NCEE Evaluation Studies, dated March 22, 2005), and are similar to amounts that have been approved for other recent impact studies.^⁴

The two Teacher Data Forms each require 15 minutes to complete and are therefore considered medium burden surveys. They are administered electronically, and high response rates are needed from both the treatment and control group. The planned incentive amount is $15 for each of the instruments; this level of incentive proved sufficent for similar instruments in the Professional Development Impact Studies (Reading and Math). We feel that this stipend is appropriate for the amount of burden we are placing on adult ESL teachers, who are usually part-time and hold multiple jobs with little to no paid prepration time. While it is true that there is no evidence on effective teacher incentive levels, nominal incentives (e.g., $10 in the Early Reading First Study) offered by NCEE teacher surveys have not yielded adequate response rates very efficiently. Gaps in data collection would affect the study’s power to detect possible effects, and therefore we offer a slightly larger incentive for the teacher surveys ($15). It should also be noted that we are not offering a stipend to teachers for turning in attendance forms or for allowing study staff to observe their instruction multiple times; therefore, despite the number of demands that participating in the study places on teachers, we are only offering a total incentive of $30 plus a book to Sam and Pat teachers, and $50 to control teachers.

Exhibit A.4. Proposed Data Collection and Other Payments

Instrument	Payment
Teachers
Teacher Data Form – Baseline (2008)	$15
Teacher Data Form – Follow-up (2009)	$15
Instructional materials stipend (for control teachers) or a copy of Sam and Pat materials (for treatment teachers)	$20
Students
Pre-Tests, for students who leave the program between the point of random assignment and the in-class administration of the pre-tests	$20
Post-Tests	$40

The student pre-test includes a battery of assessments. It will be administered on-site during class time and therefore presents a low level of burden to students. However, it is anticipated that some students (approximately from 5 to 10 percent) will withdraw from the program after random assignment has been conducted, but before the pre-test is administered. These students will need to travel to the test site. An incentive of $20 is planned for the pre-test for these students, to gain their cooperation with the pre-test and motivate them to participate in the post-test later in the term. Due to the nature of this low-literate ESL population, it is difficult to contact these students and gain their trust and cooperation. In an earlier study (the What Works Study), we were only able to get a 50 percent response rate from attriters who had similar characteristics and who were receiving $60 total for their study participation.

The student post-test also consists of a battery of assessments. The post-tests for students remaining in the program will be scheduled individually and administered at the program. The incentive planned for the entire battery of post-tests is $40, which is similar to, although slightly lower than, the $60 stipend offered to a similar population in the What Works Study, which resulted in a 70 percent response rate. This indicates that, if all other factors were held constant and there were no relationship between responding and receiving an incentive, the highest response rate we could get without an incentive is 70 percent. Given that we need a response rate of at least 80 percent, we need to do more, not less to ensure high response rates. Therefore, in addition to offering a similar incentive as the What Works Study, we are also taking steps within each study site to make the study as much a part of normal program activities as possible. For example, site staff (who students are familiar with) will guide students through the intake and random assignment process, and test administrators will be hired locally to administer tests onsite and will speak the students’ native languages.

We will also offer a small instructional materials stipend for the control teachers participating in the study ($20). The materials stipend is intended to gain the control teachers’ cooperation with study data collection activities, including the classroom observations and teacher data forms. The materials stipend for control teachers will allow them to buy instructional materials for their classroom unrelated to Sam and Pat. We feel that this materials stipend is important, because control teachers, after agreeing to participate, are by luck of the draw left with no motivation for participation in the evaluation other than the advancement of science. At this point, a control teacher is subject to the full data requirements and other inconveniences of the study (e.g., three classroom observations by study staff) and must agree not to adopt a similar intervention. An incentive to the control teachers, to be used for something unrelated to the outcome of interest, not only increases motivation to participate in the study, but also reduces the likelihood of a Hawthorne effect.

A.10 Assurances of Confidentiality

No information collected during the study that would identify individual respondents will be reported or published. An explicit statement regarding confidentiality will be communicated to all respondents. To ensure that the data collected are not available to anyone other than authorized project staff, a set of standard confidentiality procedures will be followed during the study:

All project staff will agree to an assurance of confidentiality;
All project staff will keep completely confidential the names of all respondents, all information or opinions collected during the course of the study, and any information about respondents learned incidentally;
Reasonable caution will be exercised in limiting access to data collected only to persons working on the project who have been instructed in the applicable confidentiality requirements for the project;
The Project Director will be responsible for ensuring that all contractor personnel involved in handling data on the project are instructed in these procedures and will comply with these procedures throughout the study; and
The Project Director will ensure that the data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. These include the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 that requires “All collection, maintenance, use, and wise dissemination of data by the Institute: to “conform with the requirements of section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In addition, for student information, the data collection efforts will ensure that all individually identifiable information about students, their academic achievements, their families and information with respect to individual adult education sites, shall remain confidential in accordance with section 552a of Title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act. The study will also adhere to requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.

In addition, the study’s databases will be maintained under strict security. The data collection subcontractor, Mathematica Policy Research (MPR), protects its local area network (LAN) with industry standard security mechanisms available through the network operating system and certified at the National Computer Security Center’s Trusted Network Interpretation Class C2 level at the network level. All LAN servers containing confidential information are located in a controlled-access area, and are protected by a firewall. Access to confidential information stored on LAN directories is restricted to authorized project staff by means of I.D. and password.

MPR also has taken steps to ensure security in its data transmissions. Internal e-mail and other transmissions are password protected. Electronic cross-office transmissions are executed through a dedicated T1 line that is protected by a vendor-maintained firewall. Confidential information transmitted via the Internet occurs either through the Netscape V5.1 built-in encryption procedures or password protected zip files.

During the course of the study, all necessary information and documents will be kept in a file or database accessible only by project staff under the supervision of the Project Director. After the project is completed, the contractors will destroy all identifying information.

The procedures and data collections of this study were reviewed and approved by AIR’s internal review board (IRB).

A System of Records (SOR) report and notice was published on April 25^th, 2008.

A.11 Questions of a Sensitive Nature

No questions of a sensitive nature will be included in the study’s data collections.

A.12 Estimates of Respondent Burden

The total estimated hours of burden for the data collection for the Adult ESL Literacy Impact Study is 966 hours, which represents an annual burden of 322 hours. The estimated annual cost of burden is $4939. Exhibit A.5 summarizes the estimates of respondent burden for the instruments used in this study. This burden estimate includes:

Time for 80% of 40 teachers to respond to a 15 minute teacher data form before the study
Time for 80% of 20 Sam and Pat teachers to respond to a 15 minute teacher data form (Sam and Pat teacher version) at the conclusion of the study
Time for 80% of 20 control group teachers to respond to a 15 minute teacher data form (control teacher version) at the conclusion of the study
Time for 80% of 1800 students to complete student data forms during program intake (15 minutes total; one administration per respondent)
Time for 80% of 1800 students to participate in a 15 minute student background interview at pre-testing (one administration per respondent)
Time for 80% of 40 teachers to complete daily attendance sheets (approximately 5 minutes per day) for each day of class for two instructional terms, assuming classes meet, on average, 3 times per week for 15 weeks for each term.

A.13 Estimate of Cost Burden to Respondents

There are no startup costs for this collection.

A.14 Estimate of Annualized Government Cost

The total cost to the Federal government for the Adult ESL Literacy Impact Study is $6,762,445, and the annual cost is $326,257 in Year 1; $929,170 in Year 2; $2,590,348 in Year 3; $2,368,878 in Year 4 (which includes much of the data collection); and 296,847 in Year 5. Approximately $1,030,860 will be used for all the study’s data collections. Of this amount, approximately $61,700 total will be used for the data collections for which clearance is currently being requested, which represents an average annual cost of $20,567 over three years.

Exhibit A.5. Time Burden for Respondents

	A	B	C	D	E	F	G	H	I
Task	Total Sample Size	Number of Responses (80% Response Rate)	Time Estimate (in Hours)	Hourly Rate	2008 Estimated Responses	2009 Estimated Responses	2010 Estimated Response	Average Annual Burden Hours B x C ÷ 3	Estimated Annual Monetary Cost of Burden D x H
Teacher Data Form (2008)	40	32	.25	$25	32	0	0	2.7	$68
Teacher Data Form – Sam and Pat Teacher Version (2009)	20	16	.25	$25	0	16	0	1.3	$33
Teacher Data Form – Control Teacher Version (2009)	20	16	.25	$25	0	16	0	1.3	$33
Program Data Form	Fall and winter cohorts: 1800 total	1440	.25	$12	720	720	0	120	$1440
Student Background Interview	Fall and winter cohorts: 1800 total	1440	.25	$12	720	720	0	120	$1440
Daily Student Attendance Sheets	40	2880	.08	$25	1440	1440	0	77	$1925
Total	3720	5824	n/a	n/a	2912	2912	0	322	$4939

A.15 Changes in Hour Burden

This collection is a reinstatement of expired collection 1850-0811 that had received approval for the screening and recruitment stage of the study. The current package requests approval for the implementation phase of the study, which represents an annual program change of 322 hours, for a total of 966 hours.

A.16 Time Schedule, Publication, and Analysis Plan

Analysis of Data Collected Through Surveys

Data collection for the Adult ESL Literacy Impact Study will begin in summer, 2008 and will end in June, 2009. Data collected from the forms included in this clearance request will be used primarily for descriptive purposes. AIR will report findings from the impact study in March, 2010. This report will include a description of the study design (i.e., treatments, sample size, study sites) and employed methodology.

The descriptive analyses will include the following:

Description of the participating programs;
Description of the teachers and students in the treatment and control groups;
Description of the treatment and control groups’ equivalence after randomization;
Descriptive information on the fidelity of implementation of the treatment;
Descriptive information on instruction in both treatment and control classrooms; and
Description of the rate of student and teacher mobility over the period following random
assignment.

The report will also provide results regarding the effects of the Sam and Pat treatment on students’ English reading and speaking skills. Because treatment groups are determined at the student level, the primary unit of analysis will be the student, with the difference between the average outcomes for students receiving the intervention and those randomly assigned to the control group representing a reliable and unbiased estimate of the intervention’s impact.

The basic strategy to estimate treatment effects in a random assignment study is to conduct a comparison of mean outcomes for treatment and control group members. That is, the treatment effect on an outcome Y is the difference between Y_p and Y_c, where Y_p is the mean outcome for the program group, and Y_c is the mean outcome for the control group. Written as a simple equation, and representing the treatment effect with a coefficient b_1, a simple unadjusted intervention effect could be estimated as follows:

(1)

where Y_ij is outcome Y for learner i in a class taught by teacher j, b₀ is an intercept, E_ij is the treatment assignment variable, which is 1 for treatment group members and 0 for controls, b₁ is the coefficient associated with variable E_ij, _j is a random error term representing the effects of teacher j, and _ijis a random error term representing variation in outcome among students within classes. To reduce the variation in _j and _ij and increase the precision of our estimate of b₁, we propose to add two sets of control variables X_i and Z_j to this model, as follows:

(2)

where X_i is a group of individual-level background variables for the learners in the sample, and Z_j is a group of background variables for the teachers they are assigned to. An important subset of the variables Z_jwould be a series of dummy variables identifying each of the individual programs.^⁵ All of these background variables are measured before individual teachers and learners are randomly assigned and are unaffected by the random assignment. Including them in the estimation of program effect b₁ reduces their potential effect on this coefficient and also reduces the size of the error terms _jand _ij (i.e., the variance of _jand _ij in Equation 2 will be smaller than in Equation 1). Thus, for example, if random assignment produced a situation in which treatment group teachers were less experienced than control group teachers on average, our analysis would remove this random difference from the impact estimates, provided that Z_j includes a measure of teacher experience. Similarly, if through random variation, ESL learners in the program group were somewhat older on average than ESL learners in the control group, this problem would be addressed by the inclusion of the learners’ ages among variables X_ij. The statistical significance of treatment effect b₁ is then determined by conducting a t-test on coefficient b₁in a multiple regression framework.

Because the number of teachers being randomly assigned to the enhanced ESL program is so much smaller than the number of students randomly assigned, the potential for a “bad draw” is greater among the teachers. Therefore it is very beneficial for the study to collect detailed background data on teachers prior to their assignment and to include these data in the analysis. One major advantage of including teacher-level control variables is that it reduces the effective intra-class correlation in the sample, which significantly increases the study’s statistical power.

A.17 Display of Expiration Date for OMB Approval

All data collection instruments will include the OMB expiration date.

A.18 Exceptions to Certificate Statement

No exceptions are requested.

1 Torgesen, J., Myers, D., Schirm, A., Stuart, E., Vartivarian, S., Mansfield, W., et al. (2006). National assessment of Title I interim report—Volume II: Closing the reading gap: First year findings from a randomized trial of four reading interventions for striving readers.

2 Data collections for which clearance is requested in this package are indicated with an asterisk (*). The remaining data collections are either standardized assessments or measures that are only filled out by study staff.

3 Data collections involving students occur one time for the fall cohort and one time for the winter cohort.

4 For example, the Impact Evaluation of the DC Opportunity Scholarship Program and the Professional Development Impact Studies (Reading and Math), three recent U.S. Department of Education evaluations, offered payments similar to those proposed in this package.

5 These so-called “program fixed effects” would correct our estimates for differences across program sites in factors that might affect learner outcomes.

2 American Institutes for Research®

File Type	application/msword
File Title	Adult ESL Impact Study
Author	Anestine Hector-Mason
Last Modified By	katrina.ingalls
File Modified	2008-07-25
File Created	2008-07-25