Part B Justification 041613

CNCS’ contractor, NORC at the University of Chicago, will collect information for the Process and Impact Evaluation of the Minnesota Reading Corps (MRC) on behalf of CNCS. The contractor is responsible for the design and administration of the follow-up web-based survey of AmeriCorps members, as well as the collection and analysis of literacy assessment data from preschool students’ for the impact evaluation.

As presented in Part A, the data collection is being conducted for two phases of the project, the process assessment and Quasi-Experimental Design (QED) evaluation of the MRC PreK program and include the following data collection instruments to be reviewed:

Follow-up Member Survey: A follow-up survey with members (tutors) of the 2012-13 MRC program. An earlier round of the study included a baseline member survey. Approximately 1,031 subjects will be asked to participate in the survey.
PreK Assessments: The Individual Growth and Development Indicators (IGDI) assessment will be used to measure emergent literacy skills. The IGDI will administered to1,440 preschool students three times during the 2013-2014 school year: fall, winter and spring.

Minnesota Reading Corps (MRC) is the largest AmeriCorps State program in the country. For the 2012-13 school year, the MRC program plans to serve over 30,000 students in 652 elementary schools, Head Start centers, and preschools using more than 1,000 AmeriCorps members. For the member survey, the entire universe of AmeriCorps members serving in the 2012-13 school year will be invited to complete a baseline and follow-up survey (1,031). For the evaluation of the MRC PreK program, a representative sample of 40 MRC PreK programs from across the state of Minnesota was selected to participate in the PreK evaluation. In addition, 40 PreK sites that do not participate in the MRC PreK program were selected to serve as comparison sites for the PreK evaluation because they match the MRC sites on several key variables. More detail on the selection of MRC and non-MRC comparison sites is provided below. It is estimated that the 40 MRC and 40 comparison schools have more than enough students enrolled to meet the sample size requirements for the study of 1,200 MRC PreK students and 1,440 comparison students.

B.2.Information Collection Procedures

Individuals who served as an MRC literacy tutor in the 2012-2013 school year will be asked to complete a follow-up member survey. CNCS will send an advance letter by Federal Express and by email that conveys the importance of the data collection effort and contains a user ID and password that asks members to complete the survey in June of 2013. If they are interested in completing the survey, they can go to the URL for the survey’s home page. The first screen they see will ask them to enter a user ID and password to complete the survey. Upon doing so, the next screen provides a brief overview of the study, asks for their voluntary participation, informs participants about confidentiality and privacy, provides a frequently asked questions link, provides a link to a detailed description of the project, and provides a toll-free telephone number and email address if participants have any questions about the survey. By clicking "Next" at the bottom of the consent screen, the survey participant is providing their voluntary consent to participate in the survey. Actual time to complete the web-based follow-up survey may vary. However, on average, it will take approximately 20 minutes to complete based on experience with the baseline survey. Survey participants who complete the member survey will not receive any form of monetary or tangible compensation for their participation in the study.

Per NORC Internal Review Board (IRB) requirements for the protection of human subjects research in educational settings, letters informing parents about the PreK evaluation will be sent home to students parents at the beginning of the 2013-2014 school year. Parents who do not want their children participating in the evaluation will contact the evaluation team via email, phone, or writing and their children not be assessed during the evaluation. Excluding those students whose parents withdraw them from participation, the contractor’s trained field staff will administer the IGDI assessment to 1,440 students in 40 preschools not participating in the MRC program during the 2013-2014 school year. The IGDIs measure five critical emergent literacy skills: 1) rhyming 2) alliteration, 3) picture naming, 4) letter name fluency and 5) letter sound fluency. Each of the five IGDI measures takes less than 2 minutes to administer for a total assessment time of 7 minutes per administration. Field staff will conduct the five IGDI measures with a student over the course of two days so that no one student will spend more than 6 minutes of assessment time with a field staffer at any one time. Field staff will collect IGDI data three times over the course of the 2013-2014 school year: fall (September/October 2013), winter (January/February 2014) and spring (May/June 2014). Data from the assessments will be recorded on paper record sheets, and will then be entered into a centralized, web-based computer assisted data entry (CADE) system, which allows for secure data entry, data storage, and follow-up.

Since the contractor’s field staff will collect IGDI data directly from PreK students over the course of the school year, it will be necessary to obtain students’ names during data collection. To ensure confidentiality, each student will be assigned a unique identification number, which will allow the researchers to evaluate student growth over the course of the 2013-2014 school year. All assessment forms and the CADE system will use only the unique identification number.

B.2.1. Statistical Methodology for Stratification and Sample Selection

Member Selection for Follow-up Survey. All members who served as tutors for the 2012-2013 MRC program will be asked to participate in the follow-up member survey, or approximately 1,031 subjects. All respondents will be over the age of 18. No students will be part of any phase of the data collection. Respondents will be male and female and will consist of all races and ethnicities.

Sample Selection for PreK Evaluation. Forty PreK sites that do not participate in the MRC PreK program (i.e., comparison sites) will be selected to serve as comparison sites for the PreK evaluation. Identification of comparison sites involved a four step process. First, 40 MRC PreK programs from across the state of Minnesota were selected to participate in the PreK evaluation. These programs represented the diversity of PreK programs in operation statewide. Three variables were considered in selecting the 40 MRC PreK sites: 1) Urbanicity (urban, suburban, rural); 2) Program Type (Public, Head Start, Community-based); and 3) Ages Served (4/5 year olds only or a mix of 3-5 year olds). The second step in identifying the 40 comparison sites was to find sites that matched the MRC sites on these three variables (i.e., Urbanicity, Program Type, and Ages Served). In the third step, those PreK sites where teachers had previously received SEEDS training were excluded from eligibility as a comparison site for the evaluation. These sites were excluded because SEEDS training is a major component of the MRC PreK program. The fourth step was to identify the best match among the remaining schools on the following five important site characteristics, in order of importance: 1) percentage of students eligible for Free or Reduced Price Lunch; 2) percentage of students who were English Language Learners (ELL); 3) enrollment size; 4) student to teacher ratio; and 5) licensed enrollment capacity.

The table below provides an estimated timeline of data collection activities.

Activity	Estimated Start Date	Estimated End Date
Launch AC member follow up web-survey	June 2013	July 30, 2013
Send Reminder Letter/Emails	June 2013	July 2013
Phone Prompting 1	July 2013	July 2013
Phone Prompting 2	July 2013	July 2013
Complete PreK design, sampling and recruitment for feasibility study	January 2013	April 2013
PreK data collection	September 2013	June 2014

B.2.2. Estimation Procedure

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Sample Size Needs.

An important consideration in developing a sampling plan for the PreK evaluation was the number of students required to detect a difference between the program and comparison groups. The outcome used to assess the effect of the program will be student assessment scores on the five IGDI measures. To determine the number of sites to sample, a power analysis was conducted using the student and school population numbers reported by MRC from the 2009-2010 school year. Assuming a Minimal Detectable Effect (MDE) of 15% based on findings from previous one-group pre-post test evaluations of the program, the study requires 1,200 students from 40 MRC PreK sites and 1,440 students from 40 non-MRC comparison sites. This analysis assumed an intraclass correlation coefficient (ICC) of 0.15, based on current literature on ICC values for sample designs with school-based clusters.

For the web-based survey, a targeted sample size of 1,031 was selected to include members serving as MRC tutors in the 2012-2013 school year. The sample size, which will be representative of the population, will then be used to generate frequencies and means. . The baseline survey achieved a 68 percent response rate; therefore, we predict that a similar percentage will respond to the follow-up survey. This response rate is considered to be a reasonable estimate given the experience with the baseline survey, along with the fact that the survey takes a short amount of time to complete (20 minutes), respondents are well-immersed in the use of email and the web, and there is a high-level of enthusiasm among MRC tutors. We have added some additional strategies in an attempt to increase the response rate to 80 percent (including advanced letters being sent via Federal Express). However, if a lower than 80 percent response rate results, non-response bias tests will be conducted to determine if any bias resulted from the lower response rate. If these tests provide evidence of bias, statistical adjustments to the results will be made with the use of weight adjustments and/or response imputation.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

The IGDI assessments will be administered at the beginning (fall), middle (winter) and end (spring) of the 2013-2014 school year. The follow-up member survey is a one-time data collection.

B.3. Methods to Maximize Response Rates

A 68 percent response rate for the web-based survey is anticipated. However, the follow-up survey will entail a number of additional measures in an effort to increase this response rate to 80 percent, particularly in the form of the initial advance letter. The follow-up survey will first send a personalized advance letter on CNCS/MRC letterhead by Federal Express, immediately followed by email, ensuring high visibility for the initial contact and conveying the importance of the data collection effort to the MRC program and to efforts to replicate it in other states. The first reminder will take the form of a letter mailed on CNCS/MRC letterhead, and subsequent reminders will be sent by email. Lastly, telephone prompting will be used to follow up with remaining non-responders. This response rate is considered to be a reasonable estimate based on the response rate achieved for the baseline survey, and also because the survey takes a short amount of time to complete (20 minutes), respondents are well-immersed in the use of email and the web, and there is a high-level of enthusiasm among MRC tutors. However, if a lower than expected response rate results, non-response bias tests will be conducted to determine if any bias resulted from the lower response rate. If these tests provide evidence of bias, statistical adjustments will be made to the results with the use of weight adjustments and/or response imputation.

To attain a high response rate, CNCS will send an advance letter by Federal Express and by email to those individuals who are serving as MRC literacy tutors containing a user ID and password that asks members to complete the survey in June of 2013. This letter will convey the importance of the survey and how the information will be used. If they are interested in completing the survey, they can go to the URL for the survey’s home page. The first screen they see will ask them to enter a user ID and password to complete the survey. Upon doing so, the next screen provides a brief overview of the study, asks for their voluntary participation, informs participants about confidentiality and privacy, provides a frequently asked questions link, provides a link to a detailed description of the project, and provides a toll-free telephone number and email address if participants have any questions about the survey. After two weeks and three weeks, reminder invitation letters and emails will be sent to respondents who have not yet responded to the survey reminding them of the importance of their participation, why the results are important and how they will be used, and including the information necessary for them to complete the survey. At four weeks, reminder telephone calls will be placed to prospective respondents who have not yet responded to the survey. During the final two weeks of data collection, prospective participants who have still not yet responded to the survey will be called again, prompting them that the survey is about to end and that their participation is very important. Interviewers highly experienced with gaining cooperation will be used to make these calls.

For the PreK evaluation, IGDI assessment data will be collected three times a year (fall, winter, spring) from 1,440 students enrolled among 40 preschools. A 10 percent student attrition rate is expected due primarily to families moving to new locations. Of those students available for assessment, at least 95 percent of students are expected to complete the assessments. Those few students who do not complete the assessment typically have attention difficulties or behavior problems. The primary concern for the PreK evaluation could be school attrition; that is, schools deciding not to allow field staff to come back to collect IGDI data from students later in winter or spring. To guard against this possibility, preschools that agree to participate in the evaluation will be offered two incentives contingent upon completion of all three waves of data collection: 1) access to all data on all assessed students at the end of the 2013-2014 school year, and 2) priority consideration should the preschool choose to apply to participate in the Minnesota Reading Corps program in the 2014-2015 school year. These benefits have proven to be powerful incentives when recruiting preschools.

B.4. Tests of Procedures

The member follow-up survey instrument has been drafted and has undergone two reviews: (1) an internal review conducted by NORC’s Institutional Review Board and (2) a pre-test with five AmeriCorps members. In order to accurately determine the burden placed on respondents as well as further test the clarity of the survey questions, a pretest was conducted in which a total of five members from diverse backgrounds responded to the survey to assess the reliability of the instrument. Slight revisions were made to the order and wording of a small number of questions based on comments received from both of these reviews.

Modifications to the length, content, and structure of the baseline and follow-up member surveys were made based on the results of the survey pretest interviews. Respondents provided generally positive feedback indicating that they could readily answer the questions and that the time to complete the survey was not onerous (approximately 20 minutes).

The web-survey instrument has been drafted, reviewed and approved by all project staff and has undergone an internal review conducted by NORC’s Institutional Review Board. After programming the instrument for the web, the survey instrument will undergo beta testing by all project staff prior to the launch of the questionnaire.

The IGDI assessment is a widely used, reliable and research validated measure of preschool students’ emergent literacy skills. It is currently in use at all MRC PreK sites and has been used by MRC since the program was founded in 2003. The IGDI will be administered as recommended by the publisher without modifications. Per MRC’s recommendation based on experience implementing the IGDI, the contractor will administer the five IGDI measures over the course of two days, three on day one and two on day two.

B.5. Statistical Consultants

The information for this study is being collected by NORC, a research organization, on behalf of CNCS. With CNCS oversight, the contractor is responsible for the study design, instrument development, data collection, analysis, and report preparation.

The instrument for this study and the plans for statistical analyses were developed by CNCS and its contractor. The staff team is composed of Dr. Carrie Markovitz and Dr. Marc Hernandez, Co-Principal Investigators and a team of senior-level staff including Dr. Carol Hafford, Task Leader for the Process Evaluation and Heidi Whitmore, Task Leader for the Web Survey. Contact information for these individuals is provided below.

Name	Phone Number
Carrie Markovitz, PhD	301-634- 9388
Marc Hernandez, PhD	773-256-6152
Carol Hafford, PhD	301-634- 9491
Heidi Whitmore, MS	763-478-6725

Page | 6

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	REQUEST FOR CLEARANCE FOR
Author	DHHS
File Modified	0000-00-00
File Created	2021-01-29