REL-Southeast Response to OMB Questions on AMSTI Study
Part A:
You’ve indicated that participants will receive an email invitation to the survey and the emails will come from their consent forms.
Please clarify how the participants receive the consent forms?
Response: Consent forms are sent to school principals and the principals provide time during a staff meeting for teachers to read the forms, ask questions, and complete the forms.
Please explain what procedures will be used if a participant does not provide an email?
Response: A hard copy of the survey will be faxed to the school. If the teacher is not reachable by fax, then the survey will be sent by U.S. mail.
Are there mechanisms for participants to complete the survey if they do not have computer access?
Response: Yes, a hard copy of the survey will be faxed to the school. If the teacher is not reachable by fax, then the survey will be sent by U.S. mail.
Are the emails personal emails, business emails, or both?
Response: The teachers choose the e-mail addresses they prefer for us to use.
Please explain how the “previously constructed lists” of participants are developed,
Response: School principals provide a list of all of their math and science teachers in grades 4-8.
You’ve indicated that only four monthly data collections will be possible, please explain the time constraints of the study.
Response: The decision to implement only four monthly surveys was in recognition of the OMB process and how long it might take. We were attempting to include as many data collection opportunities as possible being mindful of the OMB process. We were also trying to be mindful of the teachers' time burden, and we did not want to survey teachers after April when they would be occupied with standardized testing and end-of-year activities.
Please clarify how it was determined that 42 teachers (from the universe of 324) would be selected for the classroom observations.
Response: It was decided to select a sample of teachers (n = 42) for the classroom observations from the universe of 324 teachers (who will be surveyed) to reflect resources available and to add to data obtained from the teacher survey. Teacher/classroom observations are resource-intensive, requiring school-site visits by dyads of trained data collectors. It was determined as too costly and impractical to conduct observations of the universe of teachers. The observations will be critical in obtaining in-depth qualitative and quantitative data regarding AMSTI implementation and teachers’ instructional practices. The observational data will provide a level of detail about instructional practices (e.g., objectives addressed, presence of authentic instruction principles, classroom context), student engagement, and use of materials not obtainable from the teacher survey. In addition, the observations will be conducted by objective, well trained data collectors. In contrast, the teacher survey relies on self-report, and may reflect some degree of self-report bias. The observational data will therefore also be used to confirm teacher survey self-reports regarding their instructional practices.
It was decided to select 42 teachers in order to ensure that 3 observational data points for each grade and subject strata are covered; i.e., 4th – 8th grade inclusive; 5th and 7th grade science, in both the AMSTI and control conditions. Three observational points per strata is considered as necessary to identify patterns within the data. Thus, we will collect observational data on 21 teachers/classrooms in each of the AMSTI and control school groups.
You’ve indicated that the information in the principal interviews will “provide critical information for improving the teacher survey questionnaire.” Please explain how this information will be incorporated. OMB needs to review, and approve, the final teacher survey questionnaire.
Response: Upon further consideration, we have decided that we will not make changes to the surveys based on the principal interviews.
Were any public comments received?
Response: We have not received any public comments.
Please explain if the incentives are a one-time incentive or will occur multiple times throughout the study period.
Response: Teachers will receive yearly stipends for participating in the web-based surveys. In addition, each time teachers are observed and/or interviewed, they will receive stipends. Since we will collect data at each school for two consecutive years, each teacher will receive two survey stipends and up to two observation/interview stipends.
Please explain why the paper documents need to be kept for 10 years.
Response: We retain paper documents for ten years in case questions arise about the accuracy or validity of the data. Paper documents are kept in locked cabinets and only staff who are employed in this project have access to the cabinets.
Part A indicates that this data collection is being collected under CIPSEA. Per the draft CIPSEA guidance, the RELs do not qualify to collect data under this authority. Please provide the appropriate confidentiality authority for this collection.
Response: The REL-SE and its partners, Empirical Education, the Academy for Educational Development, and Abt Associates will conduct research in compliance with the Privacy Act of 1974 (5 usc 552 a), the Freedom of Information Act (5 usc 522), the Confidentiality Provisions of The Education Sciences Reform Act (20 usc 9573), and the Family Educational and Privacy Rights Act (20 usc 14). Study participants will receive information about confidentiality protections at the beginning of each data collection instrument or prior to an observation or interview. Respondents will be informed that all of the information they provide will be kept strictly confidential, and that the identities of students, teachers, principals, and schools will be protected.
All employees and contractors in the organizations involved in this study are required to sign confidentiality agreements when they are engaged by their organizations. In addition, all employees and contractors who work on the AMSTI project will sign a confidentiality agreement in terms of the specifics of the AMSTI study.
The Institutional Review Board at the University of North Carolina at Greensboro (UNCG) has provided clearance for this project, including all of the consent forms and protocols. All employees and contractors who work on this project have signed UNCG ‘s Confidentiality Agreement for Research Involving Human Subjects.
The participants’ assurance seems more suited to a private survey than a federal survey.
Response: We have changed the assurance to the following:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx (expiration date: xx/xx/xx). Public reporting burden for this collection of information is estimated to average about xx minutes, including the time to review instructions, gather the data needed, and complete the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please contact: [email protected]. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Department of Education, 50 North Ripley Street, PO Box 302101, Montgomery, AL 36104. If you have comments or concerns regarding the status of your individual submission of this form, e-mail directly to: [email protected].
Response: Personally identifiable information in this questionnaire will not be released to anyone or any organization, except as required by law.
The project schedule you’ve provided indicates that this is a three-year project; however, the package describes this as a five-year study. Please clarify.
Response: Data collection for the original study begins during the 2006-2007 school year and takes three years. Data collection for the replication study begins during the 2007-2008 school year and takes three years. When we include time for sample selection, randomization, planning and report writing, the study requires five years.
Part B:
Are you hoping to generalize the results of this study beyond the three regions for which you are sampling? There selection appears to be more of a convenience sample rather than a random sample.
Response: The sample is not a random sample. The results can be generalized to other schools that represent similar demographics, test scores, and other similar school conditions.
You’ve referenced an “original study” please explain what this refers to.
Response: We are asking for OMB clearance to conduct two studies. We are referring to the experimental study beginning during the 2006-2007 school year as the “original study” and to the experimental study beginning during the 2007-2008 school year as the “replication study.” The original study begins this current school year in three regions of Alabama. The replication study begins next school year in two different regions of the state. The methodology is identical for the two studies.
Is this a request for approval for the replication study or the original study?
Response: It is a request for approval for both studies, which use the same methodology to answer the same research questions during two different time periods in different regions of the state.
Have you already selected the sample, recruited the schools, and assigned them to treatment groups?
Response: There was no need to recruit schools. The schools all applied to be part of the AMSTI program. We have already selected the sample and randomized which schools take part in the treatment during the first year of each study and which schools serve as control schools. With permission from the National Center for Education Research program officer, Empirical Education conducted the randomization for the original study using funding from its research grant prior to SERVE’s award of the Regional Educational Laboratory-Southeast contract in which this study was proposed.
You are predicting a response rate of over 90%, please explain the rationale for that expectation.
Response: Teacher response rates are expected to reach about 90%. REL-SE partner, Empirical Education Inc. (EEI) has considerable experience deploying web-based surveys. Last academic year, 2005 – 2006, EEI deployed on average 8 surveys each to 206 teachers located at 12 different sites across the United States. The average response rate for a 13 item survey was 86.92%, and for a 29 item survey was 95%. Response rates were calculated using a simple percentage calculation based on the ratio of the number of actual received to expected responses. These surveys were deployed approximately every two weeks. EEI has improved our system of survey deployment and follow-up this year. We have a designated Survey Specialist who follows up with teachers who have not returned surveys with e-mails and phone calls beginning 2 to 3 days after deployment. Teachers respond well to our personalized attention.
The package indicates that much of the data collection will cover the full universe of schools yet Table 1 implies that a subset of 40 schools is selected for the study. Please clarify.
Response: This may have been a misunderstanding about the meaning of “full universe.” The original study contains 40 schools and the replication study contains 40 schools, which are a subset of all of the schools in their regions. All of the teachers and principals in the study will participate in the web-based surveys, and all of the principals will participate in the interviews. A sample of the teachers will participate in the classroom observations and interviews.
Please provide information on how the data collection instruments will be administered.
Response:
Web-based survey: Participants receive an e-mail inviting them to participate in the AMSTI web-based survey. The invitations contain a personalized web link for each participant as well as information on how to contact the Survey Specialist at Empirical Education if needed. When participants click on the web link, they open the survey and their identities are encoded in their responses. Once the participants complete their surveys, their information is automatically uploaded onto Empirical Education’s server. Participants who do not respond within two to three days receive a reminder e-mail which again contains the personalized web link. After one more day, non-respondents receive a fax. One day later the Survey Specialist calls each non-respondent and then continues to follow-up with non-respondents until they respond.
Classroom Observation: Two classroom observers will take notes during the observation period, and, as soon as possible after the observation fill out a classroom observation protocol. In this protocol, observers are prompted to provide 1) descriptive information about the class, such as the setting, decorations and seating arrangements, as well as grade level and estimated ethnicity and gender composition of the class, 2) narrative descriptions of the activities that took place during the observed lesson, 3) a rating of the extent to which cognitive domains such as knowledge and comprehension were emphasized in the lesson, and 4) a rating of the extent to which principles of authentic instruction were followed (e.g., connection to students’ out-of-school experience), and ratings on the extent to which technology and manipulatives were used. The observers will discuss their ratings, resolve any differences, and submit their final ratings for processing and analysis. Observers received extensive training in the use of this instrument. It is expected, consequently, that their observations focus on the information needs specified in the protocol.
Observers are instructed to enter the classroom at least five minutes before the observation is scheduled, and to explain the confidential nature of the observation and note taking as well as teachers’ option to decline being observed, or to terminate the observation at any point.
Teacher Interview: A teacher interview protocol will be used to conduct the teacher interview. For teachers whose classrooms were observed, the teacher interviews, in principle, are linked to the observation, and the observers will conduct the interview. The interview questions concern instructional practices, student responses to instruction and training/technical assistance needs. The teachers also are provided the opportunity to provide some general closing remarks.
Teachers will be asked to identify a convenient time after the observation (e.g., their planning period) in which to participate in the interview. Interviews will be conducted as soon as possible after the actual observation. The interviewers are instructed to strictly adhere to the questions listed on the protocol and conduct the interview within a 20-30 minute period. However, the probes for further questioning provided on the protocol are merely for guidance, as interviewers can use their own probes depending on the course of the conversation. It will be explained in advance to teachers that their responses are considered completely confidential. Before the interview, teachers are also asked to fill out consent forms.
The protocols vary somewhat according to whether interviewees are in the treatment or control condition. Interview questions for treatment condition teachers are concerned with their experience implementing AMSTI, while interview questions for control condition teachers are concerned with their experience implementing their usual math or science curriculum and materials. In both the treatment and control conditions, teachers that are observed also are asked an interview question regarding the lesson that was observed; this question is not included on the interview protocol used with non-observed teachers.
Principal Interview: Principal interviews are conducted in all schools (AMSTI and control) participating in the study. Two interviewers will participate in the interview, preferably the same data collection staff who conducted the classroom observations and teacher interviews. Interviewers are expected to follow the protocol exactly. We will schedule the principal interviews to follow the classroom observations and teacher interviews to facilitate discussion with the principal about observations of math and/or science instruction (AMSTI or control) implementation in her/his school. The interviews are expected to take approximately 20 minutes. Interview questions on the treatment condition protocol focus on the implementation of AMSTI, the principals’ perceived role therein, and the perceived adequacy of technical support and assistance from the AMSTI Master training and resource sites. Principals are also prompted to reflect on the successes and challenges of implementing AMSTI. Interview questions on the control condition protocol focus on the implementation of the school’s math and science curricula and instruction, perceived adequacy of support for the curricula and instruction, and the perceived adequacy of any technical support and assistance that has been provided. Before the interview, principals are also asked to fill out consent forms for data collection in their schools.
Training Logs and Surveys: The Summer Institute trainers are asked to complete a short checklist at the end of each day of training, containing basic information about day and time of the session, content covered, instructional strategies used, and materials used. The trainers are informed that the data collection is confidential and that their participation is completely voluntary and has no influence on any performance assessments of them. The daily logs are collected each day by the Master training site coordinators.
A ten minute retrospective pretest-posttest survey is administered to participants at the conclusion of the Summer Institute training. Research staff and/or Master training site staff will collect the surveys, informing the participants of the purpose of the survey, that the data collection is anonymous, and that their participation is completely voluntary and has no influence on any performance assessments. The survey prompts participants to indicate their content-specific knowledge before and after training, their perceived skill to teach in those areas before and after training, as well as their perceived preparedness to implement AMSTI.
Please explain how the initial frame is developed.
Response: The state of Alabama sent us a list of all of the schools in the regions that had applied to participate in AMSTI. They also sent the schools’ actual applications to participate in AMSTI, containing the teachers’ and administrators’ signatures. Researchers eliminated schools that did not contain grades 4 through 8. Researchers then paired the schools within each region that were most similar based on grade levels, test scores and demographics. In addition researchers chose the pairs that resulted in a sample that closely represented the demographics of each region.
Are the training participants met before the schools agree to participate? Please clarify the order of recruitment and selection into the study.
Response: Schools apply to participate in AMSTI in December. At least 80% of teachers and their principals must sign onto the application in order to qualify for the program. Then in January or February REL-SE partner, Empirical Education, works with the Alabama State Department of Education to form pairs of schools and to randomize schools into treatment or control status. Once the schools have been selected and randomized, the state superintendent asks all of the school system superintendents to sign agreements to participate in the program. Once the school systems have agreed to participate in the study, researchers e-mail each principal in order to do all of the following:
describe the study;
ask the principals to sign and fax back a consent form to participate in the study;
ask principals to take a web-based survey; and
ask principals to supply names of teachers.
REL-SE researchers follow-up the e-mails with telephone calls to discuss the study personally with each principal. Researchers then attend summer AMSTI training sessions to discuss the study with teachers and principals. After the training Empirical Education sends a packet of teacher consent forms to each school.
Please explain where the email addresses for principals come from.
Response: Principal e-mail addresses are on the Alabama State Education Department website.
There appears to be a heavy reliance on email and the internet for this study. Do you know if all schools and participants will have access to these resources? What are your options if they don’t?
Response: Yes, we confirmed with Alabama regional directors and their staffs that teachers have e-mail access. If a teacher does not have access, we will fax a hard copy of the survey to the school. If their fax machine is not available, we will send the surveys in the U.S. mail, but we do not believe this will be necessary.
Please provide information on how often the participants will be required to submit the instruments. Please explain what your follow-up procedures will be to ensure the expect response rate (over 90 percent).
Response: Participants receive an e-mail inviting them to participate in the AMSTI web-based survey. The invitations contain a personalized web link for each participant as well as information on how to contact the Survey Specialist at Empirical Education if needed. When participants click on the web link, they open the survey and their identities are encoded in their responses. Once the participants complete their surveys, their information is automatically uploaded onto Empirical Education’s server. Participants who do not respond within two to three days receive a reminder e-mail which again contains the personalized web link. After one more day, non-respondents receive a fax. One day later the Survey Specialist calls each non-respondent and then continues to follow-up with non-respondents until they respond.
Instruments:
The surveys refer to reference dates in 2006 (i.e., on the Teacher Survey #2, it states … since school let out in May or June 2006”), please explain the schedule for completion since it’s now March 2007.
Response: The surveys ask teachers for information on any training they have received since the summer of 2006.
Please clarify where the PRA blurb, OMB Control Number, and confidentiality pledge will appear on the forms?
Response: They will appear at the top of the forms.
AMSTI Teacher Classroom Observation
Descriptive Information, the race/ethnic breakdown needs to conform to the OMB standards.
Response: We have changed the descriptive categories related to race and ethnicity on the classroom observation protocol to reflect the OMB requirements. Therefore, the categories have been revised to:
American Indian or Alaska Native
Asian
Black or African American
Hispanic or Latino
Native Hawaiian or Other Pacific Islander
White
Teacher Survey #4,
The confidentiality pledge needs to be the CIPSEA pledge, if this information is indeed being collected under CIPSEA.
Response: We have changed the assurance to the following:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx (expiration date: xx/xx/xx). Public reporting burden for this collection of information is estimated to average about 20 minutes, including the time to review instructions, gather the data needed, and complete the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please contact: [email protected]. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Department of Education, 50 North Ripley Street, PO Box 302101, Montgomery, AL 36104. If you have comments or concerns regarding the status of your individual submission of this form, e-mail directly to: [email protected].
Personally identifiable information in this questionnaire will not be released to anyone or any organization, except as required by law.
The OMB control number needs to be on the form.
Response: The following statement will be on the form:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx (expiration date: xx/xx/xx). Public reporting burden for this collection of information is estimated to average about 20 minutes, including the time to review instructions, gather the data needed, and complete the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please contact: [email protected]. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Department of Education 50 North Ripley Street PO Box 302101 Montgomery, AL 36104. If you have comments or concerns regarding the status of your individual submission of this form, e-mail directly to: [email protected].
Personally identifiable information in this questionnaire will not be released to anyone or any organization, except as required by law.
Providing a list of teachers, as opposed to requesting name of the teacher, is violating the confidentiality of the teachers participating in the study.
Response: We have designed a method whereby we can code the identity of each survey respondent into their personalized web links that are sent to them in their invitations. So we now are only asking each respondent to type in his/her name, as a check to ensure that the correct person is taking the survey. There will not be a list of respondent names on the surveys.
Why is question 6, which appears to be a screener, not asked before the self-rating?
Response: We will change the survey to place the self-rating at the end of the survey.
If this is an online web-based questionnaire, is it possible to build skip patterns into the instrument so that the participants would only need to answer questions that are relevant to them? (For example, if you answered “I don’t teach math” in 6a, you would skip question 7a.
Response: Yes, we have now built in these skip patterns so that, for example, teachers who only teach math do not receive the science questions.
How is the information on length of time to complete the survey used?
Empirical Education conducts checks on its estimations to ensure that we continue to accurately estimate survey time.
Teacher Survey #2,
The confidentiality pledge needs to be the CIPSEA pledge, if this information is indeed being collected under CIPSEA.
Response: We have changed the assurance to the following:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx (expiration date: xx/xx/xx). Public reporting burden for this collection of information is estimated to average about 20 minutes, including the time to review instructions, gather the data needed, and complete the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please contact: [email protected]. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Department of Education, 50 North Ripley Street, PO Box 302101, Montgomery, AL 36104. If you have comments or concerns regarding the status of your individual submission of this form, e-mail directly to: [email protected].
Personally identifiable information in this questionnaire will not be released to anyone or any organization, except as required by law.
The OMB control number needs to be on the form.
Response: The following statement will be on the form:
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 1850-xxxx (expiration date: xx/xx/xx). Public reporting burden for this collection of information is estimated to average about 20 minutes, including the time to review instructions, gather the data needed, and complete the information collection. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this form, please contact: [email protected]. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to the Department of Education, 50 North Ripley Street, PO Box 302101, Montgomery, AL 36104. If you have comments or concerns regarding the status of your individual submission of this form, e-mail directly to: [email protected].
Personally identifiable information in this questionnaire will not be released to anyone or any organization, except as required by law.
Providing a list of teachers, as opposed to requesting name of the teacher, is violating the confidentiality of the teachers participating in the study.
Response: We have designed a method whereby we can code the identity of each survey respondent into his/her personalized web link that is sent in the invitation. So we now are only asking the respondent to type in his/her name, as a check to ensure that the correct person is taking the survey. There will not be a list of respondent names on the surveys.
If this is an online web-based questionnaire, is it possible to build skip patterns into the instrument so that the participants would only need to answer questions that are relevant to them?
Response: Yes, we have now built-in these skip patterns so that, for example, teachers who only teach math do not receive the science questions.
How is the information on length of time to complete the survey used?
Response: Empirical Education conducts checks on its estimations to ensure that we continue to accurately estimate survey time.
Page
| File Type | application/msword |
| File Title | Preliminary Comments on |
| Author | Lisa Blumerman |
| Last Modified By | Rachel Potter |
| File Modified | 2007-04-12 |
| File Created | 2007-04-12 |