Supporting Statement Part B

Supporting Statement for Paperwork Reduction Act Submission

Part B: Collections of Information Employing Statistical Methods

1. Respondent Universe for the State Survey

Exhibit 7 shows the states that meet the maturity and Peer Review approval criteria. To be considered for selection, states had to have an AA-AAS in place since the 2005-06 administration of the assessment and have received one of three levels of approval from Peer Review as of August 2007.

Exhibit 7
States Meeting Initial Selection Criteria for State Survey

State	Date of First Administration	Peer Review Status as of August 2007
Arkansas	2004–05	Full Approval with Recommendations
Colorado	2002–03	Full Approval with Recommendations
Connecticut	2005–06	Full Approval with Recommendations
Idaho	2003–04	Full Approval with Recommendations
Kansas	2005–06	Full Approval with Recommendations
Maine	2001–02	Approval Expected
Maryland	2003–04	Full Approval
Massachusetts	Prior to 2005–06	Full Approval
Montana	Spring 2004	Approval expected
Ohio	2003–04	Full Approval
Virginia	2005–06	Approval expected
West Virginia	May-06	Full Approval

Three states will be selected for the NSAA state survey. In addition to the two criteria described above, states having access to teacher rosters at the state level will be considered first. State-level availability of rosters of teachers who work with students with significant cognitive disabilities will allow for a consistent approach in the selection of the samples of teachers in each state.

We currently plan to select Kansas, Massachusetts, and West Virginia, each of which meets the three selection criteria. Each of these states also has been cooperative in responding to inquiries concerning the number of teachers who work with students with significant cognitive disabilities and providing information about the process to follow in requesting state approval to conduct the survey.

A representative sample of respondents from each state will be drawn from the state rosters of teachers who have worked with students with significant cognitive disabilities. The three states estimate the following numbers of teachers in the state who work with students with significant cognitive disabilities: Kansas, 700; Massachusetts, 2,500; West Virginia, 900.

In each state, a random sample of 270 teachers will be selected from the state roster of teachers of students with significant cognitive disabilities. It is estimated that of the 270 teachers, 200 will meet the screening criteria for that school year and therefore be eligible to complete the full survey.

To be eligible to complete the survey a teacher must respond “yes” to each of the following three questions:

1. Do you currently (2008-09 school year) teach students with significant cognitive disabilities?

 Yes

 No

2. Will any of your students with significant cognitive disabilities take your state’s alternate assessment this school year (2008-09)?

 Yes

 No

3. Did you administer the alternate assessment for students with significant cognitive disabilities in any of the past three school years?

 Yes, I administered the alternate assessment in 2005-06.

 Yes, I administered the alternate assessment in 2006-07.

 Yes, I administered the alternate assessment in 2007-08.

 No, I did not administer the alternate assessment in the past three years.

Anticipating an 80 percent response rate, we expect 160 teachers to complete the survey in each state. This will allow for a standard error for a dichotomous variable (associated with the teacher, such as teacher background) of not more than 0.04.

In the event that Kansas, Massachusetts, or West Virginia is unable to participate, we will recruit one or more replacement states from the list that have state-level databases. If we are unable to recruit a replacement state with a statewide list of teachers, we will recruit states that have teacher rosters at the district level. Sampling procedures then will be modified as follows: sample districts within the state (with probability proportional to size), solicit lists from those districts, and then randomly select teachers from those lists (so that each teacher in the state has an equal probability of selection).

2. Data Collection Procedures

The overall approach to the data collection process is designed to maximize the response rate by starting with state support of the study expressed in the materials sent to the selected teachers and by ensuring a comprehensive and persistent follow-up and nonresponse conversion effort. The OMB requirements for this project stipulate a response rate of 80 percent. The study team will use the data collection procedures described below to obtain a response rate that is as high as possible. A key component will be the careful monitoring of response rates and use of these data to guide timely actions. The NSAA study team will monitor response rates weekly to track progress toward the desired response rate. The NSAA will develop detailed state survey procedures and document them in a procedural manual, train state survey staff, and monitor conduct of data collection.

Recruiting state education agencies and gaining state cooperation

Although initial contacts with Kansas, Massachusetts, and West Virginia have been encouraging, it will be necessary to gain their formal approval. These efforts will be guided by three key strategies to achieve agreement from the three selected states to participate in the state survey activity:

• endorsement of the state survey activity by the U.S. Department of Education in an introductory letter on U.S. Department of Education letterhead and signed by personnel from the Institute of Education Sciences (see ED letter in appendix F);

• provision of high-quality informational materials to the states and responsive research staff to address each state’s questions about the study; and

• development of the personal contacts with state staff that were forged during the State Data Summary/Interview activities conducted in 2006 and 2007.

The state’s superintendent of education, director of assessment and accountability, director of special education, and NSAA state contact (with whom relationships were established during the State Data Summary/Interview phase of the study) will receive copies of the materials describing the study and requesting the state’s agreement to participate in the study. The study data collection procedures will be described, including a timeline for the state survey activities (see notification letter from the NSAA included in this submission)

A senior NSAA staff member will contact the appropriate offices in the targeted states and work with them to obtain their agreement to participate in the study. This effort will build on the rapport built over the past 2 years during the State Data Summary/Interview phase of the study. The NSAA staff member who makes the initial contact will continue to work with the state in later phases of the teacher survey activity to maintain established relationships with state personnel. The NSAA study team will follow all state-required procedures to obtain approval to conduct the state survey activities.

When each state has agreed to participate in the state survey activities, the NSAA study team will work with the state to provide the study with a data file of teachers who work with students with significant cognitive disabilities. Personally identifiable information for each teacher is not required at this point, and a unique identifier may be used for random selection if the state so desires. Contact information for selected teachers will be needed and must be accessible through the use of the unique identifier.

State support of the survey of teachers

The NSAA study team will obtain a letter of support from the state office of education to include in the informational materials sent to selected teachers in the survey packet. Although the study team will provide each state with possible text to use in this letter (see notification letter from states included in this submission), each state will be free to compose its own letter. The NSAA will work with the state to determine the most effective language for the letter. This letter of endorsement is critical for reassuring teachers that their selection was authorized by the state.

Contact information for selected teachers

One of the state selection criteria is the availability of the necessary teacher data file. The NSAA will obtain the data file from each state and verify that it includes all the required information needed for sampling. It is critical that the state provide the following information for each teacher selected:

• First name

• Last name

• Mailing address

• E-mail address when available

• Phone number

• District affiliation

• School affiliation

• Teaching assignment

Initial mailing

The NSAA study team will initiate data collection with a mailing of the state survey packet to all sampled teachers in each state. The teacher survey packet will contain information about the study and letters of support from the state and from the U.S. Department of Education encouraging teachers to participate in the study. The packet will include a hard copy of the survey with a postage-paid return envelope, as well as instructions for completing the survey either in hard copy or online. Login information will be provided to allow teachers who wish to complete the survey online to do so. A toll-free telephone number and an e-mail address, along with the name of the NSAA contact person assigned to the teacher/respondent, will be included in all correspondence for use by a teacher if he or she has any questions or requires assistance (For letters, see appendix F).

Finally, a $5 bill will be attached to the hard-copy survey as an incentive for the teacher to complete and return the screening portion of the survey either in hard copy or online and to proceed to the full survey if the teacher meets the screening criteria. Teachers will be informed that they will receive a check for $35 for completing and submitting the full survey either in hard copy or online.

For the anticipated 5 percent of the surveys that will be returned as undeliverable, the NSAA will obtain a revised address for the respondent and mail the survey again. In some cases, it will be determined that the respondent should be removed from the study (e.g., he or she no longer works at the school).

Data tracking

Given that the study team conducting the state survey task is divided between the east and west coasts, it is important that the study team members on both coasts have the same information available. The data tracking system will be a real-time web-based tool developed by NSAA and hosted by IES to ensure data security. The system will serve as the primary communication tool between NSAA study staff and as the host for the online version of the survey. The data tracking system will be able to generate response rate reports.

Once the sample is drawn for a specific state, the data tracking system for that state will be populated with the appropriate respondent information fields:

• First name

• Last name

• Mailing address

• E-mail address if available

• Phone number

• District affiliation

• School affiliation

• Teaching assignment

• Initial mailing date

Additional fields that will be available for tracking purposes are as follows:

• Each of the proposed reminder and follow-up dates

• Replacement survey requests

• A notes field

• Fields for revised telephone number and mail and e-mail addresses

The NSAA study team will participate in a half-day video-based training delivered at the SRI offices in Menlo Park, California, and Rosslyn, Virginia, so that all researchers on the team become proficient in the use of the data tracking system. The training will provide hard-copy reference and instructional materials.

3. Methods to Maximize Response Rates

Multiple steps will be taken to maximize response rates for the state survey. NSAA will work closely with states to engage them in encouraging participation of teachers as needed.

NSAA will follow a set of procedures suggested in the literature on improving response rates to surveys (Dillman, 2000; Dillman, Christenson, Carpenter, & Brooks, 1974; Heberlein & Baumgartner, 1978) which include the following elements:

• A respondent-friendly survey. Surveys that are easy to respond to and that address topics of interest to the respondent are more likely to achieve high response rates.

The LCI and CSI have been validated by the researchers from NAAC who developed them, and the overall NSAA survey was piloted and revised for maximum clarity and ease of response.

• Multiple contacts by first-class mail. The use of first-class mail represents a gesture of respect that supports the development of a relationship between the sender and the recipient and encourages reciprocity.
  
  NSAA will make three contacts by first-class mail: (1) The survey will be sent with a detailed cover letter and informational materials. (2) A reminder postcard will be sent 1 week later to express appreciation for responding and request that they complete the screening questionnaire and survey if they have not done so. (3) A replacement survey will be sent to nonrespondents 3 weeks after the first survey mailing.

• Return envelopes with real first class postage. Return envelopes for surveys will be provided with real first-class postage. Business reply envelopes will not be used.
  
  

• Personalization of correspondence. Correspondence can be personalized in several ways, such as the use of high-quality paper, real names instead of a generic “Dear Teacher,” and a high-quality signature from the study leaders to create a sense of reciprocity with the study and to improve response rates.
  
  Each of these approaches will be used on all correspondence with teachers. All correspondence will include instructions for responding in hard-copy or online format, as well as directions for requesting a replacement hard-copy survey if needed. A toll-free telephone number and an e-mail address, along with the name of an individual contact person, will be included in all correspondence for a teacher to use if he or she has any questions or requires assistance.

• Token prepaid financial incentives. Inclusion of a small financial incentive of $1 to $10 with a request to respond to a mail survey increases response rates significantly; it evokes a sense of obligation that can be easily discharged by completing the survey.
  
  A $5 bill will be attached to the survey as an incentive for the teacher to complete and return the screening portion of the survey either in hard copy or online and to proceed to the full survey if the teacher meets the screening criteria. Teachers will be informed that they will receive a check for $35 for completing and submitting the full survey either in hard copy or online.

• Compensation for effort. Payment to teachers at a level commensurate with their work in completing the 1-hour survey reflects genuine respect for the teachers’ valuable time and investment in a survey and increases willingness to respond.
  
  A check for $35 will be promptly sent to each teacher who completes a survey online or returns a completed survey by mail.

Additionally, the NSAA will carefully monitor response rates weekly to guide timely actions and follow up with nonrespondents promptly and often during the period of data collection. The study team will employ the following procedures during multiple phases of follow-up to maximize the response rate as follows:

1. Thank you/reminder postcard. The NSAA will send a reminder postcard to all teachers 1 week after the initial survey packet mailing to express appreciation in advance for their prompt response. NSAA contact information, including the toll-free number and name of an NSAA contact person, will be provided so that if teachers have questions or need a replacement survey, they can easily request it. (See reminder postcard in appendix F)

2. Replacement survey. Three weeks after the initial survey mailing, a replacement survey packet will be mailed to all respondents who have not yet completed the survey (either by mailing in their hard-copy survey or by completing the survey online). The replacement survey packet and any other requested remails will not include the $5 incentive.

3. E-mail reminders. Each of these reminders will include a “thank you” if the teacher already mailed a completed survey and a reminder to complete either the hard-copy or online version if he/she has not yet done so. The link to the online version of the survey will be provided. The e-mail will also remind the teacher about the $35 compensation he/she will receive upon completion of the survey if the teacher meets the screening criteria. The first reminder will go to nonresponders 2 weeks after the initial mailing and the second reminder will go to nonresponders 4 weeks after the initial mailing.

4. Telephone follow-up. Five weeks after the initial mailing, NSAA staff members will attempt to contact all nonrespondents by telephone to remind them to complete the screening questionnaire and survey, if appropriate; answer any questions they may have; and send a replacement survey packet when the teacher requests one. Each nonrespondent will receive up to three telephone calls. The team members will leave messages if the respondent is not available.

4. Pilot Testing

To improve the quality of data collection and control the burden on respondents, the draft state survey was pilot-tested.

Pilot testing was conducted with six teachers in six states during the first two weeks of July 2008. Pilot test interviews focused on clarity of instructions, completion time of the different sections of the survey, clarity and contextual relevance of individual items, and potential bias of items. The pilot test was conducted by one of two individuals trained to follow a prescribed set of procedures and questions in contacting, arranging for piloting, and interviewing the pilot testers during a debriefing session.

• Pilot testers were teachers of students with significant cognitive disabilities who had multiple years of experience administering the AA-AAS in their state and were identified by NSAA state contacts in their respective states (exhibit 8).

• Each pilot test participant was contacted by e-mail. The e-mail included a brief description of the study, a copy of the draft survey, and a pilot test participant agreement.

• Each pilot test participant was asked to take the survey as if she were a respondent. Additionally, participants were asked to take marginal notes about questions that lacked clarity and to keep a log of the amount of time it took them to complete each section.

• Each pilot test participant was interviewed to get her overall impressions of the survey, the amount of time it took to complete each section, and any challenges/questions regarding specific questions. The interviews took approximately 45 minutes.

• All pilot test participants sent copies of their completed surveys to the NSAA following the interview.

• Following the discussions with the pilot test participants, the NSAA discussed possible changes to the survey that the study team discussed on July 29, 2008, and decisions were made at that time as to which revisions would be incorporated into the state survey.

Exhibit 8
Pilot Test Participation Information

Pilot Test Participant #	Years of Experience	AA-AAS Administration and Teaching Background
1	6.5	Has been administering the alternate assessment since he/she started teaching.
2	8.0	Has been administering the alternate assessment since he/she started teaching and is currently a team leader for alternate assessment scoring in Oklahoma.
3	20+	Has been teaching students with significant cognitive disabilities for the past 20 years. In addition to serving as a teacher, also is a supervisor of other teachers of students with significant cognitive disabilities and is a scorer for the alternate assessment in state.
4	20+	Has been teaching students with significant cognitive disabilities since 1985 and has administered the alternate assessment in Kansas since the instrument was put in place.
5	14	Is currently teaching students assigned to the middle school grades. Prior to that taught a self-contained classroom in a high school.
6	14	Has taught students with significant cognitive disabilities for most of teaching career. Was involved in development of the original alternate assessment in state, as well as the version that is currently in place.

The list of survey revisions shown in this document is a result of this discussion. No feedback on the draft survey or design was received from the public.

Overall, the pilot testers felt that the instrument was well organized and that it flowed well from one question to the next. Several respondents remarked that it “made them think.” All six respondents seem to have taken their role of pilot tester very seriously. They not only provided insightful comments during the interview, but the paper surveys later received in the mail made it clear that they had all followed the instructions to the letter (by making marginal notes and keeping track of the time to complete each section).

The remainder of this document describes the results of the pilot test in two areas: burden and survey revisions.

Burden

Exhibit 9 shows the burden (in minutes) reported by each pilot tester for each section of the survey. The average for each section is shown in the first column of the table. Based on these results, the appropriate burden estimate was 120 minutes, or 2 hours.

Exhibit 9
Pilot Test Participation – Burden in Minutes

		Pilot Tester
Section	Average	1	2	3	4	5	6
1. Background	34	30	35	30	45	36	30
2. Target student	20	25	20	13	15	29	20
3. Reading/ELA	38	30	15	22	10	97	50
4. Math	26	30	15	12	10	64	25
Total	118	115	85	77	80	226	125

The burdens across all six pilot testers were fairly consistent for sections 1 and 2 of the survey; the burdens reported by the pilot testers in sections 3 and 4 showed greater variability. As the pilot test debriefing interviews progressed, interviewers asked additional questions of the pilot testers to determine whether there were reportable reasons for these differences in burden but were unable to identify any.

Survey revisions

After discussions with OMB, NSAA decided to reduce the burden by removing parts 3 and 4, and to add one new item to capture some of the information that was lost. This additional item is in a similar format to other items on the survey and will be adapted to display the content standards of each participating state. The additional item adds an estimated 5 minutes of burden. Exhibit 9a displays the new burden estimate totaling to 59 minutes or 1 hour.

Exhibit 9a

Revised Burden with Removal of Parts 3 &4 and Additional Item

		Pilot Tester
Section	Average	1	2	3	4	5	6
1. Background	34	30	35	30	45	36	30
2. Target student	20	25	20	13	15	29	20
3. Additional Item	5	-	-	-	-	-	-
Total	59

None of the original pilot testers reported that they were exceptionally challenged by the questions, other than the significant amount of time it took to complete the survey. All of the pilot testers reported that they were initially challenged by the instructions associated with identifying a target student but felt that they were able to follow the instructions correctly. There were several instances where many of the pilot testers had questions or comments about the same question or item. This level of consensus resulted in a recommended change to the survey. There also were a small number of instances where a single pilot tester provided input that resulted in a revision.

Exhibit 10 summarizes the changes that have been made to the survey as a result of the pilot test.

Exhibit 10
Survey Revisions Based on Pilot Test

Question	Existing Text	Revision
1.A.6 Certifications	Special Education Elementary Education Middle Secondary National Board Other (Specify ________________) None of the above	Remove “None of the above”
1.A 7 Teaching license	Reading/English language arts Math Science Other (Specify _______________)	Add “Special education”
1.C.1 Conflicts	Time to teach versus time to test Teaching academic standards versus students’ other skill areas Student individual needs versus state expectations for academic achievement Parental preferences versus requirements of AA-AAS Routine duties and paperwork versus time with students	Revise: Time to teach vs. time to conduct the alternate assessment Parental preferences vs. requirements of the alternate assessment
1.D.2 Support	Which of the following kinds of support have you received to help with alternate assessment administration and assembly? Reduced or flexible teaching schedule Common planning time or collaboration with other teachers administering/assembling the alternate assessment Extra classroom assistance (e.g., teacher aides) Regular supportive communication with your principal, other administrators, or department chair Guidance or assistance from another teacher	Underline the text “alternate assessment administration and assembly” Add “Release time” as an option
1.D.4 AA Results	Regarding results from the alternate assessment, to what extent do you agree with the following statements? … My students are aware of the alternate assessment process. My students understand the meaning of the alternate assessment scores. …	Revise “My students” to “Most of my students” Move the two student-related and two parent-related questions to a separate new question and add a column “Don’t Know” for that question.
2.0 Overall		Clarify in the instructions that teachers should put together a list of students but not submit that information with their survey.
2.4 Grade level		Add: “At what grade level is the target student currently performing? Pre-K, K, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, Ungraded”
2.16 Who teaches the student	Who primarily teaches the identified target student in each of the following subject areas?	Revise the question so that it is in two parts – one part for instructional planning and one for day-to-day instruction. The study team will ask the pilot test participants for their input on this change.
2.17 Instruction	Over the last 30 days, how often did you provide instruction in the following content areas to the identified target student? Please note that a single lesson may address multiple content areas simultaneously.	Revise to: Over the last 30 days, how often did the identified target student receive instruction in the following content areas?
Sections 3 & 4 overall	Sections 3 and 4 consisted of detailed curriculum analysis in reading/language arts and mathematics.	Sections 3 and 4 have been removed to reduce burden. One new item has been added.

5. Contact Information

Dr. Jose Blackorby is the Principal Investigator for the study. His mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Blackorby also can be reached at 650-859-4210 or via e-mail at [email protected].

Dr. Renée Cameto is the Project Director for the study. Her mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Cameto also can be reached at 650-859-6451 or via e-mail at [email protected].

13