National Evaluation of the Comprehensive Technical Assistance Centers
Statement for Paperwork Reduction Act Submission
Part B: Collection of Information Employing Statistical Methods
Contract ED-04-CO-0028
January 22, 2007
Prepared for
Institute of Education Sciences
U.S. Department of Education
Prepared by
Branch Associates, Inc.
Policy Studies Associates, Inc.
Decision Information Resources, Inc.
Contents
Part B: Collection of Information Employing Statistical Methods 2
B.1 Respondent Universe and Sampling Methods 2
B.2 Information Collection Procedures 6
B.3 Methods to Maximize Response Rates 6
B.5 Individuals Consulted on Statistical Aspects of Design 7
Part B: Collection of Information Employing Statistical Methods
B.1 Respondent Universe and Sampling Methods
This evaluation will collect data from each of 21Comprehensive Technical Assistance Centers (CTACs).
To gain an understanding of each Center's activity, on-site visits will take place by senior evaluation staff at each Center in Spring/Summer 2007. During the site visits, we will conduct interviews using the Regional Center Site Visit Interview Protocol or the Content Center Site Visit Interview Protocol. The primary respondent for these interviews will be CTAC directors, but may also include other key CTAC staff who play a prominent role in the delivery of technical assistance.
Given limited resources, it is not possible for the evaluation team to submit each Center project to an independent review panel to rate quality and relevance; therefore, we propose here a sampling strategy that will provide a broadly representative set of projects for expert panel review from each Center.
To gain an understanding of the range of products and services undertaken each year by each CTAC, we will request each Center complete the Project Inventory Form. The Project Inventory Form will provide a complete list of Center projects1 carried out in each program year (July 2006-June 2007, July 2007-June 2008, July 2008-June 2009). Quality, relevance and usefulness of CTAC products and services represent the key outcomes for the evaluation.2 Expert review panels created by the evaluation team will each be asked to rate each of the projects sampled from the Project Inventory Form.
The Project Inventory Form is central to the evaluation team's effort to sample a representative group of projects for each Center. To select projects, we will employ the following sampling plan:
One-third of the projects for each Center (2-3 projects) will be self-nominated by Center staff as best representing the work they have undertaken each year. Center staff will indicate which projects they are nominating for review on the Project Inventory Form.
Two-thirds of the projects (4-7) for each Center will be selected using stratified random sampling methods described below.
Strata that will be used for random sampling will be developed in two stages. First, using the Project Inventory Form included in this submission, Centers will list all of their projects, within predetermined topic areas (defined in Exhibit 1)3. Second, evaluation staff will estimate relative “level of effort” for each topic area in which a Center is conducting work as part of the Site Visit Protocol during the first year of the evaluation. After removing the self-nominated projects from the Project Inventory Form, topics from each Center will be sampled proportionate to the level of effort for each Center for each topic area4.
For example, suppose a Center’s work falls into five topic areas. The Center self-nominates two projects that fall into two of those areas and together those projects constitute about 30 percent of their total level of effort. The remaining projects comprise 70 percent of their level of effort. We will compute the distribution of the Center’s level of effort across these remaining projects for sampling purposes. In this example, if the Center’s work is evenly distributed across three remaining topic areas, we will select an equal number of projects from each. However, if the Center spends a disproportionate amount of its effort in one or two of these topic areas, we will over-sample projects from those areas. The final sample for each Center will reflect the breadth of its products and services for each year. However, our reliance on a combination of stratified random selection and Center nominations is not designed to result in a statistically representative sample of projects by Comprehensive Center.
Exhibit 1. Topic Areas and Definitions
Topic Area |
Definition |
|
Work that supports an SEA and/or its service delivery system carry out state-level responsibilities related to supporting district and schools identified for improvement. Service may target School Support Teams (SSTs), Distinguished Educators (DEs), or regional education service centers (RSCs, ESCs). |
|
Includes any work designed to build district- and school-level capacity to carry out school improvement and to make AYP, as required under NCLB, including remedies taken for schools or districts in “corrective action” or “restructuring.” |
|
Design and implementation of local assessment systems, such as benchmark assessments aligned to state assessments; classroom assessments designed to support instructional change; or support for analysis and use of data to drive instructional change. Includes consultation on the design and implementation of state assessment systems. Does NOT include alternate assessments for SPED or ELL. |
|
Includes all work at all levels that addresses ELA, reading, and literacy. Includes projects on adolescent literacy. May include work related to curriculum alignment, research-based models or programs, standards and policy reviews, and teacher professional development. |
|
Includes all work at all levels that addresses mathematics and numeracy. May include work related to curriculum alignment, research-based models or programs, standards and policy reviews, and teacher professional development. |
|
Includes all work at all levels that addresses specific content areas other than language arts or math, such as science and social studies. May include work related to curriculum alignment, research-based models or programs, standards and policy reviews, and teacher professional development and any work on curriculum alignment and policies related to academic standards that are not content area-specific. |
|
Includes work related to smaller learning communities, 9th grade or high school transition, dropout prevention, high school policies.
|
|
Includes all work related to special education, such as alternate assessments for the “1 percent and 2 percent students” or instructional/assessment strategies that target needs of students with IEPs. |
|
Includes all work related to ELL (i.e. LEP) students, such as work on Annual Measurable Achievement Objectives (AMAOs) for ELL students and instruction/assessment of ELL students. |
|
All work addressing the NCLB requirement that states have 100 percent of teachers licensed to teach in their subject area and 100% of paraprofessionals with 2-years of college. |
|
All work with parents and community members EXCEPT outreach regarding Supplemental Educational Services or opportunities in association with schools in corrective action or restructuring |
|
Includes work related to helping SEAs evaluate and monitor Supplemental Education Services, as well as setting up systems for notification and outreach to districts, schools, and parents. |
|
Completing miscellaneous federal applications or reports, convening SEA staff to address miscellaneous NCLB topics; ongoing consultation on NCLB topics. |
|
Includes projects that do not fit under any other topic area, including websites, membership on SEA committees, and ongoing phone support. Does not include internal or network meetings—these types of activities should not be included in the project inventory. |
Exhibit 2: Topics Areas by Center
|
ACC |
ARCC |
CACC |
FLICC |
GLECC |
GLWCC |
MACC |
MCCC |
NECC |
NYCC |
NCCC |
NWRCC |
PCC |
SECC |
TXCC |
SWCC |
AACC |
NHSC |
CII |
COI |
NCCTQ |
1. State Systems of Support for Schools and Districts Identified for Improvement |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. Building District/Local Capacity to Support School Improvement, or Address Corrective Action and Restructuring |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3. Assessment |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
4. Language Arts Curriculum, Instruction, and Professional Development |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
5. Mathematics Curriculum, Instruction, and Professional Development |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
6. Other Curriculum, Instruction, and Professional Development including Standards-Based and Research-Based Instructional Frameworks |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
7. High School Reform |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
8. Special Education |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9. English Language Learners |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
10. Highly Qualified Teachers |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
11. Parent and Community Involvement |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
12. Supplemental Educational Services |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
13. Other NCLB-related TA |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
14. Other |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
TOTAL # TOPIC AREAS BY CENTER |
3 |
11 |
5 |
9 |
7 |
9 |
8 |
11 |
11 |
7 |
8 |
8 |
6 |
8 |
10 |
10 |
4 |
5 |
3 |
4 |
3 |
Exhibit 3. List of Comprehensive Centers and Estimated FY 2006 Funding
Regional Centers |
Abbreviation |
Estimated FY 2006 Funding* |
|
Alaska Comprehensive Center |
ACC |
$850,000 |
|
Appalachia Region Comprehensive Center |
ARCC |
$3,426,413 |
|
California Comprehensive Center |
CACC |
$5,979,778 |
|
Florida and Islands Region Comprehensive Center |
FLICC |
$3,831,075 |
|
Great Lakes East Region Comprehensive Center |
GLECC |
$3,633,348 |
|
Great Lakes West Region Comprehensive Center |
GLWCC |
$3,729,967 |
|
Mid-Atlantic Region Comprehensive Center |
MACC |
$3,426,413 |
|
Mid-Continent Region Comprehensive Center |
MCCC |
$2,135,071 |
|
New England Region Comprehensive Center |
NECC |
$1,663,372 |
|
New York Comprehensive Center |
NYCC |
$2,919,576 |
|
North Central Region Comprehensive Center |
NCCC |
$1,301,025 |
|
Northwest Region Comprehensive Center |
NWCC |
$1,649,237 |
|
Pacific Region Comprehensive Center |
PCC |
$850,000 |
|
Southeast Region Comprehensive Center |
SECC |
$4,167,531 |
|
Texas Comprehensive Center Region Comprehensive Center |
TXCC |
$3,983,816 |
|
West/Southwest Region Comprehensive Center |
SWCC |
$2,519,465 |
|
Content Centers |
|||
Assessment and Accountability Content Center |
AACC |
$1,466,096 |
|
National High School Center |
NHSCC |
$2,446,096 |
|
Innovation and Improvement Center |
CII |
$1,466,096 |
|
Instruction Content Center |
COI |
$2,446,096 |
|
Teacher Quality Content Center |
NCCTQ |
$2,446,096 |
|
*Source: http://www.ed.gov/programs/newccp/index.html
B.2 Information Collection Procedures
Prior to conducting the site visits with Center staff, respondents will be notified about the study via an introductory letter, which will include the purpose of the study. Evaluation team members will meet in person with each Center Director and other relevant staff. During the meetings, evaluation staff will conduct interviews with the Center Director using the Site Visit Interview Protocols included in this submission. Also during these meetings, evaluation staff will respond to questions raised by Center Directors and relevant staff about completing the Project Inventory Form.
B.3 Methods to Maximize Response Rates
To ensure 100% response rate from the Centers, the Department of Education has stressed to the Centers the importance of the evaluation. The Department of Education has explained the purpose of the evaluation at several meetings of Center Directors (e.g., September 11-12, 2006 and December 5-6, 2006). In addition, the evaluation team is working in conjunction with the Centers to design the data collection forms in ways that are clear, logical and that make use of available Center documents. The results of the data collection will be analyzed and written in reports that will be useful to the Centers as they seek to improve their own operations and effectiveness. Site liaisons will be available to each Center to provide technical assistance in filling out the data collection forms. Site visits will take place with each Center.
We have piloted the data collection forms with staff from six current Regional and Content Comprehensive Centers and their feedback was taken into consideration in refining the data collection forms. The piloting of the data collection forms and worksheets did not involve more than 9 individuals for each document being developed.
B.5 Individuals Consulted on Statistical Aspects of Design
These data collection plans were developed by Branch Associates, Inc., Decision Information Resources, Inc. and Policy Studies Associates. The research team is led by Alvia Branch, Project Director. Other members of the evaluation team who worked on the design include: Brenda Turnbull (PSA), Kate Laguarda (PSA), Russell Jackson (DIR), Carol Pistorino (DIR), Cynthia Sipe (Branch Associates), and Barbara Fink (Branch Associates). Contact information for these individuals is provided below.
Alvia Branch
Branch Associates, Inc.
215-731-9980
Cynthia Sipe
Branch Associates, Inc.
215-731-9980
Barbara Fink
Branch Associates, Inc.
215-731-9980
Brenda Turnbull
Policy Studies Associates, Inc.
(202) 939-5324
Kate Laguarda
Policy Studies Associates, Inc.
(202) 939-5321
Russell Jackson
Decision Information Resources, Inc.
(713) 650-1425
Carol Pistorino
Decision Information Resources, Inc.
(650) 473-1564
Paul J. Strasberg
Department of Education
(202) 219-3400
1 For the purposes of the evaluation, projects are defined as a group of closely related activities and/or deliverables designed to achieve a specific outcome for a specific audience.
2 The usefulness of Comprehensive Center products and services will be assessed through an annual survey of clients using a Client Survey Form. Request for approval of the Client Survey Form will be included in a second OMB submission (anticipated July 2007).
3 Estimates of the universe of projects and the topic areas they cover were developed through an examination of each Center’s management plan and a pilot test of the Site Visit Protocol and Project Inventory Form with six Centers in fall 2006. Across all Centers, projects fit into 14 principal topic areas (see Exhibit 1 for the list of topic areas and their definitions). However, there is considerable variation across Centers in terms of the number of projects overall and the topic areas covered. Some Centers are engaged in many projects spread across 10 or 11 topic areas whereas other Centers focus their work in fewer projects across 5 or fewer topic areas (see Exhibit 2). This variation reflects, to some extent, the varying levels at which each of the Centers is funded (see Exhibit 3 for a list of Centers and their FY2006 estimated funding).
4 In subsequent years, the evaluation team will review Center management plans it will obtain from ED to determine level of effort by topic area.
| File Type | application/msword |
| File Title | Abt Double-Sided Body Template |
| Author | Abt Associates Inc |
| Last Modified By | DoED |
| File Modified | 2007-01-23 |
| File Created | 2007-01-23 |