3145-0211 Supporting Statement A 4-15 revision

Supporting Statement

Revised Clearance: National Science Foundation, Directorate of Education and Human Resources, Division of Graduate Education

Data Collection for the Evaluation of the Computer & Information Science & Engineering’s Pathways to a Revitalized Undergraduate Computing Education Program (CPATH)

3145-0211

SECTION A

Introduction

This request for Office of Management and Budget (OMB) review asks for a revised clearance of the original 3-year clearance of the evaluation plan evaluation of the Computer & Information Science & Engineering’s Pathways to a Revitalized Undergraduate Computing Education Program (CPATH), which is administered by the National Science Foundation (NSF)’s Directorate for Computer and Information Science and Engineering (CISE). The evaluation is funded through the Directorate for Education and Human Resources (EHR).

NSF funds research and education in science and engineering. It does this through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other research and/or education institutions in all parts of the United States. NSF accounts for about 20 percent of Federal support to academic institutions for basic research. EHR is the directorate within NSF that is responsible for the health and continued vitality of the Nation’s science, technology, engineering, and mathematics (STEM) education and for providing leadership in the effort to improve education in these areas.

After the first round of data collection in the spring of 2010, the existing site visit interview protocols remained unedited but new protocols for the project evaluators and for faculty being interviewed by phone needed to be developed. After the site visits in the Spring of 2010, it was determined that project evaluators play a key role in measuring and reporting on project outcomes and will provide insight not available from the current universe of respondents. The new faculty phone interview protocol parallels the faculty site visit protocol but is intended to garner more in-depth information about a typical professors’ involvement with CPATH. The interview protocols. This package describes the updated plans for the continued evaluation of the CPATH program. The first round of data collection included face to face interviews with six types of respondents. The protocols for these respondents were not revised, but a new protocol for project evaluators has been added., The interview protocols for the faculty phone interviews and on site interviews are contained in Appendix A. These interviews will be conducted in Spring 2011, 2012, and 2013.

Overview of Program: CISE Pathways to a Revitalized Undergraduate Computing Education Program (CPATH)

As part of its mission CISE contributes to the education and training of the next generation of computer scientists and engineers. Through the CPATH program CISE is challenging its community partners – colleges, universities, and other stakeholders committed to advancing the field of computing and its impact – to transform undergraduate computing education on a national scale in order to meet the challenges and opportunities of a world where computing is essential to U.S. leadership and economic competitiveness across all sectors of society.

The use of computers has permeated and in many cases transformed almost all aspects of our everyday lives. As computer use becomes more important in all sectors of society, so does the need for preparation of a globally competitive U.S. workforce with knowledge and understanding of critical computing concepts, methodologies, and techniques. Thus, upgrading undergraduate computing education to keep abreast of the multitudes of rapid changes in computing is paramount for the U.S. economy and competitiveness.

The CPATH vision is of a U.S. workforce with the computing competencies and skills imperative to the Nation’s health, security, and prosperity in the 21^st century. This workforce includes a cadre of computing professionals prepared to contribute to sustained U.S. leadership in computing in a wide range of application domains and career fields, and a broader professional workforce with knowledge and understanding of critical computing concepts, methodologies, and techniques.

To achieve this vision, CISE is calling for colleges and universities to work together and with other stakeholders (industry, professional societies, and other types of organizations) to formulate and implement plans to revitalize undergraduate computing education in the United States. The full engagement of faculty and other individuals in CISE disciplines will be critical to success. Common challenges are fluctuating enrollments in traditional computer science programs, changes and trends in workforce demographics, the imperative to integrate fast-paced computing innovations into the curriculum, and the need to integrate computing concepts and methodologies into the undergraduate curriculum at large. Goals and strategies must be developed to address these and other challenges. Successful CPATH projects will be systemic in nature, address a broad range of issues, and have significant potential to contribute to the transformation and revitalization of undergraduate computing education on a national scale.

CPATH was first announced in FY 2006. From FY 2007 to FY 2009, NSF made awards to 96 institutions involved in 69 CPATH projects. The following categories were used to categorize the awards in FY 2007 and FY 2008 (all four were used in FY 2007; only the first two were used in FY 2008).

• Community Building (CB). CB awards support community-building efforts that bring stakeholders together to discuss the challenges and opportunities inherent in transforming undergraduate computing education, and to identify creative strategies to do so. The types of activities supported by CB grants include, but are not limited to: a) development of forums and opportunities for community stakeholders to come together to explore common interests, share lessons learned, and identify promising practices; b) engagement of stakeholders in undergraduate computing education including administrators and faculty from computer science and other disciplines in which computing is playing an increasingly important role, within one institution or more broadly; and c) efforts focused on developing strong partnerships among academic, industrial, and not-for-profit organizations with a stake in undergraduate computing education. The scope of CB activities is deliberately broad. CISE encourages the community to develop creative strategies likely to effect transformation in undergraduate computing education at the institutional, local, regional, and/or national levels and across all institution types.

• Institutional Transformation/Transformative Implementation (IT). IT grants support the implementation of innovative, integrative models for undergraduate computing education that have potential to serve as national models. IT projects are expected to: a) develop and implement innovative, integrative organizational models for undergraduate computing education at one or more institutions; b) create sustainable changes in culture and practice within the participating organizations; and c) serve as models and resources for the national computing community. Single-institution IT projects must engage multiple academic units or disciplines. IT grants also support the work of multiple institutions committed to the implementation of common or related models of undergraduate computing education.

• Evaluation, Adoption, and Extension (EAE). EAE awards support the ongoing work of institutions that have already discovered and have begun to implement innovative undergraduate computing education models and approaches to realize the CPATH vision, as well as those organizations wishing to emulate and/or evolve the models. Specifically, EAE awards support efforts to: a) evaluate the success and impact of new models currently being implemented; b) engage additional institutions in their implementation; and/or c) to expand the scope of ongoing efforts. EAE grants support either, or both, the originating institutions and the institutions committed to replicating or evolving the promising model. EAE grantees are expected to disseminate lessons learned and promising practices such that other institutions and organizations may benefit from the project outputs.

• Distinguished Education Fellow (CDEF) projects. CDEF grants recognize accomplished, creative, and talented computing professionals who have the potential to serve as national leaders or spokespersons for change in undergraduate computing education. CDEF awards are made to individuals who have achieved distinction in the computing profession, who are committed to transforming undergraduate computing education, and who have innovative ideas on how to do so. CDEF recipients may spend significant time and effort on projects focused on innovative, original, and possibly untested ideas that will benefit undergraduate computing education on a national scale.

The FY 2009 solicitation classified CPATH grants according to budget level, although awarded projects must have met the criteria described above. Class I projects have budgets totaling no more than $300,000 for 1-, 2- or 3-year durations. Class II projects have budgets totaling no more than $800,000 for 2- or 3-year durations.

Overview of Evaluation Plan

NSF has contracted with SRI International to conduct a five-year evaluation of the CPATH program. The goals of this program evaluation are to:

• Document the overall CPATH program delivery through its funding of awards;

• Describe how the program is being implemented through different project strategies; and

• Evaluate the extent to which the program is accumulating evidence supporting the transformation of undergraduate computing education.

A primary focus of this evaluation will be to describe and document the program strategies and models utilized in infusing computational thinking across different contexts and disciplines. CPATH is a new program for NSF and addresses a relatively complex problem for higher education institutions – how to reform undergraduate computer education for a world that has rapidly embraced technology in almost every facet of life and work. Therefore, this evaluation will focus on providing a comprehensive description of all project types, including curricular and pedagogical innovations and promising models of institutional change in higher education institutions. Given the interdisciplinary nature of computing, this evaluation will also examine the development of communities of practitioners and the dissemination of best practices around computational thinking. Additionally, the evaluation will examine partnerships between the different sectors with a stake in computing education.

The four main sources of information for the CPATH evaluation will be:

• Site visits to a sample of awardees (conducted annually for four years)

• Faculty telephone interviews

• Project evaluator interviews

• Project documents (reviewed annually)

• Evaluation reports from a subset of projects that are supported to use quasi-experimental designs (reviewed annually)

In addition, programmatic data collected from project Principal Investigators through the CPATH data monitoring system will provide the numbers and types of people working on CPATH projects as well as other background information required for monitoring NSF programs.

Logic Model. The conceptual logic model that guides the CPATH evaluation is depicted in the figure on the next page. The model presents a common framework for understanding the context of the CPATH program, its specifications, and its goals, strategies, and outputs. The CPATH evaluation will employ mixed-method evaluation strategies including document analyses, site visit interviews (conducted mostly face-to-face but some by telephone), and faculty telephone interviews to assess and measure site-based progress and outputs as they accumulate over the period of the evaluation. The faculty telephone instrument was informed by the first round of site visits and is included in Appendix A.

Research Questions. Five overarching evaluation questions will guide the CPATH evaluation:

1. How is the CPATH program infusing computational thinking into a wide range of disciplines serving undergraduate education?

2. What is the evidence that university and community college departments and faculty are integrating computational thinking into their courses?

3. What is the evidence that the program is supporting the development of promising models of institutional change?

4. What is the evidence that the program is developing communities of practitioners (among the different program stakeholders) that regularly share best practices across communities?

5. How has the CPATH program promoted sustainable multi-sector partnerships that represent a broad range of stakeholders (e.g., industry, higher education, K–12) and contributed to the growth of a workforce that ensures continued U.S. leadership in innovation?

Appendix B presents a crosswalk between these five overarching evaluation questions (and sub questions) and the items in the site visit and telephone interview protocols that will provide data to address the evaluation questions. The crosswalk documents the pathways for gathering data to shed light on the infusion of computational thinking across disciplines, the development of models of institutional change, community building, and the creation of partnerships.

A.1. Circumstances Requiring the Collection of Data

The CPATH program was initiated in FY 2007. The program has not been evaluated previously by any agency or individual. While CPATH projects are required to have a project-level evaluation, no data on the extent to which expected programmatic outputs are being achieved are available.

The evaluation of CPATH at the program level requires a different approach than evaluation of the individual projects it supports. A program evaluation determines the overall value of a collection of projects that address an identified issue. It assumes variation in approaches to addressing the issue but looks for patterns that can help explain both successes and challenges. Some data used to track individual project activities and measure their impacts can be used for program-level evaluation, but program evaluation is more than an aggregation of project-level evaluation data.

An aggregation of project data will not suffice for evaluation in this case is that the CPATH program itself may influence the community, including those who are not directly involved in the program. Although difficult to identify, these “spill-over” effects can be significant. If CPATH were an unchanging program that supported nearly uniform projects, or if this were a summative evaluation, a fixed evaluation plan would be methodologically appropriate. However, by its very nature CPATH requires a flexible and adaptive evaluation plan. This evaluation is therefore designed to have elements that track the same data over time through the use of mixed methods that will capture the changing dynamics within undergraduate education.

The CPATH program falls under the jurisdiction of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Act (HR 2272), a legislative statute that provides increased support for education programs in the United States to ensure that STEM students, teachers, workers, and businesses remain competitive in the global economy and mandates (through the oversight of the Academic Competitiveness Council) that Federal STEM education programs receiving support undergo rigorous evaluation. In the case of CPATH, rigorous evaluation includes general data collection from all the projects as well as implementing quasi-experimental designs (using comparison groups) with a subset of projects to collect more in-depth data and build evidence to make causal claims about the program’s effectiveness. In addition to the proposed original qualitative data collection activities, the program evaluation will draw from reports of these project-level evaluations to the extent practical, using an analytic approach called narrative review to summarize evaluation results.

A.2. Purposes and Uses of the Data

The overall purpose of the data collections is program evaluation. The data obtained from the data collections will be used to document the effectiveness and outputs of the CPATH program and assess achievement of program goals.

The specific purpose of the site visit interviews is to learn more about the CPATH projects, how they are being implemented, the organizations and stakeholders involved, and the effect that the projects are having on various groups and organizations.

The site visit interview protocols, except the evaluator protocol, are the same as the protocols approved in last year’s OMB clearance. The interview protocols for the Principal Investigators (including Co-PIs), administrators, faculty, project staff, project evaluators and external partners seek to obtain background information about the individual and/or organization, strategies being used for the CPATH project, factors related to implementation of the project, perceived outputs, and community building and development of partnerships. Many of the questions are the same, or very similar, on all six protocols. Occasionally the wording of questions has been revised somewhat to make it more relevant to a certain group. Some questions do not appear on a particular protocol because they are not relevant to the group. More detail on the specific content being collected via these six protocols is provided below. Specifics on the student focus group protocol are described after the other protocols.

Background information: roles at the institution (PI protocol only); length of service; other positions held; role/responsibilities related to the CPATH project; prior involvement in undergraduate reform activities (for administrators, past and current reform initiatives on campus); perceptions of the most pressing issues in improving undergraduate education (not on staff protocol); how/why the institution/person became involved with CPATH; principal goals of CPATH project (staff protocol only). The protocol for external partners includes questions on the primary mission and focus of the organization, its size, its history including the year it was established, other major projects they are involved in, and their main sources of support.

Project strategies: nature of the teaching/learning environment for computing that the project is trying to create at the institution/partner sites; core strategies being used and how successful they have been; core computing concepts/competencies that the project focuses on (not on administrator protocol); how the project has integrated these concepts into courses outside of traditional computing disciplines (not on administrator or partner protocols); the primary beneficiaries of the project (not on partner protocol).

Implementation factors: highlights and successes/failures thus far; challenges/barriers; lessons learned; support offered by the institution to help faculty with curriculum development or devising pedagogical strategies for teaching computational thinking (not on staff or partner protocols); factors that have supported project implementation; sustaining the project beyond the end of the funding period (not on faculty protocol).

Outputs: results of the project influencing faculty and students, e.g., enrollment in computing courses (not on partner protocol); changes in faculty culture (not on partner protocol); faculty publications on computational thinking (not on staff or partner protocols); institutional changes that can be attributed to the project, e.g., integration of computational thinking into other disciplines; rewards/incentive structure at the institution (not on staff protocol); whether or not the project has created a model that could be used at other institutions. The administrator protocol includes a question on how institutional changes might be documented.

Community building and partnership development: kinds of stakeholders and how information is shared with them; how inclusive the partnerships are; whether or not there is a shared understanding about computing competencies and/or computational thinking among stakeholders; communication among partners/sharing best practices; other organizations with which the partner has an ongoing relationship because of the project; extent to which NSF funding of the project has created new opportunities for partnerships among multiple sectors; extent to which the partnerships leveraged pre-existing relationships or new opportunities; interdependency of all partners (not on administrator protocol), effectiveness of partnership (not on administrator protocol), ways/barriers to sustaining multi-sector partnerships to replicate computational thinking models over the long term.

The focus group protocol for undergraduate students seeks to obtain information on the background of the students, their participation in the CPATH project, benefits of their participation, and their future plans. Specific items being collected are identified below.

Background information: year in college; major.

Participation in CPATH project: courses taken/currently enrolled in; other non-classroom activities that involve computing.

Outputs: benefits/gains from participating in computing courses/activities (interest in computing, computing ability, impact on future career plans, contacts with faculty/companies/ other organizations, jobs/internships/interviews/mentoring obtained through the CPATH project, whether classmates would say the same); description of a learning experience in the classroom or lab that involved computing (including resources/tools used, role of instructors/TAs, teamwork); how CPATH computing courses/experiences compare to previous ones (including caliber and diversity of students); what “computational thinking” means and how computing courses/ experiences through CPATH have shaped that understanding; perceptions of CPATH’s goals for students at their institution and whether or not the goals are being met.

Future plans: for continued involvement in computing courses/activities; for graduate school vs. job using computing vs. job in a non-computing specialty; how computing courses/ experiences have influenced the students post-graduation plans.

A.3. Use of Information Technology to Reduce Burden

The data collection for this revised clearance will take place in Spring 2011, 2012, and 2013. This collection will involve face-to-face interviews using existing protocols during site visits to institutions with CPATH projects, some telephone interviews with off-site partners, and the faculty telephone interviews. Section A.3 does not apply to this data collection.

A.4. Efforts to Identify Duplication

The evaluation of the CPATH program does not duplicate other NSF efforts. The data being collected for this program evaluation have not been and currently are not being collected by NSF or other institutions. As noted earlier, the program evaluation will draw from rigorous project-level evaluations to the extent practical, using an analysis approach called narrative review to summarize evaluation results.

A.5. Small Business

It is unlikely that this program evaluation will have an impact on small business. Site visits will include speaking with CPATH grant partners, who may represent large companies, K-12 school districts, higher education institutions, government offices, non-profits, and professional membership organizations. Partners will be asked questions about their CPATH project, how it is being implemented, and the extent to which various organizations and stakeholders have been involved and been affected by the project. If the program ultimately succeeds in reforming undergraduate computer education, many small businesses may benefit in the longer term.

A.6. Consequences of Not Collecting the Information

If the information is not collected, NSF will be unable to document the effectiveness and outputs of the CPATH program. Moreover, it will not be able to meet its accountability requirements because it will be unable to assess the degree to which the program is meeting its goals. This lack of information may hamper program management. In addition, NSF will be unable to comply fully with the Congressional mandate that NSF evaluate its science, technology, engineering, and mathematics (STEM) education programs.

A.7. Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6

The data collection will comply with 5 CFR 1320.6.

A.8. Consultation Outside the Agency

Two notices have been published to solicit comments from the public. The first notice was published in the Federal Register on October 25, 2010 (75 FR 65527). This package follows the second notice. A copy of the text of both notices is included in Appendix C. No substantive public comments were received in response to the first notice. No comments have been received to date in response to the second notice.

The evaluation design was developed in consultation with NSF staff in the Directorate for Education and Human Resources (EHR), through which the evaluation of the CPATH program is funded, and with staff in the Directorate for Computer and Information Science and Engineering (CISE), which funds and administers the CPATH program.

A.9. Payments or Gifts to Respondents

No payment or gifts will be provided to participants in any data collection activities.

A.10. Assurance of Confidentiality

Interviewees will be advised that any information on specific individuals will be maintained in accordance with the Privacy Act of 1974. The data that are collected will be available to only NSF officials and staff and to the evaluation contractor. The data will be processed according to Federal and State privacy statutes. Detailed procedures for making information available to various categories of users are specified in the Education and Training System of Records (63 Fed. Reg. 264, 272 January 5, 1998). That system limits access to personally identifiable information to authorized users. The data will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c. The information requested may be disclosed to qualified researchers and contractors in order to coordinate programs and to a Federal agency, court or party in a court, or Federal administrative proceeding, if the government is a party.

Participants in the site visit interviews will be assured that the information they provide will not be released in any form that identifies them as individuals. Evaluation findings about the CPATH projects will be reported in aggregate form in all reports. The contractor, SRI International, has extensive experience in collecting information and maintaining the confidentiality, security, and integrity of data.

The following standards and procedures will safeguard the privacy of interviewees and the security of the data that are collected, processed, stored, and reported.

• Project team members will be educated about the Privacy Act of 1974, the need to ensure study participants about confidentiality of their responses, and ways data and other sensitive materials are to be handled. They will be cautioned not to discuss interview results with others outside the evaluation. Within the evaluation team, discussions will be restricted to the essential needs of a particular set of site visits.

• An initial letter of invitation from the National Science Foundation will inform all individuals that their participation in the CPATH evaluation study is voluntary and that, if they are willing to participate, their privacy will be assured. This assurance will be reiterated at the time data collection begins. Participants will also be informed of the purposes of the data collection and the potential uses of the data collected.

• Prospective interviewees will be given a Consent Form that includes the same assurance of confidentiality, as well as the purposes of the study, potential risks and discomforts, and benefits of participation.

• Personal information (names, addresses, phone numbers, email addresses) will be collected solely for the purpose of identifying and contacting study participants, and will not be distributed outside the site visit team.

• All electronic recordings of interviews, interview notes, and other project-related documents will be stored in secure areas that are accessible only to authorized staff members. Electronic files and databases will be stored on a secure server and will be accessible only to authorized staff members. Access to response databases, as well as to other electronic and hard-copy materials used to record collected data, will be limited to Nancy Adelman (PI), Raymond McGhee (Co-PI), and only those researchers who are granted access by the PI or Co-PI.

• All interview results recorded on paper containing identifiable data will be shredded as soon as the need for the hard copies no longer exists.

• All basic computer files will be duplicated on backup servers to allow files to be restored in the event of unrecoverable loss of the original data. These backup files will be stored under secure conditions in an area separate from the location of the original data.

• Reports to NSF will include participants’ responses only in aggregate form. Responses will not be associated with any specific institution or individual. No information that could be used to identify individuals or their institution will be revealed to anyone outside the study team.

A.11. Questions of a Sensitive Nature

The interview protocols for Principal Investigators (and Co-PIs), administrators, faculty, and project staff request information on the participation of females, students with disabilities, and minorities (especially groups underrepresented in undergraduate computer education), and which groups, if any, are being targeted by the project. The student focus group protocol asks students about the degree of diversity in their project-related classes with respect to these same groups. No personal information is requested about any particular individual that could be used to identify that person, and reporting of the requested information is voluntary. Respondents may choose not to provide information that they feel is privileged. The interview protocol for external partners does not include any questions of a sensitive nature.

The basic data on these demographic groups are being collected because diversity in education is of key interest to NSF and to the entire nation. This information will provide context in the study analyses and will be reported only in aggregate form.

A.12. Estimates of Response Burden

For this clearance request, the evaluation study relies on interviews with CPATH Principal Investigators (and Co-PIs), administrators, university faculty, project staff, project evaluators external partners, and students. The interview protocols used in this data collection appear in Appendix A. This section provides estimates for site visit and telephone interviews.

A.12.1. Number of Respondents, Frequency of Response, and Annual Hour Burden

The data collection instruments for this revised clearance include the six previously used interview protocols which are unedited and the addition of a new project evaluator interview. As was mentioned in our previous clearance but not included, this revised clearance also contains the new faculty telephone interview protocols. For all groups except Principal Investigators/Co-PIs, respondent burden consists of the time spent being interviewed at their sites, expected to average one hour. Principal Investigators will spend an estimated four hours, in addition to the interview time, working with their staff on: compiling lists of faculty, staff, administrators, students, and partners to be interviewed; helping to arrange interviews; gathering documents; and meeting with the site visitors. Respondents will not incur any equipment, postage, or travel costs. We will continue interviews through 2013 with the additional protocols.

The table below shows the annual number of respondents and the annual hour burden for the site visit interviews, faculty telephone interviews and other PI activities. The total number of individuals to be interviewed during site visits and telephone interviews in 2011–2013 is estimated to be 340 annually. The annual burden, across all six interview protocols, is estimated at 340 person hours. In addition, the 10 PIs whose sites will be visited will spend an additional 50 hours (combined) on advance preparations for the site visit interviews and in meetings with contractor staff during the site visits. The annual burden is calculated by multiplying the number of anticipated respondents by the estimated response burden per person.

Annual Burden Hours by Instrument/Activity

Instrument/Activity (Years of Data Collection)	Respondent Type	Number of Respondents	Burden Hours Per Respondent	Annual Burden Hours
Site Visit Interview & Telephone Interview Protocols ( 2011, 2012, 2013)	CPATH Principal Investigators/Co-PIs, administrators, faculty, project staff, external partners, students	340	1	340
Preparation for Site Visits; On-Site Meetings (2010, 2011, 2012, 2013)	CPATH Principal Investigators	10 (also included above)	5	50

A.12.2. Hour Burden Estimates by Each Form and Aggregate Hour Burdens

Section A.12.1 above gives estimates of the combined annual burden for the seven site visit interview protocols and one telephone interview protocol. This section provides burden estimates for each of the eight protocols. The annual burden for each of the data collection instruments is calculated by multiplying the number of anticipated respondents by the estimated response burden per person. The time estimates are based on prior experience with interviews of this nature.

Annual Burden Hours by Respondent Type

Respondent Type	Number of Respondents	Burden Hours Per Respondent	Annual Burden Hours
CPATH Principal Investigators	10	5	50
CPATH Co-PIs	10	1	10
University Administrators	20	1	20
CPATH Faculty (on-site)	50	1	50
CPATH Project Staff	20	1	20
CPATH External Partners	20	1	20
CPATH Project Evaluator	10	1	10
CPATH Students	100	1	100
CPATH Faculty (Telephone Interview)	100	1	100
TOTAL, All Interviewees	340	5 for 10 PIs 1 for 340 others	380

In FYs 2007–2009, NSF made awards to 96 institutions for 69 CPATH projects. Site visits will continue annually over the next three years. For the duration of this data collection activity, the total number of respondents is estimated to be 1,020, with a total burden of 1,140 hours.

Interviews will be conducted at 10 sites each year. Selection of sites will be based on three of the project types used to classify awards in FYs 2007–08 (community building; institutional transformation; evaluation, adoption, and extension); project maturity; models and approaches used (new curricular or pedagogical development, supporting communities of practice, other); Carnegie classification of institution; and geographical location. Interviews will be conducted with a variety of key informants including: Principal Investigators/Co-PIs, university administrators, faculty, project staff, project evaluators, external partners, and students. With the possible exception of university administrators, participants will be actively involved in the site’s CPATH project or very knowledgeable about it. The program evaluation contractor will identify a representative number of individuals from the various groups to be interviewed, working with the PI(s) for each project.

Approximately 25 participants will be interviewed at each site, for an annual total of 340 and an estimated annual burden of 380 hours. The number of respondents in each group is approximate; the final distribution across groups and the total number may vary among CPATH awardees in order to fully understand the unique experience of each awardee.

Principal Investigators (including Co-PIs): The PIs/Co-PIs of each site will be interviewed. The total number to be interviewed annually is estimated at 20 (an average of 2 PIs/Co-PIs per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The 10 primary PIs are also responsible for compiling lists of potential interviewees, helping to arrange interviews, gathering documents, and meeting with contractor staff on-site, for an estimated 4 hours per PI. The total respondent burden per primary PI is estimated at 5 hours. The annual burden for the 10 primary PIs is estimated at 50 hours, and the annual burden for the 10 Co-PIs is estimated at 10 hours, for a total of 60 person hours for this group.

Administrators: The total number of administrators to be interviewed annually is estimated at 20 (2 per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The total annual burden for this group is estimated at 20 hours.

Faculty: The total number of faculty members to be interviewed annually is estimated at 50 (5 per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The total annual burden for this group is estimated at 50 hours.

Project Staff: The total number of project staff to be interviewed annually is estimated at 20 (2 per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The total annual burden for this group is estimated at 20 hours.

External Partners: The total number of external partners to be interviewed annually is estimated at 20 (2 per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The total annual burden for this group is estimated at 20 hours.

Project Evaluators: The total number of project evaluators to be interviewed annual is estimated at 10 (1 per site). It is expected that 100% of these individuals will be interviewed as the meetings will be pre-arranged. Each interview will take about 1 hour. The total annual burden for this group is estimated at 10 hours.

Students (focus group): The total number of undergraduate students expected to participate in focus groups is 100 (10 per site). It is expected that 100% of these individuals will participate as the focus groups will be pre-arranged. Each focus group session will take about 1 hour. The total annual burden for this group is estimated at 100 hours.

CPATH Faculty (telephone): A random, stratified sample of 100 faculty will be interviewed by telephone during the spring of 2011 and 2013. These interviews will be pre-arranged and will last approximately 1 hour. The total annual burden for this group is estimated at 100 hours.

A.12.3. Estimates of Annualized Cost to Respondents for the Hour Burdens

The table below gives the overall annual cost, based on labor burden, for all site visit interviewees, and also the annual cost for each type of respondent. The total annual cost for all interviewees is estimated to be $11,114.80. The cost for each type of respondent is calculated by multiplying the total annual burden hours by their average hourly rate.

Annual Cost to Site Visit Interviewees for Burden Hours, by Respondent Type

Respondent Type	Number of Respondents	Burden Hours Per Respondent	Total Annual Burden Hours	Average Hourly Rate	Estimated Total Annual Costs
CPATH Principal Investigators	10	5	50	$35.45	$1,772.50
CPATH Co-PIs	10	1	10	$35.45	$354.50
University Administrators	20	1	20	$71.00	$1,420.00
CPATH Faculty	50	1	50	$35.45	$1772.50
CPATH Project Staff	20	1	20	$23.09	$461.80
CPATH External Partners	20	1	20	$35.45	$709.00
CPATH Project Evaluator	10	1	10	$35.45	$354.50
CPATH Students (undergraduates)	100	1	100	$7.25	$725.00
CPATH Faculty (telephone interview)	100	1	100	$35.45	$3,545.00
TOTAL, All Interviewees	340	5 for 10 PIs 1 for 340 others	380	$7.25 – $71.00	$11,114.80

The estimated hourly rate for PIs, Co-PIs, and faculty is based on national median salaries for associate professors in computer and information sciences, education, engineering, engineering technologies, and mathematics and statistics. The average median salary of these five job titles combined is $73,740. Divided by the 2,080 hours in a standard work year, this calculates to an average hourly rate of $35.45. The source of this information is the 2008/2009 National Faculty Salary Survey, conducted by the College and University Professional Association for Human Resources (CUPA-HR), www.higheredjobs.com/salary.

The faculty rate is also used for the external partners and evaluators to be interviewed. Because partners and evaluators include individuals from a variety of sectors (e.g., business, K–12 schools), there is likely to be a wide range of actual salaries. The faculty rate is considered to be a reasonable proxy for a mix of partner and evaluator rates.

The rate for university administrators is based on national median salaries in higher education for the job titles of CEO, single institution (e.g., President), Provost, and Deans of Arts and Sciences, Computer and Information Sciences, Education, Sciences, and Undergraduate Programs. The average median salary of these seven job titles combined is $147,692. Divided by the 2,080 hours in a standard work year, this calculates to an average hourly rate of $71.00. The source of this information is the 2008/2009 Administrative Compensation Survey, conducted by the College and University Professional Association for Human Resources (CUPA-HR), www.higheredjobs.com/salary .

The rate for project staff is based on national median salaries for the job titles of research computer specialist and research assistant and senior research assistant in natural/physical sciences. The average median salary of these three job titles combined is $48,029. Divided by the 2,080 hours in a standard work years, this calculates to an average hourly rate of $23.09. The source of this information is the 2008/2009 Mid-Level Administrative and Professional Salary Survey, conducted by the College and University Professional Association for Human Resources (CUPA-HR), www.higheredjobs.com/salary.

The hourly rate for undergraduate students is based on minimum wage information effective July 24, 2009, obtained from the U.S. Department of Labor at http://www.dol.gov/dol/topic/wages/minimumwage.htm.

A.13. Estimate of Annualized Capital and Maintenance Costs to Respondents

There are no respondent costs associated with these data collections beyond those included in the estimates presented in Section A.12.

A.14. Estimates of Annualized Costs to the Federal Government

The estimated total cost to the Federal government of all data collection, analysis, and reporting activities associated with the CPATH program evaluation is $2,066,429 (including data collection activities already completed). The average annual cost to the Federal government is estimated at $413,286. The CPATH contract period covers five years, from FY 2009 to FY 2013. Site visit interviews were conducted in 2010 and will continue in 2011, 2012, and 2013. Faculty phone interviews will be conducted in 2011 and 2013. Site visit briefing reports, summaries of phone interview results, and annual reports are drafted over the course of the project. The final report will be delivered in September 2013.

The table below breaks down the total cost of the CPATH evaluation to the Federal government by labor vs. other direct costs. Figures are given for the overall 5-year period of the contract and for an average year. The average annual cost was obtained by dividing the total cost by 5. These estimates are based on actual figures for the first year and modified contractual amounts for the second year. The projected budgets for the final three years are expected to be about the same as the contractual amounts for the second year.

Estimated Total and Annual Cost to the Federal Government

	Total Cost for 5 Years	Average Annual Cost
Labor	$1,886,119	$377,224
Other Direct Costs*	$180,310	$36,062
Total, All Costs	$2,066,429	$413,286

* Includes local and non-local travel, materials and supplies, report production, telephone and fax communications, shipping, support costs, and G&A on support costs.

A.15. Changes in Burden

The overall annual burden on interviewees increases from 300 hours to 340 hours. While we are visiting two fewer sites, we are now including project evaluators (10 additional interviewees) and non-site visit faculty (100 additional telephone interviews). The estimated annual burden for the individual respondents will remain the same (5 hours for PIs and 1 hour for all other respondents). A.16. Schedule and Plans for Data Collection and Reports

Timeline for Data Collection, Analysis, and Report Writing. The evaluation of the CPATH program is being conducted over the course of five fiscal years, FY 2009 through FY 2013. Work on this evaluation began in late 2008 with a review of existing CPATH project reports and project evaluations, and development of a logic model and evaluation plan to guide the overall evaluation. Our first round of data collection took place April through June 2010 with site visits to 12 CPATH project sites. These visits included interviews with Principal Investigators/Co-PIs, administrators, faculty, project staff, external partners, and students. These annual site visits will be continued in January through May of 2011-2013 and include interviews with the aforementioned individuals plus the project evaluator.

Analysis will be ongoing from the beginning of data collection in Spring 2010 through September 2013, when the final report will be delivered to NSF. A briefing report was prepared in July 2010 on the Spring 2010 site visits. Future briefing reports on the site visits will be prepared in May 2011 and 2013. In April-May of 2011 and 2013, faculty telephone interviews will be conducted. Summaries of the telephone interview results will be prepared in June 2011 and 2013.

Annual reports synthesize the findings to date from all document reviews and data collections relative to the evaluation questions. The report in the first year focused on identifying trends and patterns of implementation across CPATH awards, implementation challenges, levels of participation, and the extent to which new models are developing and how community building is occurring throughout the field of computing education. The year two report will have the same focus. Reports in the later years will highlight new emerging models for advancing computational thinking and identifying breakthrough interventions that have promise. Reports in the later years will also focus on examining the accumulation of evidence on program effectiveness.

The schedule for CPATH data collections and reports is presented in the table below.

Schedule of CPATH Data Collections and Reports

Project Activity		Time Frame
Site Visit Interviews	Conduct site visits	January–May 2010 (completed), 2011, 2012, 2013
	Prepare briefing report on site visits	May 2010 (completed), 2011, 2012, 2013
Faculty Telephone Interviews	Interview faculty by telephone	March-April 2011, 2012
	Prepare summary of telephone interview results	May 2011, 2012
Annual Reports	Draft report, revise after NSF review, submit revised report to NSF	2010 (completed) January–August 2011-2013
Final Report	Deliver final version of last annual report reflecting all accumulated evidence	September 2013

Publications.

Before reporting the results, NSF Program Directors and the Contracting Officer Technical Representative examine the quality of the data and the reporting; review the document to ensure that there are no errors in the descriptions of the program and its awardees; and assess compliance with privacy laws, regulations, and policies. After the review, if the COTR and Lead Program Director are in agreement for the report to be available to selected audiences (e.g., project directors for grants) or to the general public, then the COTR can give approval to SRI do so. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF, i.e., NSF is the exclusive publisher of the information being gathered. Often it is only after seeing the quality of the information delivered by the study that NSF decides the format (raw or analytical) and manner (in the NSF-numbered product Online Document System (ODS) or simply a page on the NSF Web site) in which to publish.

A.17. Approval to Not Display Expiration Date

Not applicable.

A.18 Exceptions to Item 19 of OMB Form 83-I

No exceptions apply.

3