Revised Clearance: National Science Foundation, Directorate of Education and Human Resources, Division of Graduate Education
Data Collection for the Evaluation of the Computer & Information Science & Engineering’s Pathways to a Revitalized Undergraduate Computing Education Program (CPATH)
SECTION B
Two types of data collections are undertaken for the CPATH evaluation: 1) site visit interviews with seven types of respondents, and 2) telephone interviews of faculty. The site visit interviews were conducted in 2010 and will continue through 2013. The faculty phone interviews will be administered in 2011 and 2013. Both data collections will be conducted with a sample of all potential respondents.
Site Visits
The interview data collection involves a two-stage sampling process: selection of a sample of CPATH project sites to be visited followed by selection of a sample of CPATH participants to be interviewed at each of the selected sites. The sampling method is nonprobabilistic, purposive, and heterogeneous. CPATH sites are selected as a broad representation of the various project models, and CPATH interviewees at the selected sites will be drawn from compiled lists to reflect seven different respondent groups: PIs/Co-PIs, university administrators, faculty, project staff, project evaluators, external partners, and students.
Site visits will occur annually over four years, with approximately 10–12 sites visited each year. The first series of site visits began in April 2010. In the spring of 2011, we will visit 10 sites, a sample of 13% of the institutions that received CPATH awards in FYs 2007, 2008, and 2009. Some projects involve more than one institution or organization, so the total number of sites is larger than the total number of projects.
The 36 awards made in FY 2009 will be included in site visits beginning in 2011.. The table below presents the universe of sites to date.
Universe of CPATH Sites, FY 2007 to FY 2009
Fiscal Year of Award |
Number of Sites (Awardees) |
Number of Projects |
Estimated Number of Sites to be Visited in 2011 |
2007 |
43 |
29 |
3 |
2008 |
17 |
15 |
4 |
2009 |
36 |
25 |
3 |
Total to Date |
96 |
69 |
10 |
The universe of CPATH participants who are potential interviewees is unknown at this time but is in the process of being compiled. There are three potential sources of this information: 1) past annual reports from awardees, which can provide preliminary estimates of numbers; 2) the CPATH data monitoring tool, which will collect the numbers of the different types of participants each year and also the names of PIs, Co-PIs, and faculty members on each project (this collection awaits clearance under EHR’s generic clearance); and 3) lists of names compiled by the PIs.
Similar to the first year of data collection, a sample size of 25 individuals per site is expected to be sufficient to represent the views of the different respondent types. The total number of interviewees per year was initially estimated at 250, but with the addition of the project evaluators and faculty telephone survey, the estimate is at 390. Multiplied by 3 years of data collection, the total number of individuals to be interviewed is expected to be around 1,170. With the exception of university administrators, all potential interviewees must be actively involved or very knowledgeable about a site’s CPATH project. All Principal Investigators will be interviewed, for a total of 10 PIs at the 10 sites visited annually. Some Co-PIs will also be interviewed. The other individuals will be selected to be representative of the five other respondent groups: university administrators, faculty, project staff, external partners, and students. The primary PI for each project will work with the site visitor team to identify the individuals to be interviewed.
Faculty Telephone Survey
The faculty telephone survey data collection involves a purposive sampling process. The selected faculty will come from a broad representation of the four different project models. The pool of faculty come from grants that are funded beyond June of 2011 (this includes grants with no cost extensions), include faculty other than the PI participating, and are not slated for site visits in the Spring of 2011. The purpose of the telephone survey is to gauge faculty perceptions about CPATH implementation and the potential of their projects to emerge as model that could be adapted or transferred elsewhere. The interviews with faculty members will provide important information about the implementation of the model that will be critical in the process of examining how different CPATH strategies or approaches are developing potential models.
Site Visits
Interviews with the seven respondent types will be conducted during site visits, which occur annually over the next three years. Interviews are conducted at 10–12 sites each year. The site visit teams participated in two training sessions in December 2009 and February 2010, and the first series of visits began in April 2010. Refresher training will occur in each year prior to data collection.
Each team consists of two members who meet with all respondent types except for students, who are interviewed in a focus group setting. Persons who are not on site at the time of the visit are interviewed by telephone. Each interview takes about one hour. Some sites are visited in more than one year, so it is possible that some interviewees may be interviewed more than once at different points in time. Because these interviews are scheduled ahead of time in collaboration with the PI, response is expected to be 100%.
The main limitation of this data collection is that CPATH projects vary considerably and only a sample of all projects can be selected. However, the evaluators have extensive experience in conducting site visit interviews for the purpose of program evaluation, and in selecting sites and individuals that will appropriately represent the universe. The contractor will propose sites to NSF, based on a thorough review of CPATH project documents.
A potential problem in focus groups is dominance of the discussion by a small number of enthusiastic participants. Focus groups can be an efficient method of obtaining information because they provide a non-threatening setting for students to discuss their experiences in computing courses in a spontaneous and open manner. Individual interviews may be intimidating for students. Conducting focus groups with students in a relaxed, safe environment alongside their peers can facilitate candid and honest discussions in which diverse points of view can emerge from the participants. The contractor has used focus groups successfully in a variety of data collection activities on many program evaluations. Each site visit team conducting these focus groups will include an experienced moderator who will ensure that all students have a chance to respond to every question.
Faculty Telephone Interviews
Faculty members at each CPATH site play a central role in the efforts to define the competencies and tools used in computational thinking and to develop curricula and pedagogical innovations to support computational thinking. SRI will survey a sample of faculty across the different project types, drawing on faculty knowledge of curricular and pedagogical strategies to advance computational thinking as well as faculty culture and institutional climate. SRI will administer the questionnaire twice, once in spring of 2011 and again in the spring of 2013.
Stage 1: Selection of Sites
As noted in the original OMB clearance, the first stage of sample selection for this data collection entails selecting the CPATH sites to be visited each year. 2011 sites will be selected in order to ensure representation of the following 4 distinct reform models:
Curriculum Development
Pedagogical Delivery
Interdisciplinary Models for Computer Science Programs
Interdisciplinary Models for non-Computer Science Programs
The ten sites selected will be based on the quality of model implementation. The table below presents the total number of awards in the CPATH universe for FYs 2007, 2008, and 2009 and their distribution among the 4 models identified above. The number of sites to be visited in each of the subsequent years is expected to be about the same (10–12).
Universe of CPATH Awards
Selection Criteria |
FY 2007 |
FY 2008 |
FY 2009 |
Total Number of Sites (Awards)* |
39 |
17 |
36 |
Models/Approaches Used |
|||
Curriculum Development |
15 |
2 |
5 |
Pedagogical Delivery |
9 |
4 |
13 |
Interdisciplinary Models – Computer Science |
3 |
6 |
5 |
Interdisciplinary programs – Non-Computer Science |
12 |
5 |
13 |
Stage 2: Selection of Interviewees
The second stage of sample selection for this data collection entails selecting the individuals to be interviewed at each site. The evaluation contractor will identify informants using extant personnel lists on CPATH annual reports. Interviews will be conducted with seven types of informants:
Principal Investigators/Co-PIs
University administrators
Faculty
Project staff
Project evaluators
External partners
Students
With the exception of university administrators, all potential sample members must be actively involved or very knowledgeable about a site’s CPATH project. All Principal Investigators will be interviewed, and some Co-PIs will also be interviewed. The remaining sample members will be selected to be representative of the other six groups. The primary PI for each project will work with the site visitor team to identify the individuals most appropriate for interviews.
Approximately 25 CPATH participants will be interviewed at each site – the primary PI plus 24 others, for an annual total of 300 across 10 sites. The students will be interviewed together in a focus group setting. All other interviews will be conducted individually, and most of them will be face-to-face at the site. Sample members who are not on site during the visit will be interviewed by telephone; this is expected to be the case with external partners. Because the interviewees will be selected in collaboration with the PI, a 100% response rate is expected.
We intend to select a purposive sample of faculty members from CPATH institutions
to interview using a telephone interview protocol. The number of faculty members in the universe is approximately 180 based on compiling names of participating faculty drawn from annual reports. A random sample of 100 faculty will be drawn from this population, representing faculty implementing different models to improve computing education.
The estimated number of interviewees in each respondent group is presented in the table below (faculty interviewed by telephone are not included). These numbers are approximate averages across sites. The final distribution among groups and the total number per site are likely to vary, as the goal is to fully understand the unique experience of each awardee. It is anticipated that some sites will require interviews with more than 25 people and others will require fewer interviews.
Sample of Interviewees for 2010 Site Visits, by Respondent Type
Respondent Type |
Number per Site |
Total for 10 Sites |
CPATH Principal Investigators/Co-PIs |
2 |
20 |
University Administrators |
2 |
20 |
CPATH Faculty |
6 |
60 |
CPATH Project Staff |
2 |
20 |
CPATH Project Evaluators |
1 |
10 |
CPATH External Partners |
2 |
240 |
CPATH Students |
10 |
100 |
TOTAL, All Interviewees |
25 |
250 |
Not applicable to the site visit interviews.
The site visit interviews and faculty telephone interviews will be broadly representative of the CPATH projects and of the various types of individuals working on them. Because these interviews will be pre-arranged with the assistance of the PIs, a 100% response is anticipated.
Not applicable to the site visit interviews.
Not applicable. The site visit interviews are conducted annually. One round was completed in 2010 with the original protocols and will continue through 2013.
Because the interviews will be pre-arranged with the cooperation of the PIs, a 100% response is expected.
The interview protocols were pretested with 3 Principal Investigators representing 3 different types of CPATH awards. The pretests suggest that the interviews will take approximately one hour to complete, with allowance for questions from the interviewee. The pretesters were debriefed following the pilot interviews for the purpose of identifying any issues with comprehension of the questions, knowledge to answer the questions, organization, and flow. The faculty telephone interview protocols were piloted by 2 different faculty members at 2 types of CPATH award institutions.
Agency
Bernice T. Anderson, National Science Foundation, 703.292.5151
Paul W. Jennings, National Science Foundation, 703.292.5307
Harriet G. Taylor, National Science Foundation, 703.292.8950
Contractors
SRI International will be responsible for data collection and analysis, under the direction of Nancy Adelman, 703.247.8434.
| File Type | application/msword |
| File Title | Supporting Statement (3145-0204) |
| Author | Nyema Mitchell |
| Last Modified By | nmitchell |
| File Modified | 2010-12-23 |
| File Created | 2010-12-23 |