Request for Clearance: National Science Foundation, Directorate of Education and Human Resources, Division of Graduate Education
Data Collection for the Evaluation of the Computer & Information Science & Engineering’s Pathways to a Revitalized Undergraduate Computing Education Program (CPATH)
SECTION B
Two types of data collections will be undertaken for the CPATH evaluation: 1) site visit interviews with six types of respondents, and 2) a survey of faculty. The site visit interviews will be conducted in 2010, 2011, 2012, and 2013. The faculty survey will be administered in 2010 and 2012. Both data collections will be conducted with a sample of all potential respondents.
Only the site visit interviews are being submitted for clearance at this time. A revised package with details on the faculty survey will be submitted in 2010.
The interview data collection involves a two-stage sampling process: selection of a sample of CPATH project sites to be visited, followed by selection of a sample of CPATH participants to be interviewed at each of the selected sites. The sampling method is nonprobabilistic, purposive, and heterogeneous. CPATH sites will be selected to be broadly representative of the various project types and strategies, and CPATH interviewees at the selected sites will be drawn from compiled lists to reflect six different respondent groups: PIs/Co-PIs, university administrators, faculty, project staff, external partners, and students.
Site visits will occur annually over four years, with approximately 10–12 sites visited each year. The first series of site visits is scheduled to begin in March 2010. The 2010 site visits will take place at a 20% sample of institutions that received CPATH awards in FYs 2007 and 2008, as their projects will have had time to become operational. Some projects involve more than one institution or organization, so the total number of sites is larger than the total number of projects. The universe of sites for FYs 2007 and 2008 is 60: 43 for FY 2007 (29 projects); 17 for FY 2008 (15 projects).
The 36 awards made in FY 2009 will be included in site visits beginning in 2011. New awards in subsequent years will be added to the site visit schedule when the projects have been in operation for two years. The table below presents the universe of sites to date.
Universe of CPATH Sites, FY 2007 to FY 2009
Fiscal Year of Award |
Number of Sites (Awardees) |
Number of Projects |
Estimated Number of Sites to be Visited in 2010 |
2007 |
43 |
29 |
9 |
2008 |
17 |
15 |
3 |
2009 |
36 |
25 |
0 |
Total to Date |
96 |
69 |
12 |
The universe of CPATH participants who are potential interviewees is unknown at this time but is in the process of being compiled. There are three potential sources of this information: 1) past annual reports from awardees, which can provide preliminary estimates of numbers; 2) the CPATH data monitoring tool, which will collect the numbers of the different types of participants each year and also the names of PIs, Co-PIs, and faculty members on each project (this collection awaits clearance under EHR’s generic clearance); and 3) lists of names compiled by the PIs.
Although the size of the participant universe has yet to be determined, a sample size of 25 individuals per site is expected to be sufficient to represent the views of the different respondent types. The total number of interviewees per year is estimated at 300. Multiplied by 4 years of data collection, the total number of individuals to be interviewed is expected to be around 1,200. With the exception of university administrators, all potential interviewees must be actively involved or very knowledgeable about a site’s CPATH project. All Principal Investigators will be interviewed, for a total of 12 PIs at the 12 sites visited annually. Some Co-PIs will also be interviewed. The other individuals will be selected to be representative of the five other respondent groups: university administrators, faculty, project staff, external partners, and students. The primary PI for each project will work with the site visitor team to identify the individuals to be interviewed.
Interviews with the six respondent types will be conducted during site visits, which will occur annually over four years. Interviews will be conducted at 10–12 sites each year. The site visit teams will participate in two training sessions in December 2009 and February 2010, and the first series of visits will begin in March 2010. Refresher training will occur in each year of data collection.
Each team will consist of two members who will meet individually with all respondent types except for students, who will be interviewed in a focus group setting. Persons who are not on site at the time of the visit will be interviewed by telephone. Each interview is expected to take about one hour. Some sites may be visited in more than one year, so it is possible that some interviewees may be interviewed more than once at different points in time. Because these interviews will be scheduled ahead of time in collaboration with the PI, response is expected to be 100%.
The main limitation of this data collection is that CPATH projects vary considerably and only a sample of all projects can be selected. However, the evaluators have extensive experience in conducting site visit interviews for the purpose of program evaluation, and in selecting sites and individuals that will appropriately represent the universe. The contractor will propose sites to NSF, based on a thorough review of CPATH project documents.
A potential problem in focus groups is dominance of the discussion by a small number of outgoing participants. Care must be taken to bring all attendees into the discussion. The contractor has extensive experience in monitoring focus groups to ensure that all students have a chance to respond to every question.
Stage 1: Selection of Sites
The first stage of sample selection for this data collection entails selecting the CPATH sites to be visited each year. Sampling of sites to be visited in 2010 will be based on the following criteria, with the goal of achieving a broad representation of the variety of CPATH projects and institutional characteristics:
Project maturity: FYs 2007 and 2008 awards only (operational for at least two years)
Project type: Community Building (CB); Institutional Transformation (IT); Evaluation, Adoption, and Extension (EAE)
Models and approaches: new curricular or pedagogical development; supporting communities of practice; other
Carnegie classification of institution: Doctorate-granting; Master’s; Baccalaureate/Associate’s; Other
Geographic location: Northeast; South; Midwest; West
The table on the next page presents 1) the total number of awards in the CPATH universe for FYs 2007 and 2008 and their distribution by the sample selection criteria, and 2) the estimated number of sites that will be selected for the 2010 site visits, based on those same criteria. The number of sites to be visited in each of the subsequent years is expected to be about the same (10–12), with more recent awardees added to the mix.
Universe of CPATH Awards and 2010 Sample of Sites to be Visited, by Sample Selection Criteria
|
Universe (actual) |
Site Visit Sample (estimated) |
||
Selection Criteria |
FY 2007 |
FY 2008 |
FY 2007 |
FY 2008 |
Total Number of Sites (Awards)* |
43 |
17 |
9 |
3 |
Type of award |
|
|
|
|
Community Building (CB) |
27 |
8 |
5 |
2 |
Institutional Transformation (IT) |
4 |
2 |
2 |
1 |
Evaluation, Adoption, & Extension (EAE) |
10 |
0 |
2 |
0 |
Carnegie Classification of Institution |
|
|
|
|
Doctorate-granting |
24 |
8 |
4 |
3 |
Master’s |
8 |
2 |
1 |
1 |
Baccalaureate/Associate’s |
6 |
0 |
3 |
0 |
Other |
3 |
0 |
0 |
0 |
Models/Approaches Used |
|
|
|
|
New Curricular/Pedagogical Development |
24 |
6 |
2 |
2 |
Supporting Communities of Practice |
7 |
1 |
1 |
0 |
Other |
10 |
3 |
3 |
4 |
Geographic Location |
|
|
|
|
Northeast |
12 |
5 |
2 |
2 |
South |
12 |
2 |
2 |
1 |
Midwest |
7 |
1 |
2 |
1 |
West |
10 |
2 |
2 |
0 |
Stage 2: Selection of Interviewees
The second stage of sample selection for this data collection entails selecting the individuals to be interviewed at each site. Interviews will be conducted with six types of informants:
Principal Investigators/Co-PIs
University administrators
Faculty
Project staff
External partners
Students
With the exception of university administrators, all potential sample members must be actively involved or very knowledgeable about a site’s CPATH project. All Principal Investigators will be interviewed, and some Co-PIs will also be interviewed. The remaining sample members will be selected to be representative of the other five groups. The primary PI for each project will work with the site visitor team to identify the individuals most appropriate for interviews.
Approximately 25 CPATH participants will be interviewed at each site – the primary PI plus 24 others, for an annual total of 300 across 12 sites. The students will be interviewed together in a focus group setting. All other interviews will be conducted individually, and most of them will be face-to-face at the site. Sample members who are not on site during the visit will be interviewed by telephone; this is expected to be the case with external partners. Because the interviewees will be selected in collaboration with the PI, a 100% response rate is expected.
The estimated number of interviewees in each respondent group is presented in the table below. These numbers are approximate averages across sites. The final distribution among groups and the total number per site are likely to vary, as the goal is to fully understand the unique experience of each awardee. It is anticipated that some sites will require interviews with more than 25 people and others will require fewer interviews. (It is not possible at this time to provide the actual universe of individual CPATH participants because those data will be collected from PIs through the CPATH program’s online data monitoring tool, which is awaiting approval under EHR’s generic clearance.)
Sample of Interviewees for 2010 Site Visits, by Respondent Type
Respondent Type |
Number per Site |
Total for 12 Sites |
CPATH Principal Investigators/Co-PIs |
2 |
24 |
University Administrators |
2 |
24 |
CPATH Faculty |
5 |
60 |
CPATH Project Staff |
2 |
24 |
CPATH External Partners |
2 |
24 |
CPATH Students |
12 |
144 |
TOTAL, All Interviewees |
25 |
300 |
Not applicable to the site visit interviews.
The site visit interviews will be broadly representative of the CPATH projects and of the various types of individuals working on them. Because these interviews will be pre-arranged with the assistance of the PIs, a 100% response is anticipated.
Not applicable to the site visit interviews.
Not applicable. The site visit interviews will be conducted annually over the course of the next four years, 2010–2013.
Because the interviews will be pre-arranged with the cooperation of the PIs, a 100% response is expected.
The interview protocols were pretested with 3 Principal Investigators representing 3 different types of CPATH awards. The pretests suggest that the interviews will take approximately one hour to complete, with allowance for questions from the interviewee. The pretesters were debriefed following the pilot interviews for the purpose of identifying any issues with comprehension of the questions, knowledge to answer the questions, organization, and flow.
Agency
Bernice T. Anderson, National Science Foundation, 703.292.5151
Paul W. Jennings, National Science Foundation, 703.292.5307
Harriet G. Taylor, National Science Foundation, 703.292.8950
Contractors
SRI International will be responsible for data collection and analysis, under the direction of Nancy Adelman, 703.247.8434.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement (3145-0204) |
| File Modified | 0000-00-00 |
| File Created | 2021-02-03 |