3145-0209 Part B

3145-0209

II. Section B

B.1. Respondent Universe and Sampling Methods

Telephone interviews will be conducted with the ADVANCE Project Directors at all IT grantee institutions (N=19) in the cohorts of interest and hence no sampling methods are needed.

Case study site selection will involve purposeful sampling. Through the document review and analysis of telephone interviews, we have identified two implementation models. We will select three examples of each model to conduct a total of 6 case studies. We will select sites in three stages. In the first stage, we will divide the universe of grantee sites by model (9 in model 1 and 10 in model 2). In the second stage, we will classify sites according to their progress in gender equity at the time of funding (as a measure of prior efforts in the field). We want to ensure that we have institutions represented in each of the three categories of prior effort (significant, some, no prior efforts). In the third stage, we will draw sites within each of these categories while ensuring, whenever possible, adequate representation of the following characteristics: (a) institutional size (to distinguish large from medium/small institutions, N=2); and (b) geographic location.

B.2. Information Collection Procedures/Limitations of the Study

This request for clearance includes two types of data collection—telephone interviews and case studies (site visit interviews and focus groups).

The collection will be based on protocols developed for this evaluation. The development of these protocols—for telephone and in-person interviews and focus groups—will be informed by a review of related research, relevant theoretical literature, project documents and, in the case of site visit protocols, results from the telephone interviews.

Telephone Interviews. After a letter of introduction informing grantees of the interviews and requesting cooperation, interviewees will be contacted to schedule interviews. A team of four interviewers will carry out the telephone interviews, which will take approximately one hour to one hour and fifteen minutes each. The team of expert interviewers will undergo training designed to familiarize them with the goals of the evaluation, the purpose of the interviews, the instruments, the specific goals of each question, the method for data entry of interview data, and security measures to follow. Interviews will be confidential and will be tape recorded if permission to record is granted by the informant.

Site Visits. Each site selected for case study will be contacted to seek participation. After obtaining agreement from the sites, evaluation staff will work with a representative at each of the case study sites to identify prospective participants and to coordinate interviews and focus groups that make up the data collection activities of the site visit. Teams of three experienced site visitors will conduct two-day visits to each case study site. Site visitors will undergo training to brief them on the purpose of the site visits and to familiarize them with the data collection instruments. Site visitors will be provided with background information on each of their sites, including the telephone interview data, proposals and other relevant reports, and institutional background data.

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

This study does not include survey questionnaires, but does include telephone interviews with the project directors at each of the nineteen ADVANCE projects funded by NSF in 2001 and 2002, as well as case studies at six sites. Experience with prior evaluations suggests that we should not encounter a participation problem.

Nonetheless, we will engage in activities that, in the past, have helped increase participation:

● Encourage study participation by making a presentation at the annual JAM meeting so that project staff understand the importance of the evaluation and have an opportunity to ask questions and raise concerns;

(this activity was conducted in June 2009)

● Procure from NSF ADVANCE program directors a letter of introduction informing grantees of the interviews and/or case study site visits, and requesting cooperation prior to contacting interviewees for a telephone interview or to schedule site visits;

● Allow sufficient time for data collection. For example, telephone interviews will be carried out over the course of two months to ensure that busy schedules of respondents can be accommodated;

● Design protocols to target the specific information that each respondent group is likely to possess.

B.4. Tests of Procedures or Methods

We field tested the interview protocol with directors/principal investigators at two different ADVANCE sites that, according to documents reviewed, were employing strategies commonly found at ADVANCE projects (such as recruitment and policy revisions).  As expected, both interviews lasted about one hour (one slightly less).  Because two questions did not have to be administered as neither site had PAID or a Leadership awards, we estimate that the few interviews conducted at sites that have these awards could last slightly more than one hour. We took this into account in our burden estimates.

After completing the interview, we asked respondents to comment on the clarity and wording of the questions. Evaluation project staff reviewed and discussed the information gathered during the field tests and subsequently used results to revise the attached interview protocol. Revisions included the addition of a question, clarification of instructions to interviewers, and reordering of two questions.

We are currently in the process of field testing the site visit instruments. The drafts are based on similar instruments that we have used in the past. Consequently, we do not anticipate having to make major revisions to stay within the expected time to complete the interview. But experience suggests that the wording or ordering of some questions may need minor revisions. If revisions to the attached protocols are made, we will forward to OMB the final set of instruments prior to data collection.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit:

• Suzanne Plimpton, Reports Clearance Officer Information Dissemination Branch, National Science Foundation, 703.292.7556.

• Jessie DeAro, Program Director ADVANCE, National Science Foundation, 703.292.5350.

• Kelley Mack, Program Director ADVANCE, National Science Foundation, 703.292.8575.

• Elmima Johnson, Division of Research, Evaluation, and Communication, COTR National Science Foundation, 703.292.5137.

• Bernice Anderson, Office of the Assistant Director, National Science Foundation, EHR, 703.292.5151.

Contractor or Grantee:

• Clemencia Cosentino de Cohen, Co-PI, ADVANCE Evaluation, Mathematica Policy Research, 202.264.3499.

• Beatriz Chu Clewell, Co-PI, ADVANCE Evaluation, Evaluation Studies and Equity Research Program (PEER), The Urban Institute, 202.261.5617.

The Urban Institute will be responsible for the evaluation design, data collection and analyses under the direction of the Co-PIs of this evaluation, Beatriz Chu Clewell (202.261.5617) and Clemencia Cosentino (202.264.3499).

ATTACHMENTS

Federal Register Notice (Vol. 75, No. 151, Friday, August 6, 2010)

Telephone Interview Protocol*

Draft Correspondence and Introductions*

Case Study Protocols

• IT Lead Co-PI

• IT Leader

• IT Department Chair

• IT Provost

• IT Project Coordinator

• IT Additional Grantee

• IT Focus Groups

• Leadership Grantee

• PAID Grantee

*Received clearance (OMB # 3145-0209)