3145-STC Section B

Section B

Statistical Methods

B.1 Respondent Universe and Sampling Methods

The universe of respondents for which the clearance is sought includes the universe of students and postdoctoral fellows who participated in the STCs. The target population for this survey is all graduate students in the 17 Science and Technology Centers in the U.S. It is important for the study to have students selected from each of these Centers.

For this survey, a representative probability sample of students will be selected. The sampling frame for the selection of students in each Center has been obtained from the program monitoring database that contains information extracted from the annual reports submitted by Centers to NSF. Data are available through 2009, therefore the 2010 numbers are projections; once OMB approval is received, Centers will be contacted to update their participant information. The projected number of former student participants for each Center is provided in the table below.

Student Participants in Each Center, and Projected Number of Former Participants for Study

Initial Year of Center Funding	Center	Students Participants as of 2009 (current and former)	Former Student Participants as of 2009	Projection of Former Participants as of 2010*
2002	CAMPWS	203	118	142
2000	CBN	113	51	61
2002	CBST	125	77	92
2002	CENS	320	183	220
2000	CERSP	219	147	176
2002	CISM	71	45	54
2006	CLiPS	47	14	17
2002	CMDITR	249	131	157
2006	CMMAP	31	1	1
2006	CMOP	26	8	10
2006	CMORE	50	13	16
2005	CReSIS	107	44	53
2000	CfAO	168	80	96
2000	NBTC	230	153	184
2002	NCED	153	88	106
2000	SAHRA	196	154	185
2005	TRUST	179	101	121
	TOTALS	2487	1408	1690
* 2010 Projection was calculated as 20% increase over 2009

The sample size for the study was based on the requirement to have a sample selected from each of the 17 centers and to have a margin of error for sample percentages of some characteristics of interest (such as as yes/no on participation in specific STC activities) around plus or minus 3.5 percentage points at 95% confidence level. The sample size for the required precision is around 800. Assuming an 80% response rate, we propose to select a sample of 1,000 students.

The sampling design for the survey will be a stratified random sample with 17 Centers as strata. The total sample of students will be allocated to each Center in proportion to the total number of students in that Center. The table below shows the distribution of projected former graduate students in the population by Center and also the sample size allocated to each Center.

Distribution of the Population and Sample by Strata

Center	Number of Students in the Population	Number in the Sample
CAMPWS	142	84
CBN	61	36
CBST	92	54
CENS	220	130
CERSP	176	104
CISM	54	32
CLiPS	17	10
CMDITR	157	93
CMMAP	1	1
CMOP	10	6
CMORE	16	10
CReSIS	53	31
CfAO	96	57
NBTC	184	109
NCED	106	63
SAHRA	185	109
TRUST	121	71
Total	1690	1000

Students within Centers belong to different institutions. For the selection of the sample we propose to sort the list of students within each stratum by institution and then select an equal probability systematic sample. Because of stratification and proportional allocation of the sample to strata, we expect the precision of the estimates to be slightly higher than the precision stated above.

We anticipate a response rate of at least 75 percent from the respondent group based on previous surveys conducted with students of NSF-sponsored programs.

Response rates were projected based on similar surveys conducted with samples of graduate students and early career researchers who participated in NSF programs. Table 3 illustrates the response rates for various evaluation studies of NSF programs that surveyed graduate students and early career individuals, which were used to estimate the expected response rates for this project.

Response Rates in Previous Studies Used to Predict Response Rate for Current Effort

Program	Response Rate	Length of Time Between Participation and Data Collection
CAREER Fellows	84%	0-10 years
IGERT Former Students	74%	0-10 years
GK-12 Fellows	MS 45% PhD 57%	5-10 years
GK-12 Fellows	MS 83% PhD 92%	0-5 years

B.2 Procedures for the Collection of Information

This request for clearance includes data collection from student participants via an on-line survey. During the design of the survey, the survey was pilot-tested with current and former STC participants. Although surveys were not pre-tested in the on-line format, they were pre-tested as electronic forms. In addition, many of the questions were crafted in from similar surveys conducted on evaluations of the National Science Foundation’s programs, including evaluations of the Interdisciplinary Graduate Research Training program, which was conducted on-line. Respondents will be surveyed only once.

Respondents will receive an email invitation describing the goals of the study and the procedures being used. The invitation email will also contain a link to the survey. Three reminder emails will be sent only to non-respondents two, four, and six weeks following the initial invitation. The survey will be closed at the end of eight weeks. Individuals who did not respond after two reminder emails will be contacted by telephone.

Given the descriptive nature of the information sought, the use of simple descriptive statistics—such as counts, ranges, and frequency—is most appropriate for the analyses of the data; for example, frequencies of students who participated in particular types of center activities.

For producing population-based estimates, each responding student will be assigned a sampling weight. This combines a base weight which is the inverse of the probability of selection of the student and an adjustment for student nonresponse. The student weights may be further adjusted using poststratification adjustments or ranking to agree with known subgroup population totals. Standard errors of the estimates will be produced.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

Several methods will be used to maximize response rates and to deal with non-response, these include:

• Sending an email from the NSF program officer to inform respondents about the survey;

• Providing a sufficient timeframe for data collection. The survey will be fielded over an eight-week period and will be extended for an additional two weeks, if necessary;

• Providing a toll-free number that students can call to ask questions and verify the legitimacy of the survey; and

• Following up with non-respondents via email reminders and phone calls.

Bias in a survey estimate because of nonresponse consists of two components. The first is the nonresponse rate and the second is the difference between respondents and nonrespondents in the population parameter that is being estimated. For example, if we are estimating a population percentage by selecting a simple random sample and computing the sample percentage and there is nonresponse, the bias in the sample percentage due to nonresponse is given by

where is the sample percentage based on respondents, is the response rate, is the population percentage among the respondents and is the population percentage among the nonrespondents. Therefore, it is important to examine both the response rate and the differences between the responding and nonresponding groups in the analysis of bias in the estimates due to nonresponse. We will make every attempt to get a high response rate.

We will examine the bias in estimates because of nonresponse by some graduate students in the sample following the four steps described below. Based on the analysis we will adjust the sampling weights of responding students to account for student nonresponse.

1. Examination of Response Rates

The first step will be to monitor the overall response rate and the response rates in each stratum (Center) and also for some subgroups like gender and race. High response rates (over 80 percent) for the entire sample but also for subgroups might indicate that there is no need for further analysis of bias due to nonresponse. Large differences in the response rates by strata and for subgroups serve as indicators that potential bias may exist. For example, if response rate from an important subgroup is very low then any difference in the characteristic of interest between this subgroup and other subgroups would result in a bias in the estimates. From the survey results we will examine whether there are differences in the characteristics in the subgroups especially in a stratum where the response rate is low.

2. Comparison of Sample and Frame Estimates

We will use the sampling weight based on the probability of selection of responding students without any nonresponse adjustment and the data from these students to compute population estimates of some characteristics available (not used for stratification at the time of selection of schools) on the sampling frame. These estimates will be compared with the population values. If there is a large difference between the estimate and the population after accounting for sampling error, then this may be an indication of the bias in the estimates as these are based only on respondents.

3. Comparison of estimates based on respondents to estimates from external sources.

For questions where there is some data available from an external source for some characteristic of interest (e.g graduation rate), we will compare the estimates from our survey responses to those from nationally available data. A large difference may indicate bias in the survey estimates assuming that the external source provides an unbiased estimate.

4. Nonresponse Propensity Model

Finally, should the response rate fall below 80 percent we will construct a propensity model to estimate the probability of a student in the sample responding to the survey both for responding and nonresponding students; this is called a propensity score. The estimated propensity scores come from a logistic regression model. The model will be based on variables which are available both for nonresponding and responding students. Students will be grouped using the estimated propensity scores. Within each group we will compare the frame characteristics of responding and nonresponding students. This grouping in addition to assessing the bias will also provide a method of forming weighting classes for adjusting the weights of responding students to reduce the bias due to nonresponse.

B.4 Test of Procedures or Methods to be Undertaken

Experts in the field—including the STC Directors and the principal investigator from the American Association for the Advancement of Science (AAAS) who will be leading the program review—reviewed the draft and final instruments. The survey was pilot-tested with five current or former STC students. Respondents were asked to comment on the clarity and content on the questions and to record the time required to complete the survey. Minor revisions, including shortening the length of the survey, resulted from this feedback. (The median time to completion was 38 minutes. The survey was shortened by removing individual items and sections in order to reduce the respondent burden to 30 minutes.)

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor for collection and analysis of data in this study is Abt Associates Inc., Cambridge, MA. Staff have knowledge of statistical methods, experience in evaluation of research programs, and expertise in scientific research. K.P. Srinath, statistician and expert in survey sampling and methodology was consulted in the preparation of the sampling, estimating, and nonresponse bias plans.

Key personnel from Abt have been involved in the statistical aspects and who will be involved in collecting and analyzing data are presented in the table below, along with key members of the AAAS review who may also be involved in the analysis of data.

Abt Associates
Alina Martinez, EdD	Project Director, Abt Associates	617-349-2312
Hilary Rhodes, PhD	Task Leader, Survey of STC Students	617-349-3516
Luba Katz	Associate	617-349-2313
Jennifer Carney	Project Technical Advisor	301-634-1747
K.P. Srinath	Statistician, Survey Sampling and Methodology	(301) 634-1836
AAAS
Daryl Chubin	Principal Investigator, Review of STC Program	202-326-6785
Irwin Feller	Project Director, Review of STC Program	814-865-0691

Attachments

Attachment 1. STC Graduate Student Survey

Attachment 2. First Federal Notice