This data collection will be a census of the universe of individuals funded by the ADVANCE Fellows program and of the first two cohorts of ADVANCE IT grantees.
The universe of ADVANCE projects to be included in this data collection is shown in Exhibit 4. Since the full universe of projects will be included, no statistical sampling methods will be implemented.
Population |
Universe size |
Sample size |
IT Cohorts 1 and 2 |
19 |
19 |
Fellows |
59 |
59 |
For the Fellows survey, we will include all 59 recipients of the 2002 and 2004 ADVANCE Fellows awards, and for the IT survey, all 19 institutions in the first two (2001 and 2003) cohorts of IT grantees.
For both the IT and Fellows components, each dataset will first be analyzed descriptively and then combined in a series of multivariate statistical models.
We will first generate descriptive statistics for each item in both the IT and Fellows surveys. In both surveys, the majority of these items will only apply to the treatment groups. Descriptive analyses will help to answer questions about how IT grantees and the Fellows implemented their projects and some of the resulting outcomes. In addition, since there is an internal pre-post component to the IT design, we will also look at changes in these outcomes over time across the grantees.
Comparative analyses will examine the extent to which observed differences in outcomes can be attributed to the ADVANCE program. For the IT component, we will compare changes in outcomes for IT grantees to aggregated responses from the SDR for other similar four-year universities and colleges that have not subsequently received ADVANCE IT awards. For analytic purposes, for the Fellows QED component, we plan to use a regression approach that examines the relationship between program status and relevant outcomes while controlling for other confounding variables. Specifically, we anticipate using a special type of regression, two-level hierarchical linear modeling (HLM). HLM produces more accurate results than ordinary least square models by modeling the hierarchical structure in the organization (e.g., individual faculty nested within institutions) correcting for aggregation bias, misestimated precision, and the unit of analysis problem. At level 1, individual outcomes will be predicted as a function of program status (treatment vs. comparison) and other demographic and academic characteristics. At level 2, institutional characteristics will be used as covariates. The differences between the treatment and comparison groups will be assessed in terms of 1) p-values, or the probability that the observed differences could be due to chance, and 2) effect sizes, which measure the magnitude of the differences.
We believe we have done as much as possible within the constraints of the relatively small number of grantees served and the requirements of the evaluation design to achieve sufficient rigor to support causal claims for both the IT and Fellows QED components. The likelihood of being able to support such claims is quite high for the Fellows component. However, for the IT component, while the QED design is rigorous, the statistical power is somewhat lacking. The extent to which we can make definitive statements in this regard will also depend on the data to be collected. Needless to say, our write-up of the results will appropriately reflect any needed caveats and considerations.
Most of the statistics produced will be descriptive. However, in cases where group differences are examined statistically, for the Fellows component we plan to set the α at 0.05 level. For the IT component, where the sample size is 19, we will explore whether to use the Wilcoxon signed rank test, a nonparametric test that does not require that the data be normally distributed.
No unusual problems are anticipated.
As described previously, only one data collection is required for both the IT and Fellows components.
IT ADVANCE PIs and their designees will receive several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. An explanatory cover letter will accompany the IT survey outlining the instructions, definitions, and guidelines for completing the form. The cover letter also provides the names, telephone numbers, and email addresses for contractor staff who can be contacted should respondents have questions or concerns regarding the data collection or the evaluation more broadly. Within one week of sending the email package, Westat project staff will follow up by email and/or telephone to make sure that the materials were received and to schedule a mutually acceptable date and time for the individual WebEx videoconference session with respondents from that institution. This will also provide project staff the opportunity to answer any additional questions respondents may have.
As noted, Westat staff will directly facilitate the IT survey individually for each institution using WebEx technology as well as video via webcam. Throughout the data collection period, we will also provide respondents with online and telephone support to address specific issues concerning information requested on the survey. We anticipate a 100 percent response rate for both the IT survey and the outcome data indicator form.
The Fellows survey will be mailed via U.S. mail to the most recent addresses available for the 59 former ADVANCE Fellows awardees. We will first send out an email to the verified email addresses of the former Fellows with an advance letter introducing the study and requesting the best mailing address to which to mail the survey as well as a telephone number to call in case we need to contact them. If the email does not yield a response, we will use an address location service. We expect that barring extremely unusual circumstances, we will be able to obtain up-to-date contact information for all of the awardees.
In the cover letter that accompanies the survey, respondents will be asked to return the survey to Westat in a self-addressed stamped envelope by a specified date approximately two calendar weeks from the date of receipt. The letter will also contain the names, email addresses, and telephone numbers of Westat staff available to assist respondents with any questions or concerns they may have about the survey or the evaluation. Immediately after the due date for the survey, Westat will contact those who have not yet returned their surveys to remind them that the surveys are now past due. The following week, Westat staff will contact all individuals who still have yet to respond and offer to conduct the survey by telephone at a mutually convenient time. Past experience suggests that guaranteeing the highest possible response rate will likely require getting on the telephone with about 35 percent, or 20, of the 59 respondents. We have budgeted for the possibility of administering up to half of the surveys by telephone, although we expect that most Fellows will be highly motivated to respond. Our goal is a 100 percent response rate for the Fellows as well.
Both data collection forms have been reviewed by experts in the field to ensure that the methods will be properly implemented. An expert Advisory Panel has been consulted about the overall methodology as well as the specific data collection methods to be used in this evaluation. In addition, as described in Section A.8., seven representatives of IT grant projects and two experts in survey design have been consulted in development and testing of the surveys and other data collection materials.
Suzanne
Plimpton, Reports Clearance Officer, Information Dissemination
Branch,
National Science Foundation, 703.292.7556
Kelly Mack, Program Director, ADVANCE, National Science Foundation, 703.292.8575
Elmima
Johnson, Division of Research, Evaluation, and Communication, COTR,
National Science Foundation,703.292.5137
Bernice Anderson, Office of the Assistant Director, National Science Foundation, EHR, 703.292.5151
Westat will be responsible for data collection and analysis under the direction of Dr. Susan Berkowitz, 301-294-3936 or [email protected].
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | splimpto |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |