This data collection will be a census of the universe of individuals funded by the ADVANCE Fellows program and of the first two cohorts of ADVANCE IT grantees.
The universe of ADVANCE projects to be included in this data collection is shown in Exhibit 4. Since the full universe of projects will be included, no statistical sampling methods will be implemented.
Population |
Universe size |
Sample size |
IT Cohorts 1 and 2 |
19 |
19 |
Fellows |
59 |
59 |
For the Fellows survey, we will include all 59 recipients of the 2002 and 2004 ADVANCE Fellows awards, and for the IT survey, all 19 institutions in the first two (2001 and 2003) cohorts of IT grantees. In addition, we plan to collect in-depth outcome data at the departmental level for a smaller subset of IT grantees. However, this will be a convenience sample of IT grantee institutions that have already collected data at this level or would be willing to work with us to collect and report such data. We estimate that four, or at most five, institutions will be able to do this activity.
For both the IT and Fellows components, each dataset will first be analyzed descriptively and then combined in a series of multivariate statistical models.
We will first generate descriptive statistics for each item in both the IT and Fellows surveys. In both surveys, the majority of these items will only apply to the treatment groups. Descriptive analyses will help to answer questions about how IT grantees and the Fellows implemented their projects and some of the resulting outcomes. In addition, since there is an internal pre-post component to the IT design, we will also look at changes in these outcomes over time across the grantees.
Comparative analyses will examine the extent to which observed differences in outcomes can be attributed to the ADVANCE program. For the IT component, we will compare changes in outcomes for IT grantees to aggregated responses from the SDR for other similar four-year universities and colleges. For the Fellows program, we propose taking a regression approach that examines the relationship between program status and relevant outcomes while controlling for other confounding variables. Specifically, we anticipate using a special type of regression, two-level hierarchical linear modeling (HLM). HLM produces more accurate results than ordinary least square models by modeling the hierarchical structure in the organization (e.g., individual faculty nested within institutions) correcting for aggregation bias, misestimated precision, and the unit of analysis problem. At level 1, individual outcomes will be predicted as a function of program status (treatment vs. comparison) and other demographic and academic characteristics. At level 2, institutional characteristics will be used as covariates. The differences between the treatment and comparison groups will be assessed in terms of 1) p-values, or the probability that the observed differences could be due to chance, and 2) effect sizes, which measure the magnitude of the differences.
Most of the statistics produced will be descriptive. However, in cases where group differences are examined statistically, for the Fellows component we plan to set the α at 0.05 level. For the IT component, where the sample size is 19, we will explore whether to use the Wilcoxon signed rank test, a nonparametric test that does not require that the data be normally distributed.
No unusual problems are anticipated.
As described previously, only one data collection is required for both the IT and Fellows components.
IT ADVANCE PIs and their designees will receive several types of assistance to ensure that they understand the importance of the data collection effort, their responsibilities for providing the data, and the technical aspects of data submission. An explanatory cover letter will accompany the IT survey and outcome indicator data form outlining the instructions, definitions, and guidelines for completing both forms. The cover letter also provides the names, telephone numbers, and email addresses for contractor staff who can be contacted should respondents have questions or concerns regarding the data collection or the evaluation more broadly. Within one week of sending the email package, Westat project staff will follow up by email and/or telephone to make sure that the materials were received and to schedule a mutually acceptable date and time for the individual WebEx videoconference session with respondents from that institution. This will also provide project staff the opportunity to answer any additional questions respondents may have.
As noted, Westat staff will directly facilitate the IT survey individually for each institution using WebEx technology as well as video via webcam. Throughout the data collection period, we will also provide respondents with online and telephone support to address specific issues concerning information requested either on the survey or the quantitative indicator data collection form, which respondents will be completing over a two- to three-month period after the survey session. As the due dates for submission of the indicator data form approach, Westat will monitor the response patterns of individual projects to identify those needing additional prompting and/or assistance. We anticipate a 100 percent response rate for both the IT survey and the outcome data indicator form.
The Fellows survey will be mailed via U.S. mail to the most recent addresses available for the 59 former ADVANCE Fellows awardees. Westat uses National Change of Address (NCOA), a company that charges a nominal fee to check addresses to make sure they are current and accurate. Should any of the mailings still be returned to sender due to problems with the address, Westat project staff will conduct an Internet search to try to locate an institutional affiliation and/or email address for the individual. We expect that barring extremely unusual circumstances, we will be able to obtain up-to-date contact information for all 59 of the awardees by using one of these methods. In the cover letter that accompanies the survey, respondents will be asked to return the survey to Westat in a self-addressed stamped envelope by a specified date approximately two calendar weeks from the date of receipt. The letter will also contain the names, email addresses, and telephone numbers of Westat staff available to assist respondents with any questions or concerns they may have about the survey or the evaluation. Immediately after the due date for the survey, Westat will contact those who have not yet returned their surveys to remind them that the surveys are now past due. The following week, Westat staff will contact all individuals who still have yet to respond and offer to conduct the survey by telephone at a mutually convenient time. Past experience suggests that guaranteeing the highest possible response rate will likely require getting on the telephone with about 35 percent, or 20, of the 59 respondents. We have budgeted for the possibility of administering up to half of the surveys by telephone, although we expect that most Fellows will be highly motivated to respond. Our goal is a 100 percent response rate for the Fellows as well.
All three data collection forms have been reviewed by experts in the field to ensure that the methods will be properly implemented. An expert Advisory Panel has been consulted about the overall methodology as well as the specific data collection methods to be used in this evaluation. In addition, as described in Section A.8., seven representatives of IT grant projects and two experts in survey design have been consulted in development and testing of the surveys and other data collection materials.
Suzanne
Plimpton, Reports Clearance Officer, Information Dissemination
Branch,
National Science Foundation, 703.292.7556
Kelly Mack, Program Director, ADVANCE, National Science Foundation, 703.292.8575
Elmima
Johnson, Division of Research, Evaluation, and Communication, COTR,
National Science Foundation,703.292.5137
Bernice Anderson, Office of the Assistant Director, National Science Foundation, EHR, 703.292.5151
Westat will be responsible for data collection and analysis under the direction of Dr. Susan Berkowitz, 301-294-3936 or [email protected]
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | splimpto |
| File Modified | 0000-00-00 |
| File Created | 2021-02-03 |