OMB Responses to NSF's CCI passback

Statement to OMB

CCI Evaluation

Design Limitations

We acknowledge the limitations of the proposed design and have developed our analytic approach to minimize their effect to the extent possible. A key limitation of the design is that it does not use a rigorous quasi-experimental approach or randomized control trial design that would allow us to make conclusions about whether the observed outcomes were caused by the CCI Program.

Given the length of time the CCI Program grants have been operating and the variation across sites regarding administrative records, projects, staffing, and potential confound, it’s likely that a more rigorous evaluation design may not necessarily be feasible. The design of the evaluation is realistic given the circumstances, and appears appropriate for the purposes of gaining an understanding of the contributions of the CCI Program.

For questions about research productivity, this limitation is partially mitigated by using a comparison group of PIs of individual-investigator grants in the bibliometric analysis to gain a better understanding of the benefits of a center mechanism. However, these comparison groups have their own limitations, which include funding history of individual investigators (who may have also participated in other centers) and important differences between the programs being compared.

Concur. This limitation is partially mitigate by a comparison group. The comparison can reasonably add some evidence of the effect of the CII Program regarding research productivity.

The second limitation is that the information collected from program participants may be prone to social desirability bias. That is, respondents may exaggerate their accomplishments and minimize their challenges. Reliance on additional data sources which are not subject to this bias, such as publications and program documents, particularly external documents from oversight activities, partially mitigates this problem.

Less concerned about reporting on accomplishments as these are products and there is the assumption that reports and administrative data will provide some evidence of growth or change (trends). More concerned about the perspectives on contributions of center funding to the process (“enabled changes as a result of CCI funding …”).

It may be reasonable to assume that the person responding to the survey may have written the report or at least has read it? Unfortunately, there are few options around this and the information is valuable for partially assessing effect. Importantly, the Grad Student survey results and Industry survey results may provide a measure of concurrence and strengthen the validity of the findings.

Third, our design includes heavy reliance on program data. While clearly an extremely valuable source of information, these documents present challenges due to consistency and scope of reporting. In fact, our review of the documents revealed that CCIs vary greatly in the level of detail they choose to include in annual reports. Furthermore, in some cases we observed variability from year to year within a center. For example, it was not always possible to tell whether some program or activity described was the same or different from the previous year. Our approach to mitigating these challenges was twofold. First, we used an Access database to abstract and code the open-ended data. All coders were trained and their work was monitored to increase inter-rater reliability. Second, to summarize the information, we decided against counting instances, and instead focused on capturing the types of activities the centers engaged in or the types of accomplishments they reported.

These two solution work well to address some of the data challenges. Types of activities and accomplishments are clear and consistent outcomes for this analysis and focusing on these themes is helpful.

Triangulating with Other Data Sources

As noted above, one way to mitigate the bias due to self-reporting is to collect data from multiple data sources. For most of the items in the Principal Investigator (PI)/Co-Investigator (Co-I) survey and interview that OMB raised concerns about, we will also rely on program documents, including from individuals involved in oversight of the centers. Some examples of additional data sources are listed below.

• For Survey item 13 (Please indicate whether the following improvements have occurred as a result of CCI, focused on URGs and students), we will compare to information provided in annual reports and feedback from individuals responsible for oversight on URGs and student experiences. We will also triangulate PI/co-I survey responses with student responses about their experiences.

• For Survey item 16 (To what extent have these elements [various center activities characteristics] contributed to the success of your center?), we will compare to information provided in annual reports and feedback from individuals responsible for oversight on center activities, characteristics, and successes.

As indicated, these efforts to triangulate have some merit.

• For Interview questions 9 and 10 (What do you see as the most important scientific and non-scientific accomplishments of the Center? Would they have been possible without the center? If not, how were these accomplishments enabled by the Center?), we will compare to information provided in annual reports and feedback from individuals responsible for oversight on research accomplishments.

This is an interview question and is important for a perspective on how the CCIs function and are productive. And again, cross validating is an acceptable approach to attempt to mitigate bias however it should be acknowledged that most of the participants in the survey have an interest in the Center’s success.

• For Interview question 13 (What aspects of the Center evolved in unexpected ways and/or deviated from the original goals? Did these changes lead to positive outcomes for your Center?), there are no other data sources, but the question does not necessarily rely on original goals (which we do have high level articulations of in the phase 1 and phase 2 proposals). We expect respondents to describe any major changes in plans that were unexpected.

Reporting on the Findings

For findings based on self-report, in contrast to counts of objective indicators such as publications or students, we will make it clear that these findings represent self-reporting by respondent. We will make clear the data source(s) contributing to each statement and also report on the count of respondents in data tables.

For narrative findings on objective measures such as years of participation in the center, we will report descriptive statistics based on the information reported. However, for items that are based on perceptions, we might include a statement such as “PI/Co-I perceptions of the contribution of the center to increases in various student outcomes were high, with X percent reporting that students and postdocs are more easily able to obtain a position after leaving CCI, and Y percent reporting that the quality of education in chemistry for students in their center increased.” As another example, we would report the percent of PIs/Co-Is who thought that certain outcomes increased or decreased and whether they thought these increases were due to CCI or not. When possible, we would also triangulate with other data sources as described above.

These efforts to triangulate the findings from the different data sources are appropriate in order to provide a reasonable assessment of the CCI Program. Where necessary, framing findings as the perceptions of effect is critical given that potential changes to the CCI Program model may result from this assessment. Additionally, these findings may inform decisions to more broadly adopt Center based research models and phased funding mechanisms, resulting in a substantial amount of funding for these type of programs. Consequently, it is critical to stress the limitations of this evaluation.