Part_B_Supporting Statement_7-16-09

B.

COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Selection method for Research Projects in the Multiple Case Study Design

This is an exploratory study designed to develop hypotheses about the relationship between the characteristics of projects and innovation and collaboration/communication. Because we are not drawing a random sample of research projects--even within the population of public research laboratories--special care must be taken in the design of the selection of research projects. Hence, we have endeavored to approximate a quasi-experimental longitudinal research design with multiple cases (see for example Experimental and Quasi Experimental Designs for Generalized Causal Inference by Shadish, Campbell and Cook, 2001).

For this study, attempts have been made to identify some of the major sources of variation in the selection of the cases so that there is some level of external validity. One major source of variation is the context of the national research laboratories, which vary in terms of their mission, history, size and organizational cultures. For this reason, we have selected five national laboratories within the Department of Energy (Brookhaven, Pacific Northwest, Sandia, Ames, and the National Renewable Energy Laboratory) and one mission agency, the National Oceanographic and Atmospheric Agency (NOAA).

In a similar vein, the literature suggests that another major potential source of variation is the background of a project’s research area and the resulting current discipline or mix of disciplines of project team members. Therefore, we have selected five major research areas: chemistry, biology, material sciences, alternative energy, and geophysical science (see Table 1). These research areas vary in terms of their history of funding, the required skill mix, the mixture of basic and applied research, and current policy priorities. For example, alternative energies is related to the desire for independence from oil, the geophysical sciences are related to the problem of global warming and control over carbon, and the material sciences are central in the concerns of developing nanotechnologies. Again, these research area differences would provide more confidence in any patterns found to be common to several research areas. However, this exploratory study is not intended to explain or draw any conclusions about differences that are seen among research areas. As can be observed in Table 1, each of the five research areas is represented in at least two national laboratories.

For the purposes of this study, departments are defined as the internal, functional divisions within each laboratory. Collaborative programs are categorized in two different ways. First, collaborative programs are organizational units that sponsor projects involving researchers from different departments within a single laboratory, such as the MESA Center at Sandia National Laboratories. Second, collaborative programs are organizational units that sponsor projects involving researchers from diverse organizations, such as the Center for Integrated Nanotechnologies which brings together researchers from Sandia and Los Alamos National Laboratories.

Table 1. Potential Research Sites
Identified Laboratories	Identified Research Areas/Departments	Identified Collaborative Programs/Initiatives
Brookhaven	1. Chemistry Department	1. Environmental Sciences
	2. Biology Department	2. Center for Translational Neuroimaging
Pacific Northwest	1. Microbiology/Cell Biology and Biochemistry Divisions	1. Global Change Institute
	2. Chemical and Materials Divisions	2. Joint Center for Nanoscience
Sandia	1. Geosciences	1. Combustion Research Facility
	2. Material Sciences	2. Center for Integrated Nanotechnologies
Ames Laboratory	1. Division of Materials Sciences and Engineering	1. Institute for Physical Research and Technology
	2. Chemical and Biological Sciences Program
National Renewable Energy Laboratory	1. Biological Science Groups and Chemical and Nano-sciences Group	1. National Center for Photovoltaics
	2. Concentrating Solar Power Group	2. National Wind Technology Center
National Oceanic and Atmospheric Administration	1. Chemical Science (Earth Systems Research Laboratory)	2. Joint Center for Satellite Data Assimilation
	2. Air Resources	3. Regional Mesoscale Meterology Branch

Within this multiple case study, we will further select research projects that focus on two variations that are associated with innovation: the relative strategic emphasis on high risk research, and the relative focus on broad scope or systemic research projects. The classification of the selected projects will be done in consultation with the middle managers that are in charge of the departments/centers listed in Table 1. Our goal is to have sixteen projects (eight small and eight large, four each reflecting an emphasis on high risk) in two departments and two collaborative programs in each of the three large laboratories of Pacific Northwest, Brookhaven, and Sandia National Laboratories, and eight projects in one department and one center in each of the three smaller national laboratories. In total, we anticipate selecting up to 72 research projects.

The respondents for this study will consist of upper-level managers, selected middle managers, and the project leaders, and scientific staff on the selected projects. Table 2 provides the number of anticipated respondents, but further information is provided in the table, “Detailed Estimates of Respondents and Interviewee Burden and Cost” found in Form A.

Table 2	Respondents
Project Staff Survey	1,008
Project Leader Interviews	72
Middle Manager Interviews	42
Top Manager Interviews	18

The selection process of the projects will be done through discussion with the middle managers of the departments/centers indicated above in Table 1. They will be asked to select projects on the basis of two important criteria:

• Relative emphasis on high risk research;

• Relative emphasis systemic or broad scope research.

The managers will be asked about qualitative evidence that justifies their categorization including peer reviews, publications or patents, the kinds of discoveries made, the difficulty of the problem, and the like. After the projects are selected, we will interview the project leaders on how much and what kind of technical progress the project has set as its goal, and this will be compared to the middle manager’s categorization. Further, we plan to followup with the project leaders in two years time in order to validate the categorization of the project and to gauge the amount of innovation.

All members of the selected projects will receive a survey that measures 40 project characteristics that might be associated with either the amount of innovation and/or the amount of cross-fertilization/collaboration. These 40 project characteristics have been culled from a research project funded by Office of Basic Energy Sciences within the Department of Energy that used the following techniques to identify what might be important:

1. Literature review of the science and the management of innovation literatures;

2. Many focus groups of scientists and engineers;

3. Survey field tests at Sandia National Laboratories, Pacific Northwest National Laboratory and Ford’s industrial research laboratory;

4. A three wave panel study of the STAR division of NOAA.

Thus, there is considerable evidence that this is a meaningful list to explore in this qualitative study.

Middle managers will be interviewed about the nature of the research area and their policies to encourage complex research teams and cross-fertilization of ideas. Top management will be given structured interviews about the laboratory’s policies to build diverse work teams and encourage the exchange of information. Since our multiple case study includes centers where transdisciplinary research is occurring, our design might provide some insights for policies designed to stimulate transdisciplinary collaborations.

Given the purposive nature of the selection of projects in a multiple case study, we can not generalize to the population of research projects even in national laboratories. But at the same time, our quasi-experimental longitudinal design attempts to speak to some of the concerns about external validity by varying the research area and the national laboratory.

2. Establishing Patterns in the Data Analysis

Since this is a qualitative multiple case study with a small number of projects (72), which are further classified into several sub-categories, the most appropriate statistical test is ANOVA or analysis of variance. The analysis will first examine whether any significant associations between any of the 40 practices in the survey and the amount of innovation, where innovation has been categorized by middle managers and by the actual measurement from the reports of the 72 project leaders. Another set of analyses will examine the relationship between any of the 40 practices in the survey and the amount of reported cross-fertilization and number of collaborations obtained from the interviews and surveys.

We believe it is unlikely that any particular practice will stimulate innovation or cross-fertilization in all kinds of research projects. Therefore, our next step in the analysis is to explore different ways of characterizing research projects. We will examine: 1) the impact of strategic emphasis on basic science vs. high risk research; 2) the impact of strategic emphasis on small scope vs. large scope projects; and 3) the combination of these two. Any findings, of course, would then have to be demonstrated in a much larger study that would represent the population of research projects in at least public research laboratories and mission agencies...

Once this basic analysis is completed, we will begin the second stage of the exploration for hypotheses for future study by examining the following contextual factors:

• Each national laboratory vs. the other five;

• Each of the five research areas vs. the other four.

When some significant differences are detected, an attempt will be need to determine the causes that best explain the differences, principally through the interviews of laboratory leadership.

3. The Unit of Analysis and Ensuring a High Participation Rate

Since our unit of analysis is projects and not individuals, we would average scores of the team members. One important advantage of this procedure is that is some people do not respond to the survey and the response is voluntary, then we can still obtain reasonable estimates of the projects characteristics as long as we have at least three individuals reporting on a specific project. In addition, we will contact middle managers through contacts at the Department of Energy, which increases the likelihood of interest in the study. The middle managers will be asked for us to be present at one of the departmental/center meetings where we can explain the project to them and how it will be used to develop ideas about projects characteristics that are associated with innovation and collaboration.

It is expected that the response rate will be higher than 50 percent because the survey will be administered at a regular project meeting, and middle managers will have been involved in project selection. For those project members unable to attend the project meeting, we will send them the survey by mail and follow-up with two email reminders. In early tests of the survey, response rate was typically 80 percent or higher for surveys administered this way.

4. Pre-testing of the List of 40 Project Characteristics

As the survey items have been previously culled and tested, further testing of methods is not proposed. Furthermore, in the panel study of the STAR division at NOAA, some of the recommendations that followed the first wave of research were implemented by the director, particularly those concerning communication and collaboration. It resulted in a considerable increase in the innovation rate over the next two years and improvement in morale. Thus, there is some experimental evidence that some of the practices are related to innovation.

5. Name and Telephone

Sandia National Laboratory assisted by the University of Maryland will oversee the survey and conduct the data analysis. SNL contact: Gretchen Jordan, Principal Member of Technical Staff, Department 01012, Phone:  505-844-9075; Email: [email protected].