Attachment 6_Star Metrics Pilot Project Guide

STAR METRICS - Science and Technology in America’s Reinvestment – Measuring the EffecTs of Research on Innovation, Competitiveness and Science

Academic Institution Participation Guide

First Steps

1. About the STAR METRICS Program

People have asked important questions about the impact of federal investments in science, particularly with respect to job creation and economic growth. It is important to collect and analyze data so that such questions can be answered in a credible fashion. There is currently no data infrastructure that systematically couples science funding with outcomes. There are also no mechanisms that exist to engage the public with the scientific funding.

However, there are substantial existing investments that could be leveraged to remedy the situation. Federal agencies already collect data on federal investments at the award, individual, and institutional level for the purposes of managing awards. Academic institutions collect data on all individuals working on projects in their financial and human resources systems. Academic researchers have collected large bodies of data on such scientific and innovation outcomes as citations, patents, business startups and IPOs . And there is a deep body of knowledge about creating measures of job creation and the associated earnings drawn from the experience of Longitudinal Employer-Household Dynamics program at the Census Bureau. Finally, there has been substantial investment in visualization and other tools that convey complex information about science to a lay audience. The existence of these separate investments motivates the STAR METRICS approach to studying the impact of science funding and disseminating the information to the public.

The STAR METRICS program is anticipated to be a broad partnership of Federal Science and Technology funding agencies with a shared vision to develop data infrastructures and products to support evidence based analyses of Science and Technology returns on investment, as well as to inform policy making. The goal of the STAR METRICS Program is to utilize existing administrative data from Federal agencies and their grantee institutions, and match them with existing research databases on economic, scientific and social outcomes.

STAR METRICS is being created in direct response to OMB and OSTP’s request that Federal agencies to develop outcome-oriented goals for their science and technology activities¹. It is also in direct response to the reporting requirements of the ARRA, and aims to provide American taxpayers with precise information on the value of their investments.

The aim of STAR METRICS is twofold. The initial goal of STAR METRICS is to provide mechanisms that will allow participating universities and federal agencies with a reliable and consistent means to account for the number of scientists and staff that are on research institution payrolls, supported by federal funds. In subsequent generations of the program, it is hoped that STAR METRICS will allow for measurement of science impact on economic outcomes (such as job creation), on knowledge generation (such as citations and patents) as well as on social and health outcomes.

The STAR METRICS Project is a working initiative of the Science of Science Policy Interagency Group and was developed from a very successful pilot project tested with the Federal Demonstration Partnership in 2009. The Office of Science and Technology Policy NSTC Committee on Science established the Science of Science Policy Interagency Group to develop an evidence-based framework for making policy investments in research and development.

2. What Data Elements are Institutions Required to Submit in order to Participate?

We will accept data in any simple comma delimited file format (CSV). We plan to provide you with sample code that will pull the relevant data sets for you. This is particularly straightforward if you use PeopleSoft or Banner.

The data elements required from the academic institutions to participate in the pilot are as follows:

	Data Element	Definition
Information on Scientists and Awards	For each Award
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University DUNS Number	The organization’s 9-digit Data DUNS number
	University Award Id	University's internal number for the award.
	Overhead Charged	The overhead amount charged in the reporting period.
	For each individual
	Anonymized Employee Id	Unique Employee ID (not Social Security number) of grant funded personnel
	Occupational Classification	Occupational classification / Job description of the funded personnel (ex. Faculty, Undergrad Student, Grad Student, Admin, Technical Support, Post grad Student)
	FTE Status	Designation of the status of the funded personnel (full time = 1.0, half time = .5)
	Proportion of time allocated to award	Calculated portion of the time expended by the funded personnel during the reporting time period.
Information on Overhead	Report to cognizant agency	Report submitted to cognizant agency that supports the recipients recovery of administrative overhead costs through an indirect cost rate.
Payments to vendors	For each payment to a vendor
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University Award Id	University's internal number for the award.
	Vendor DUNS Number	Vendor DUNS Number. The Vendor's 9 digit DUNS number
	Payment Amount	The amount invoiced to the vendor in the reporting period
Subcontracts and subawards	For each Subaward
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University Award Id	University's internal number for the award.
	Subaward recipient DUNS Number	The sub recipient organization’s 9- digit DUNS number
	Subaward Funds Disbursed	The amount of cash disbursed to the sub-awardee in the reporting period.

3. Next Steps

We would like to schedule approximately 3 hours of time for an introductory meeting (that will most likely be set up as a webinar/videoconference) to introduce you to the basic approach and products and what we would in turn, need from you in order to participate. 

The people we need to participate, including yourself, include your primary IT person, your primary contact finance person and your primary contact HR person. It would also be useful to have someone from your Vice President for Research Office

Following the introductory meeting we will follow up with your designated university team to help define the most appropriate way to approach instituting the process for developing automated ways of extracting data from your institution.

We may need your assistance to clarify fields, and you may need our help to determine the best approach to extract data from your systems and send the files to us (this needs defined).

The goal is to make the process of providing us data as simple as possible the first time, and then, easily repeatable for future updates. The next section reviews the University step by step process.

Appendix A: University Step by Step Process

The following is a more detailed description of the process that will be used by the University and STAR METRICS team to gather the data from the Universities and transmit it to the STAR METRICS data repository.

Step 1: Reviewing the project

The first step in the STAR METRICS data gathering process is to set up a meeting with the designated members of the university implementation team. The purpose of the meeting is to provide an understanding of the project and the information that will need to be collected with the. This first meeting may be done via webinar, teleconference, or on site and will cover the following topics:

• The STAR Metrics team will review details of the STAR Project, discuss the information that will be collected, the process for collection and the reports that will be generated from the data.

• The STAR METRICS Data Dictionary will be made available to the university participants prior to the meeting so that questions may be identified ahead of time.

• Following the meeting, the Star Metrics team will assist the university implementation team in developing a map of the requested fields to the target sources at the university. The sample University Data Sheet shows the desired mapping structure.

Step 2 Preparing for Data Extraction

The second step of the process involves running the data extract routines.

• Using the data sheet as input, the STAR METRICS team will review the mapping of the field level names against the sources for required data. The rules associated with the mapping and intended calculations from the data will also be reviewed to insure accuracy.

• The STAR METRICS sample code library (query) examples will be evaluated for use or modification.

• The STAR METRICS team help select pre-defined queries or assist in building custom queries as required.

• The University team will run the queries to extract the required data.

• The extracted data will be validated with the University team

Step 3 Create Export Report Tables

The third step in the process is to prepare the extracted data from the University systems for transmission to the STAR METRCS central repository.

• The STAR METRICS team will help build custom translate tables as required (examples include Occupational Classification translations)

• The STAR METRICS team will run a data cleansing routine to match federal award numbers and address any data anomalies with the University team.

• Final formatted export files will be created and validated by the University team

Step 4 Export Report Tables

The fourth step in the process is to transmit the University data to the secure STAR METRICS data repository.

• The STAR METRICS team will work with the University team to upload the files via a Secure FTP (file transfer protocol) or other accepted secure protocol data transfer.

• The University team will upload the files.

• The transmitted data will be validated post successful transmission. Checks will be provided to confirm the number rows of information transmitted, the number fields transmitted per row, the format and order of the fields.

After the submission of the University data to the STAR METRICS central repository, the STAR METRICS team will receive the transmitted data and perform a number of data cleansing, matching and processing steps in preparation for producing summarized university reports.

Step 5 Create and Publish University Reports

The final step is to process the information that has been loaded into the STAR METRICS database , produce and review the reports and publish to the University Web site .

• The STAR METRICS team will run the processes that will translate and compute new variables that include: Direct jobs calculated from Individuals on a project, from overheard, and from vendors and sub-awards recipients

• The STAR METRICS team will run jobs to create and format reports

• The STAR METRICS team will work with the University team to validate the created reports

• The STAR METRICS team will upload the reports to the University Web site

Appendix B: Calculation Jobs Created & Retained to Help Inform Grant Recipient Reporting to Recovery.gov: the STAR METRICS Approach

STAR METRICS was established in response to the guidance which also states federal agencies, in coordination with the Director of the Office of Management and Budget, shall provide for user-friendly means for recipients of covered funds to meet the requirements of this section. (Section G)

Direct jobs created and retained from science awards are calculated from four sources:

1. Individuals working directly on the project;

2. Employment within overhead

3. Employment of vendors

4. Employment of individuals on sub-awards

This maps to the data element: "Description of Jobs Created/Retained" which requires

1. A narrative description of the employment impact of the Recovery Act funded work. This narrative is for each calendar quarter and at a minimum, will address the impact on the recipient’s or federal contractor’s workforce (for grants and loans, recipients shall also include the impact on the workforces of sub recipients and vendors).

2. Provide a brief narrative description of the types of jobs created and jobs retained in the United States and outlying areas. This description may rely on job titles, broader labor categories, or the recipient’s existing practice for describing jobs as long as the terms used are widely understood and describe the general nature of the work.

Source: Recipient Reporting Data Model - for quarter ending 12/31/2009 (www.recovery.gov)

Prime recipients of grants, cooperative agreements, and loans must include an estimate of jobs created and retained on projects and activities managed by their funding recipients (i.e. sub-recipients) in the numeric and narrative data fields mentioned in 5.2.3 above. See Section 5.7 for further details. http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-08.pdf

Direct Jobs Calculated from Individuals Employed
Information Requested	Required for element calculation
De-identified Employee ID #	Parts A and B below
Federal Award ID #	Verification and data quality check
University Award ID #	Verification and data quality check
Occupational Classification	Part B below
Proportion of time allocated to award	Parts A and B below
FTE status	Parts A and B below
Calculation for Part A:
Calculation for Part B is the same as Part A

The approach is identical to the discussion in the Peter Orszag memo of Dec 18 2009 (attached) - although using proportion of earnings, rather than proportion of hours (which is often not captured in HR systems).

Direct Jobs Calculated from Overhead
Information Requested	Required for element calculation
Federal Award ID #	Verification and data quality check
University Award ID #	Verification and data quality check
Overhead charged	Calculation
Report to cognizant agency	Calculation

The overhead expenditures will be combined with the information provided in the report to cognizant agency² to generate an estimate of the salaries paid out of the grant. This will be converted to jobs by using County Business Patterns data for NAICS code 61. http://www.census.gov/econ/cbp/index.html

Direct Costs Calculated from the Employment of Vendors
Information Requested	Required for element calculation
Federal Award ID #	Validation
University Award ID #	Validation
Duns #	Calculation
Amount of Contract	Calculation

The Duns # will be used to derive an industry code and geographic location. The amount of contract revenues will be ratio-ed to generate employment estimates derived from Economic Census data

http://www.census.gov/econ/census07/

The guidance requires that grant recipients directly get that information from vendors. The results will help guide the assessment of the quality of the responses. As the project broadens, these calculations can be directly generated from the administrative records of the respondents themselves.

Direct Jobs Calculated from the Employment on Sub Awards
Information Requested	Required for element calculation
Federal Award ID #	Validation
University Award ID #	Validation
Duns #	Calculation
Amount of Contract	Calculation

The Duns # will be used to derive an industry code and geographic location. The amount of contract revenues will be ratioed to generate employment estimates derived from Economic Census data

http://www.census.gov/econ/census07/

As above, the Duns # will be used to derive an industry code and geographic location. The amount of contract revenues will be ratioed to generate employment estimates derived from Economic Census data

http://www.census.gov/econ/census07/

The guidance requires that grant recipients directly get that information from vendors. The results will help guide the assessment of the quality of the responses. As the project broadens, these calculations can be directly generated from the administrative records of the respondents themselves.

http://rates.psc.gov/fms/dca/shortform1.pdf

1 http://www.ostp.gov/galleries/press_release_files/Final%20Signed%20OMB-OSTP%20Memo%20-%20ST%20Priorities.pdf

2 http://rates.psc.gov/fms/dca/shortform1.pdf

4/11/2010