support_stmt_a_Extension_v1 (2)

Supporting Statement A for Extension of

STAR METRICS

Science and Technology for America’s Reinvestment: Measuring the EffecTs of Research on Innovation, Competitiveness and Science

Date: 28, February, 2012

Name: George Chacko

Address: Center for Scientific Review, National Institutes of Health

6701 Rockledge Dr, Rm 3030

Bethesda, MD 20892

Telephone: 301 435 1111

Fax: 301 480 3965

Email: [email protected]

Table of contents

A. JUSTIFICATION

A.1 Circumstances Making the Collection of Information Necessary

A.2. Purpose and Use of the Information COLLECTION

A.3 Use of Information Technology and Burden Reduction

A.4 Efforts to Identify Duplication and Use of Similar Information

A.5 Impact on Small Businesses or Other Small Entities

A.6 Consequences of Collecting the Information Less Frequently

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A.9 Explanation of Any Payment of Gift to Respondents

A.10 Assurance of Confidentiality Provided to Respondents

A.11 Justification for Sensitive Questions

A.12 Estimates of Hour Burden Including Annualized Hourly Costs

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record

keepers

A.14 Annualized Cost to the Federal Government

A.15 Explanation for Program Changes or Adjustments

A.16 Plans for Tabulation and Publication and Project Time Schedule

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

List of Attachments:

Attachment 1: Memorandum of Understanding

Attachment 2: First Addendum to the Memorandum of Understanding

Attachment 3: Recipient Reporting Data Model V3.0

Attachment 4: Updated Guidance on the American Recovery and Reinvestment Act – Data Quality, Non-Reporting Recipients, and Reporting of Job Estimates The American Recovery and Reinvestment Act of 2009, P.L. 111. Memo M-10-08

Attachment 5: Background calculations

Attachment 6: Star Metrics Pilot Project Guide

Attachment 7: Star Metrics website screenshot

A.1 Circumstances Making the Collection of Information Necessary

This is a extension of the previous STAR METRICS data collection, OMB number 0925-0616.

Authority: The NIH is acting pursuant to Sections 301, 402 and 405 of the Public Health Service Act (42 U.S.C. §§ 241, 281 and 284.) See Attach 1 (Memorandum of Understanding)

Background: NIH, in conjunction with NSF, DOE, EPA and OSTP, has engaged in a partnership with universities entitled STAR METRICS¹. STAR METRICS is a multi-year project being created in direct response to the reporting requirements of the ARRA, and aims to provide American taxpayers with accurate information on the value of their investments.

Rationale for Federal Government involvement: The Office of Science and Technology Policy, through the NSTC Committee on Science, established the Science of Science Policy (SoSP) Interagency Working Group to develop an evidence-based framework for informing policy investments in research and development, and assess the impacts of those investments broadly. Themes and goals for this emergent research field were outlined in a Federal Research Roadmap^2, and this effort was also highlighted in the FY 2011 Budget Priorities Memorandum from OSTP and the Office of Management and Budget (OMB). The memorandum asks Federal agencies to “develop outcome-oriented goals for their science and technology activities, establish procedures and timelines for evaluating the performance of these activities, and target investments toward high-performing programs. Agencies should develop ‘science of science policy’ tools that can improve management of their research and development portfolios and better assess the impact of their science and technology investments. Sound science should inform policy decisions, and agencies should invest in relevant science and technology as appropriate.”³

The initial goal of STAR METRICS is to provide mechanisms that furnish participating universities and federal agencies with a reliable and consistent means to account for the number of scientists and staff that are on research institution payrolls, supported by federal funds.

Section 1512 of ARRA⁴ requires recipients to submit reports that contain the following

RECIPIENT REPORTS.—Not later than 10 days after the end of each calendar quarter, each recipient that received recovery funds from a Federal agency shall submit a report to that agency that contains—

(1) the total amount of recovery funds received from that agency;

(2) the amount of recovery funds received that were expended or obligated to projects or activities; and

(3) a detailed list of all projects or activities for which recovery funds were expended or obligated, including—

(A) the name of the project or activity;

(B) a description of the project or activity;

(C) an evaluation of the completion status of the project or activity;

(D) an estimate of the number of jobs created and the number of jobs retained by the project or activity; and

(E) for infrastructure investments made by State and local governments, the purpose, total cost, and rationale of the agency for funding the infrastructure investment with funds made available under this Act, and name of the person to contact at the agency if there are concerns with the infrastructure investment.

(4) Detailed information on any subcontracts or subgrants awarded by the recipient to include the data elements required to comply with the Federal Funding Accountability and Transparency Act of 2006 (Public Law 109–282), allowing aggregate reporting on awards below $25,000 or to individuals, as prescribed by the Director of the Office of Management and Budget.

STAR METRICS was established in response to the guidance which also states federal agencies, in coordination with the Director of the Office of Management and Budget, shall provide for user-friendly means for recipients of covered funds to meet the requirements of this section. (Section G)

Past data collections: The STAR METRICS team worked with seven universities participating in the Federal Demonstration Partnership (FDP)⁵ to identify all individuals supported by federal science funding in order to generate information on jobs created under ARRA. The consensus by the seven universities is that it is not difficult to create the required files, and there is substantial value added from standardized reports (more burden information is available in subsequent sections). The results from the pilot were presented at the FDP national meetings Sept 21-22 and January 25-26.

The STAR METRICS team has briefed the Office Ed Deseve and Frank Giammarino on August 31, and briefed them on the STAR pilot project.

A.2 Purpose and Use of the Information Collection

The aim of STAR METRICS is twofold. The initial goal of STAR METRICS is to provide mechanisms that will allow participating universities and federal agencies with a reliable and consistent means to account for the number of scientists and staff that are on research institution payrolls, supported by federal funds. In subsequent generations of the program, it is hoped that STAR METRICS will allow for measurement of science impact on economic outcomes (such as job creation), on knowledge generation (such as citations and patents) as well as on social and health outcomes.

Since the approval of the data collection for STAR METRICS in 2010, the program reached a total number of 80 participating research institutions, testifying its tremendous success and the perception of its usefulness by the academic institutions. There have also been two additional Federal agencies, the Department of Energy (DOE) and the Environmental Protection Agency (EPA) that have joined OSTP, NIH and NSF in the STAR METRICS consortium. The goal is now to expand the program to at least additional 20 research institutions and one Federal agency, to reach a total of 100 participating institutions.

The data elements required from the academic institutions to participate in the pilot are as follows:

	Data Element	Definition
Information on Scientists and Awards	For each Award
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University DUNS Number	The organization’s 9-digit Data DUNS number
	University Award Id	University's internal number for the award.
	Overhead Charged	The overhead amount charged in the reporting period.
	For each individual
	Anonymized Employee Id	Unique Employee ID (not Social Security number) of grant funded personnel
	Occupational Classification	Occupational classification / Job description of the funded personnel (ex. Faculty, Undergrad Student, Grad Student, Admin, Technical Support, Post grad Student)
	FTE Status	Designation of the status of the funded personnel (full time = 1.0, half time = .5)
	Proportion of time allocated to award	Calculated portion of the time expended by the funded personnel during the reporting time period.
Information on Overhead	Report to cognizant agency	Report submitted to cognizant agency that supports the recipients recovery of administrative overhead costs through an indirect cost rate.
Payments to vendors	For each payment to a vendor
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University Award Id	University's internal number for the award.
	Vendor DUNS Number	Vendor DUNS Number. The Vendor's 9 digit DUNS number
	Payment Amount	The amount invoiced to the vendor in the reporting period
Subcontracts and subawards	For each Subaward
	Federal Award Number	The identifying number assigned by the awarding Federal Agency, such as the federal grant number, federal contract number or the federal loan number.
	University Award Id	University's internal number for the award.
	Subaward recipient DUNS Number	The sub recipient organization’s 9- digit DUNS number
	Total Subaward Funds Disbursed	The amount of cash disbursed to the sub-awardee in the reporting period.

The use of the data elements: Direct jobs created and retained from science awards are calculated from four sources:

1. Individuals working directly on the project;

2. Employment within overhead

3. Employment of vendors

4. Employment of individuals on sub-awards

The calculations are provided in Attachment 4.

Use of the data The administration has committed to producing an open, transparent and auditable set of reports of the impact of ARRA on job creation. In addition, as noted in Figure 2, the media has asked important questions about the impact of federal investments in science, particularly with respect to job creation and economic growth. It is important to collect and analyze data so that such questions can be answered in a credible fashion. There is currently no data infrastructure that systematically couples science funding with outcomes.

Figure 2: Articles in Science and Nature on science job creation

Use by Universities

The STAR METRICS team will gather the data from the Universities and transmit it to the STAR METRICS data repository.

The first step in the STAR METRICS data gathering process is to set up a meeting with the designated members of the university implementation team. The purpose of the meeting is to provide an understanding of the project and the information that will need to be collected with the. This first meeting may be done via webinar, teleconference, or on site and will cover the following topics:

• The STAR Metrics team will review details of the STAR Project, discuss the information that will be collected, the process for collection and the reports that will be generated from the data.

• The STAR METRICS Data Dictionary will be made available to the university participants prior to the meeting so that questions may be identified ahead of time.

• Following the meeting, the Star Metrics team will assist the university implementation team in developing a map of the requested fields to the target sources at the university.

The second step of the process involves running the data extract routines.

• Using the data sheet as input, the STAR METRICS team will review the mapping of the field level names against the sources for required data. The rules associated with the mapping and intended calculations from the data will also be reviewed to insure accuracy.

• The STAR METRICS sample code library (query) examples will be evaluated for use or modification.

• The STAR METRICS team helps select pre-defined queries or assist in building custom queries as required.

• The University team will run the queries to extract the required data.

• The extracted data will be validated with the University team

The third step in the process is to prepare the extracted data from the University systems for transmission to the STAR METRCS central repository.

• The STAR METRICS team will help build custom translate tables as required (examples include Occupational Classification translations)

• The STAR METRICS team will run a data cleansing routine to match federal award numbers and address any data anomalies with the University team.

• Final formatted export files will be created and validated by the University team

The fourth step in the process is to transmit the University data to the secure STAR METRICS data repository.

• The STAR METRICS team will work with the University team to upload the files via a Secure FTP (file transfer protocol) or other accepted secure protocol data transfer.

• The University team will upload the files.

• The transmitted data will be validated post successful transmission. Checks will be provided to confirm the number rows of information transmitted, the number fields transmitted per row, the format and order of the fields.

After the submission of the University data to the STAR METRICS central repository, the STAR METRICS team will receive the transmitted data and perform a number of data cleansing, matching and processing steps in preparation for producing summarized university reports.

The final step is to process the information that has been loaded into the STAR METRICS database, produce and review the reports and publish to the University Web site .

• The STAR METRICS team will run the processes that will translate and compute new variables that include: Direct jobs calculated from Individuals on a project, from overheard, and from vendors and sub-awards recipients

• The STAR METRICS team will run jobs to create and format reports

• The STAR METRICS team will work with the University team to validate the created reports

• The STAR METRICS team will upload the reports to the University Web site

Use by NIH, OSTP, NSF, DOE and EPA

STAR METRICS creates a reliable and consistent inter-agency mechanism to account for the number of scientists, students and support staff that are on research organization payrolls and supported by federal funds. It will develop standardized measures of the impact of science investments on job creation stemming from grants, contracts, intramural research efforts, and other job

The continuation and expansion of STAR METRICS requires the creation of a broadly based, multipurpose, data infrastructure. The resulting information will be made available to agencies and institutions to enhance their management information systems.

A.3 Use of Information Technology and Burden Reduction

The current approach to estimating job creation is extremely labor intensive and burdensome.

Figure 3 provides a stylized description of how administrative data can be used to identify, on a monthly basis, both the universe of individuals funded by federal science agencies (PIs, coPIs, graduate and undergraduate students, lab technicians, science administrators, etc.) and the amount of federal expenditure on the individuals.

Figure 3: Using the university financial system to generate microdata

In order for the data elements to be generated, the STAR METRICS team will need to make minor modifications of the major software packages used by the different institutions.

A Privacy Impact Assessment has been undertaken. NIH legal counsel has determined that no PII are being collected

A.4 Efforts to Identify Duplication and Use of Similar Information

The STAR METRICS team has presented the proposed approach to all relevant national organizations (AAU, APLU, AAMC, FDP, NAS, NCURA, AUTM) and all science agencies represented on the Science of Science Policy interagency working group. No similar system exists.

A.5 Impact on Small Businesses or Other Small Entities

The participant institutions are all universities, and are not small businesses.

A.6 Consequences of Collecting the Information Less Frequently

ARRA requires quarterly reporting, so it is not feasible to collect the information less frequently.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

No special circumstances are anticipated. All the Guidelines of 5 CFR 1320.5 are met and the project fully complies

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

The 60 day notice was published on [Oct 5, 2011, VOLUME 76, PAGE 61714]. One comment was received. The Science of Science Policy Interagency Group has been consulted in general. Detailed consultations have been held with the National Science Foundation and with the Office of Science and Technology Policy.

A.9 Explanation of Any Payment of Gift to Respondents

Participants responding to this study will not receive remuneration for their participation.

A.10 Assurance of Confidentiality Provided to Respondents

The NIH Privacy Information Act Officer has determined that the data collected are not PII.

A.11 Justification for Sensitive Questions

No questions are considered to contain sensitive information.

A.12 Estimates of Hour Burden Including Annualized Hourly Costs

At its full operating capacity, Level I of the STAR METRICS program will include approximately 100 universities. At the time of this extension we have estimates of burden, based on the information gathered from the first 80 participating universities. Results are reflected in the tables below. The current ARRA reporting burden depends on the number of ARRA grants. The estimates of current reporting burden of job creation varies substantially depending on the approach taken. University of Texas, Austin, estimates that the federal reporting burden is about 2 ¼ hours per award for 165 awards; the state reporting burden is even higher. About 25% of this is making job creation estimates. George Mason estimates that for 26 awards, the burden is 96 hours total per quarter. 25 hours of this is for job creation estimates.

T

We only have on half-time person dedicated to ARRA reporting but collectively we spent approximately 370 hours on this quarter’s reporting, and that is conservative. This doesn’t include all the additional hours for reporting to our State since they make us report weekly or the Work Study reporting that was handled by Student Financial Aid but it does reflect reporting for State Stabilization Funds through the state of Texas, which is handled by our Budget Office. This is an average of 2.25 hours/ award. This does not include the time I have spent and will spend this week answering questions from our sponsors and State Comptrollers Office. As a comparison, we spent approximately 15 hours developing the requirements for STAR reporting to extract the data from our homegrown financial system for the pilot, and unless significant changes are made in the future extracts, our IT person estimates it will take one hour to pull the data for each report. Yes, one hour.

Email from Suzi Sedwick, UT Austin, April 19, 2010.

here are two stages of data collection for Level I, as described in Table A.12-1. There is a one-time data input, which requires a meeting, followed by technical meetings with IT staff to set up the collection system and guiding the first data collection. The meetings will be with groups of between three to four participants. The process showed that it takes a median time of 45 hours for stage 1. Stage 2 is subsequent to Stage 1. In stage 2, for each quarter, there are regular submissions, which require 2.5 hours per quarter, as indicated in table A.12-1.

The current process for ARRA reporting (in the absence of STAR METRICS) is manual, and hence duplicated for each award in a linear fashion the reduction in burden will depend on the number of awards a university has. For the purposes of this document, we have used an informed estimate of a linear ¼ * 2.25 hours per award on the manual process, and a 2.5 hour time cost for the automatic, as shown in Figure 5.

The estimates of burden for the STAR METRICS system are illustrated in Tables A12.1 and A12.2. We currently already have 80 universities in the STAR METYRICS system, and expect to enroll 20 more in the next three years, or about 7 per year, as the tables below show.

We estimate that STAR METRICS contributes to a significant burden reduction when compared to the manual reporting system. The regular reporting burden, based on a median university time is of 40 hours. Therefore in the absence of STAR METRICS, 100 universities, responding four times per year, and employing, on average, 40 hours per response, would use 16,000 hours to satisfy the reporting request. Instead, the one-time burden to setup the Star Metrics system for 100 universities is of 4,500 hours, with a net saving of 11,500 hours for the first time reports. In the absence of STAR METRICS the burden for each submission remains the same at 16,000 hours per year. However, once the STAR METRICS system is setup, the burden drops to 2.5 hours for all subsequent responses. Therefore, 100 universities using the STAR METRICS will see a total burden of 250 hours per year, compared to the unchanged situation without STAR METRICS remains at 16,000 hours, achieving therefore a net reduction of 15,750 hours per year. Because of this, we estimate that, for each year, by using STAR METRICS technology universities save about $780,000.

Figure 5: Burden reduction as function of number of awards

A.12 - 1 Estimates Annual Burden Hours
Form	Number of Respondents	Frequency of Response	Average Time per Response (in hours)	Annual Hour Burden
Stage 1: One time data input	7	1	45	315
Stage 2: Ongoing quarterly data input	100	4	2.5	1000
TOTAL				1,315

A.12 - 2 Estimates Annual Burden Hours
Form	Number of Respondents	Frequency of Response	Average Time per Response (in hours)	Hourly Wage Rate	Respondent Cost
Stage 1: One time data input	7	1	45	50	15,750
Stage 2: Ongoing quarterly data input	100	4	2.5	50	50,000
TOTAL					65,750

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no capital costs, operating costs or maintenance costs to report.

A.14 Annualized Cost to the Federal Government

The annualized costs to the Federal Government are expected to be $1.5 million/year

A.15 Explanation for Program Changes or Adjustments

This is an extension of OMB number 0925-0616 current. An adjustment of three additional institutions was made to Stage I. The previous burden estimate involved four institutions (George Mason University, California Institute of Technology, University of Texas-Austin, and University of Massachusetts Dartmouth). In addition, Harvard University, The University of Pennsylvania, and The University of California San Francisco were added). This increases the total annual hour burden from 180 to 315 hours, an increase of 135 hours. An adjustment has also been made to Stage II where, following discussions with participating institutions, the frequency of response has been set to 4 to correspond with quarterly submissions typical of financial reporting.

A.16 Plans for Tabulation and Publication and Project Time Schedule

As soon as we receive OMB approval (Month 1) we will engage universities. We will begin the implementation of dropin code and quality checks in Month 2, and we expect to generate reports by month 4. The reports will be produced on a quarterly basis, to coincide with recovery.gov reporting.

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

Forms, surveys and databases will display the OMB expiration date.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

No exemptions to the certification of the PRA submissions are requested

1 Science and Technology in America’s Reinvestment – Measuring the Effects of Research on Innovation, Competitiveness and Science

2 The Science of Science Policy: a Federal Research Roadmap, http://www.ostp.gov/galleries/NSTC%20Reports/39924_PDF%20Proof.pdf

3 M-09-27 Memorandum for the Heads of Executive Departments and Agencies, Science and Technology Priorities for the FY 2011 Budget, August 4, 2009.

4 http://206.151.87.67/docs/Section1512ARRA.pdf

5 The Federal Demonstration Partnership (FDP) is an association of federal agencies, academic research institutions with administrative, faculty and technical representation, and research policy organizations that work to streamline the administration of federally sponsored research. FDP members of all sectors cooperate in identifying, testing, and implementing new, more effective ways of managing the more than $15 Billion in federal research grants. The goal of improving the productivity of research without compromising its stewardship has benefits for the entire nation.

11