Supporting Statement A For:
Process and Outcomes Evaluation of
NCI Physical Sciences in Oncology Centers (PS-OC) Initiative (NCI)
June 12, 2015
Nicole M. Moore
Division of Cancer Biology
National Cancer Institute
9609 Medical Center Drive, Room 6W508
Bethesda, MD 20892-9714
Tel. 301-325-7534
Email. [email protected]
Table of Contents
A. Justification 1
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information Collection 3
A.3 Use of Improved Information Technology and Burden Reduction 6
A.4 Efforts to Identify Duplication and Use of Similar Information 6
A.5 Impact on Small Businesses or Other Small Entities 7
A.6 Consequences of Collecting the Information Less Frequently 7
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 7
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 7
A.9 Explanation of Any Payment or Gift to Respondents 8
A.10 Assurance of Confidentiality Provided to Respondents 8
A.11 Justification for Sensitive Questions 9
A.12 Estimates of Annualized Burden Hours And Costs 9
A.13 Estimates of Other Total Annual Cost Burden to Respondents and
Record Keepers 11
A.14 Annualized Cost to the Federal Government 11
A.15 Explanation for Program Changes or Adjustments 11
A.16 Plans for Tabulation and Publication and Project Time Schedule 12
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 12
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 12
List of Attachments
Attachment 1: Background Information about the Physical Sciences - Oncology Center (PS-OC) and Program Evaluation Plans
Attachment 2: Memo Summarizing Evaluation Plan
Attachment 3: Screenshots of Survey for Current Trainees
Attachment 4: Screenshots of Survey for Former Trainees
Attachment 5: Screenshots of Survey for NCI Grantees
Attachment 6: Instructions and Scoring Sheet for Expert Peer Review Panel
Attachment 7: Evaluation Advisory Committee
Attachment 8: National Institutes of Health Privacy Act Memo
Attachment 9: Office of Human Subjects Research Review
Attachment 10: Email Invitations
This request is to gain OMB approval for the new submission titled, “Process and Outcomes Evaluation of NCI Physical Sciences in Oncology Centers (PS-OC) Initiative (NCI)” for 1 year.
The NCI launched the Physical Sciences - Oncology Center (PS-OC; http://physics.cancer.gov/) program in 2009 as Phase I of the Physical Sciences in Oncology (PSO) Initiative. The PSO Initiative seeks to establish research projects that bring together cancer biologists and oncologists with scientists from the fields of physics, mathematics, chemistry, and engineering to address some of the major questions and barriers in cancer research. As part of this initiative, evaluation plans were developed and consisted of three components, dependent on which year the initiative is in: prospective for beginning, structured for mid-point, and summative/full outcome evaluation for a decade after the program started. In 2015 the PSO Initiative is transitioning from the beginning to a mid-point phase, which represents a critical time to reflect on the initial outcomes and restructure the process evaluation to account for changes mid-way through the initiative. This proposed request is to conduct on-line surveys with current and former trainees and NCI grantees associated with the program and comparable NCI programs. Additionally, an assessment of publications generated through the PS-OC program will be conducted via a virtual expert review panel. The evaluation will address trainee development and career path post program involvement as well as the impact of the program involvement on program outputs. Results from both the surveys and the expert peer reviewer panel will assess research innovation from the program and inform the future development of the PSO Initiative.
A.1 Circumstances Making the Collection of Information Necessary
The Public Health Service Act, Section 413 (42 USC § 285a-2) authorizes the Director of the National Cancer Institute (NCI) to support education and training programs that encourage coordination and collaboration amongst scientists. Specifically, the mission of the Division of Cancer Biology (DCB) of NCI is to ensure continuity and stability in basic cancer research while encouraging and facilitating the emergence of new ideas, concepts, technologies and possibilities. DCB strives to achieve this goal by promoting a balance between the continued support of existing research areas and selective support of emerging research areas. The scientific discoveries from this research base are critical to the goal of the NCI since they form the intellectual and scientific foundation on which strategies for prevention, diagnosis and treatment of cancer are developed. The expansion of new research areas is encouraged through a range of initiatives, such as the Physical Sciences in Oncology (PSO) Initiative (http://physics.cancer.gov/). The PSO Initiative seeks to establish research projects that bring together cancer biologists and oncologists with scientists from the fields of physics, mathematics, chemistry, and engineering to study key questions in cancer research from a physical sciences perspective. Additional background information about the PSO and evaluation plans is located in Attachment 1.
As part of the PSO Initiative, evaluation plans were developed in 2009 and divided into three components dependent upon the phase of the program (Attachment 2):
Phase I (Year 1-3) - Prospective evaluation. This phase is in its last stages and involves continuous extant data collection and analysis for in-the-moment programmatic improvement.
Phase II (Years 4-5) - Structured evaluation. An expert peer-review panel will assess program design, implementation and preliminary outcomes.
Phase III (Years 10+) – Summative/full outcome evaluation. This phase is conducted at least 10 years after the start of the program.
In 2015 the PSO Initiative is transitioning from Phase I to Phase II representing a critical time for the initiative to reflect on the outcomes of Phase I and restructure the process evaluation to account for changes in Phase II. The timeliness and scope of the proposed evaluation reflects recommendations of the Science and Technology Policy Institute’s (STPI) evaluation plan outlined above and general recommendations from NIH, Institute of Medicine, and National Research Council reports in regards to large center initiatives for biomedical sciences, which include:
An evaluation should be built into the entire life of the initiative, from design through initiative completion, and needs to be able to change and adapt in its focus as the initiative evolves.1
A set of metrics for assessing the technical and scientific output (such as data and research tools) of large-scale projects should be developed.2
The assessment should include tracking of any trainees involved in a project to determine the value of the training environment and the impact on career trajectories.2
The primary purpose of this request is to gain OMB approval to conduct the structured evaluation that involves conducting on-line surveys with trainees, current and former, and NCI grantees and an assessment of the program publications via an expert peer reviewer panel. The evaluation objectives include to:
Assess the extent to which the initiative has been successful in reaching its goals,
(2) Determine whether the program in Phase I was conducted as planned, and
(3) Identify strengths and weaknesses of the initiative structure for adjustment in Phase II.
A.2 Purpose and Use of the Information
Results from the survey questions (Attachments 3, 4, & 5) and expert reviewer panel (Attachment 6) will assess to what extent the goals of the PSO Initiative have been met by evaluating:
To what extent did the PS-OC program generate and accelerate innovative/ impactful trans-disciplinary solutions to outstanding questions in oncology relative to comparison groups?
To what extent did the PS-OC program accelerate the career development path of trainees relative to comparison groups in the field of physical sciences - oncology relative to comparison groups?
To what extent did trainees maintain a greater number of trans-disciplinary collaborations after leaving the PS-OC program compared to trainees of other programs?
What aspects of the program components or program involvement have further enhanced:
Connecting physical scientists and cancer researchers?
Connecting physical sciences with translational research? And
The generation of innovative and impactful cancer research?
Data for this evaluation will consist of a combination of archived data and new data compiled from the on-line surveys and the expert peer review panel. Archived data sources will include progress reports such as the Research Performance Progress Report (RPPR), scientific databases such as MEDLINE and Thomson Reuters Web of Science, and NIH administrative databases such as QVR and iEDISON. Bibliometrics, such as impact factor, citation count, citation benchmark, and bibliometric percentile will be collected and analyzed to complement data from surveys and expert peer review panel.
The surveys will target former and current trainees (N = 550) and NCI grantees (N=300 total) associated with the PS-OC program (60 grantees) and 5 other comparable NCI programs (240 grantees). The surveys will address (1) trainee development and career paths associated with the PS-OC program and (2) assess the impact of program involvement and activities on the PS-OC program and more broadly for NCI programs with U54 mechanisms. Additional information will be collected from an expert peer review panel asked to review and score publications generated by the PS-OC program and comparable NCI programs.
For the expert review panel, experts will be selected based on their scientific background in the field of physical sciences in oncology and will have no association with the PSO Initiative and comparison group program. Each expert panelists will be asked to review a packet consisting of 4 - 6 publications. Publications will be assigned to panelists such that two panelists will be reviewing each publication. The following data will be collected from the panelists regarding their assigned publications from the PS-OC program or comparison groups using an online scoring sheet (Attachment 6):
The impact of a grant's publications on cancer research, assessed individually, using a five-point scale from extremely to not at all,
The impact of a grant's set of publications on cancer research, assessed as a whole, on a five-point scale from extremely to not at all,
The innovativeness of the approaches of a grant's individual publications on a five-point scale from extremely to not at all, and
The innovativeness of a grant's publications, assessed as a whole, on a five-point scale from extremely to not at all.
Primarily, the PS-OC program officials will use the findings from the program evaluation to guide the PSO initiative in Phase II and inform program activities. Also, information gained is expected to guide future critical funding decisions of PSO centers and projects through program announcement review (PAR) mechanisms for which applications will be solicited over the next two years and a potential reissuance of the PARs in FY 2016.
In addition, information from our program evaluation will be disseminated to other NCI program staff and other ICs to help support several other team science based programs that may benefit from findings of this study. Evaluation findings are expected to provide strong insights on programmatic structure, management, and activities that support productive team science initiatives leading to changes in current program activities or future program structure development. This will complement existing studies of team science based initiatives supported through the NCI Division of Cancer Control and Population Sciences (DCCPS), including the Team Science Toolkit.3
In terms of trans-NIH relevance, there has been an increasing interest at NIH in supporting team science based initiatives using complex mechanisms and cooperative agreements. Data collected from the PSO initiative evaluation will be disseminated to other NIH Institutes through presentations, publications, and the Team Science Toolkit. This data is expected to represent a series of baseline metrics for team science based initiatives and evaluations that goes beyond publication output.
Web-based surveys will be employed to reduce the burden to the respondent. The trainee and NCI grantee surveys will be conducted via Survey Gizmo that will provide a simple interface for respondents to answer questions (Attachments 3, 4, & 5). A web-based scoring sheet will be used for expert peer review panelist that has been pre-populated with the titles associated with the publications they will be reviewing (Attachment 6). All data will be compiled into a searchable file for quick review and analysis of results. The NCI Privacy Act Coordinator was consulted and determination of a Privacy Impact Assessment (PIA) is underway.
To date, NCI has conducted a prospective evaluation of the PSO initiative as proposed in the evaluations strategy developed in 2009 (Attachment 2). The prospective evaluation collected data from the progress reports, including publications, patents, collaborations, and leverage funding. Additionally bibliometric analysis and collaboration analysis were performed on outputs collected from the progress reports using external databases and algorithms.4 A web-based survey of trainees and investigators participating in the PS-OC program was conducted at the 3-year point to assess satisfaction and participation of participants in the program at the mid-point5. The data collected in the prospective evaluation was successful in providing information to program officials on the status of the program at the mid-point to promote needed adjustments. The proposed evaluation plan defined above will conduct a structured outcome evaluation. This evaluation differs from the prospective evaluation in that is will assess the program as a whole and compare it to other comparable program and initiatives at NIH. This will be the first structured evaluation completed on the PS-OC program and the PSO initiative. Data collected from the surveys and expert review panel is not available through other resources and databases.
A.5 Impact on Small Businesses or Other Small Entities
No small entities will be involved in this survey. All respondents will be individuals who participate voluntarily.
This is a one-time collection.
This study complies fully with the guidelines of 5 CFR 1320.5. No exceptions to the guidelines are required.
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on March 23, 2015 Vol. 80, P. 15228. No comments were received.
This evaluation was developed through NCI’s collaboration with the Science and Technology Policy Institute. Additionally, an evaluation advisory committee has been established for the proposed evaluation. Committee members include PS-OC program staff, individuals with evaluation experience, and program officials/scientists outside of the PS-OC program (Attachment 7). The committee responsibilities will include review of evaluation outputs and provide expert guidance as needed. The committee will meet a minimum of twice during the evaluation process.
The Privacy Act also provides privacy for the treatment of data maintained by a Federal agency according to either the individual’s name or some other identifier. This information collection is covered by NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” (Attachment 8). In accordance with the Privacy Act of 1974, the privacy of individual respondents will be protected. The data sets created will contain no means of identifying individual respondents. The following describes the measures taken to protect the privacy of the participants. Additionally, this project has been reviewed by the Office of Human Subjects Research Board and deemed that Federal regulations for the protection of human subjects do not apply to this project (Attachment 9).
Respondents will have the option to skip any question they would prefer not to answer and to quit the survey at any time. Unless express permission is provided by the respondent, all data will be de-identified and reported in the aggregate. Respondents will not be asked to complete a consent form. Respondents will be invited via email (Attachment 10) to complete the survey or participate in the expert review panel. Each respondent’s willingness to initiate the survey or scoring sheet via a link in the invitation letter will be interpreted as evidence of implied consent. Respondents are informed in the letter that any information they provide will be private, to the extent provided by law.
Security protocols will be implemented to ensure that all data are recorded and stored in such a manner that individual research subjects cannot be identified directly or through identifiers. Each scoring sheet will include a unique ID number for each respondent, but only the data management contractor will have the secure database to link ID numbers with individuals. No identifying information will be recorded in the data file and there will be no way to detect the identification of any respondent. Electronic data will be password protected and stored by the data management contractor, and also will be destroyed after a year. All data will be kept private to the extent of the law.
A.11 Justification for Sensitive Questions
Though personally identifiable information (PII) is known in the form of their name prior to the invitation, the questions being asked do not constitute sensitive questions. There are no sensitive questions.
The total estimate of respondent burden is 955 hours over the one-year request for approval. The burden estimate is based on four instruments, three 25-minute web-based surveys and an eight-hour peer review scoring sheet. The surveys will be completed by 210 current trainees (Attachment 3), 340 former trainees (Attachment 4), and 300 grantees (Attachment 5). The peer review scoring sheet will be completed by 75 expert peer review panelists (Attachment 6). (Table A.12-1).
Table A12-1. Estimate of respondent hour burden
Instrument |
Type of Respondent |
Number of Respondents |
Number of Responses Per Respondent |
Average Burden Per Response (in hours) |
Total Annual Burden Hour |
Survey (Attach 3) |
Current NCI Trainees |
210 |
1 |
25/60 |
88 |
Survey (Attach 4) |
Former NCI Trainees |
340 |
1 |
25/60 |
142 |
Survey (Attach 5) |
NCI Grantees |
300 |
1 |
25/60 |
125 |
Scoring Sheet (Attach 6) |
Expert Reviewers |
75 |
1 |
8 |
600 |
Total |
|
925 |
|
|
955 |
The wage rate was calculated from the Bureau of Labor Statistics, (http://www.bls.gov/oes/current/oes_nat.htm) using the median hourly wage. The total and annualized cost is calculated with a wage rate of $21.25 per hour for NCI trainees (Life, Physical and Social Science Technician - #19-4099) and $32.18 per hour for NCI grantees and expert reviewers (Life Scientist - #19-1099) and is estimated to be $25,094 (Table A.12-2).
Type of Respondent |
Total Annual Burden |
Hourly Wage Rate |
Total Respondent cost |
Current NCI Trainees |
88 |
$21.25 |
$1,870.00 |
Former NCI Trainees |
142 |
$21.25 |
$3,017.50 |
NCI Grantees |
125 |
$32.18 |
$4,022.50 |
Expert Reviewers |
600 |
$32.18 |
$19,308.00 |
Total |
|
|
$28,218.00 |
There are no costs to respondents . There are no operating, maintenance or capital costs associated with the collection.
The annual cost to the federal government is $76,533 (Table A.14-1). The annual contractor costs for this evaluation are $60,000. These costs include study design and analysis plan and instruments, data collection, data cleaning, summary of results and final report. The annual costs of Federal employees for overseeing and coordinating this evaluation are estimated to be $16,532.50. These costs are based on 10% of the Health Scientist Administrator’s time and 10% of a Program Analyst’s time, using the Federal General Schedule Salary Table for 2015. The total and annualized costs for this one-year request are outlined below.
Table 14-1 Annual cost to the government
NCI Personnel |
Grade/Step |
Annual Salary |
Percent time |
Total Cost |
Health Scientist Administrator |
14/1 |
$107,325.00 |
10% |
$10,732.50 |
Program Analyst |
10/1 |
$58,000.00 |
10% |
$5,800.00 |
Total government personnel costs |
$16,532.50 |
|||
Contractor costs |
$60,000.00 |
|||
Annual cost to the government |
$76,532.50 |
|||
This is new information collection.
All quantitative data will be standardized and compiled into Excel spreadsheets. Internal validity will be checked as necessary for analysis. Contractors with established experience and procedures in place to handle these issues will be hired. Similarly, qualitative data will be compiled into word documents and validated. Data analysis is highly dependent on the evaluation question. There will be both quantitative and qualitative analysis performed. For any quantitative data, descriptive and summary statistics (mean, median, mode) will be analyzed. For comparative analysis, t-tests will be used to compares mean or frequencies of quantitative data. For some evaluation questions, data from multiple sources (quantitative and qualitative) may be cross-referenced to address the question.
Project Time Schedule
The project time schedule is outlined in Table A.16-1.
Table A.16-1. Project Timeline.
Task |
After OMB Approval |
Identify and invite expert reviewers, trainees and grantees. |
1-3 Months |
Conduct Expert Panel Review and web-based survey. |
3-4 Months |
Summarize results from expert reviews and survey. |
4-5 Months |
Develop and finalize written report. |
5-8 Months |
NCI is not seeking an exemption from displaying the OMB expiration date.
NCI is not requesting an exception to the certification requirements.
1 Trochim W et al. The evaluation of large research initiatives - A participatory integrative mixed-methods approach. American Journal of Evaluation. 2008; 29, 1:8-28.
2 National Research Council. Large-Scale Biomedical Science: Exploring Strategies for Future Research. Washington, DC: The National Academies Press, 2003
3 Hall, K et al. Assessing the Value of Team Science: A Study Comparing Center- and Investigator-Initiated Grants. American Journal of Preventive Medicine. 2012 Feb; 42(2): 157-163.
4 J.E. Basner et al.. "Measuring the Evolution and Output of Cross-Disciplinary Collaborations within the NCI Physical Sciences–Oncology Centers Network" Journal of Research Evaluation. 2013. Dec; 22 (5): 285-297.
5 The survey received Office of Management and Budget approval on March 7, 2012 under the title of, “PS-OC Survey” (OMB No. 0925-0642-07).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PLCO SCREENING TRIAL |
| Author | Linda Suit |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |