Supporting Statement B For:
Process and Outcomes Evaluation of
NCI Physical Sciences in Oncology Centers (PS-OC) Initiative (NCI)
February 11, 2015
Nicole M. Moore
Division of Cancer Biology
National Cancer Institute
9609 Medical Center Drive, Room 6W508
Bethesda, MD 20892-9714
Tel. 301-325-7534
Email. [email protected]
Table of Contents
B. Collection of Information employing statistical methods 1
B.1 Respondent Universe and Sampling Methods 1
B.2. Procedures for the Collection of Information 2
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4
B.4 Test of Procedures or Methods to be Undertaken 5
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 6
List of Attachments
Attachment 1: Background Information about the Physical Sciences - Oncology Center (PS-OC) and Program Evaluation Plans
Attachment 2: Memo Summarizing Evaluation Plan
Attachment 3: Screenshots of Survey for Current Trainees
Attachment 4: Screenshots of Survey for Former Trainees
Attachment 5: Screenshots of Survey for NCI Grantees
Attachment 6: Instructions and Scoring Sheet for Expert Peer Review Panel
Attachment 7: Evaluation Advisory Committee
Attachment 8: National Institutes of Health Privacy Act Memo
Attachment 9: Institutional Review Board Approval
Attachment 10: Email Invitations
This evaluation will target different groups through web-based surveys and an expert peer-review panel scoring sheet:
Trainees (current and former) of the National Cancer Institute (NCI) Physical Sciences in Oncology (PSO) initiative (n=550),
Grantees of the NCI PSO initiative (n=60),
Grantees from five other NCI-funded programs that are similar in scope and mechanism to the PSO initiative (n=240), and
An expert peer review panel (n=75).
A majority of survey respondents will be affiliated with the PSO initiative. The PSO initiative is composed of 12 U54 cooperative agreement centers supported by NCI. Approximately 550 trainees and 60 primary grantees have been supported by these centers over the past 5 years. These trainees and grantees reside in over 110 different academic and research institutions across the United States. For this study, the entire pool of PS-OC primary trainees and grantees will be surveyed.
An additional group of NCI grantees will be surveyed that are supported through NCI programs and initiatives that are comparable in scope and mechanism to the PSO initiative. This group will provide additional information on the role of programmatic involvement on program goals more broadly across NCI and outside of the PSO initiative. Five comparable NCI programs were selected based on similarity to the structure and goals of PS-OC program and an initiation date before 2009. All primary grantees associated with grants or subprojects of these programs (240 grantees) were selected. Hence, a total of 300 grantees will be asked to complete the NCI grantee survey, 60 grantees from the PSO initiative and 240 grantees from other NCI programs.
Additional information will be provided by a panel of expert peer-reviewers, consisting of scientists with training backgrounds in the physical sciences and oncology (Attachment 6). The number of expert peer-reviewers was estimated based on the methodology used to evaluate the NIH Director's Pioneer Award program and to minimize burden per individual. NCI will provide a list of 400 preferred participants identified based on the research topic areas of their grant applications, participation in NCI activities, and interest in the area of physics in oncology. A subset of 85 potential expert peer-review panelists will be selected jointly by the Contractor based on their ability to review selected publications identified by program officials of the PSO initiative and comparable programs. The final expert peer-review panel will consist of 75 experts in the field, external to the PSO initiative and comparison groups. The remaining 10 will serve as a reserve pool of participants. In the event less than 75 potential expert peer-review panelists respond the reserve pool will be used to reach the target response rate of 75 panelists. The number of panelists was based on the goal to have each panelist review 4 - 6 publications from the PSO initiative and comparable programs. This will ensure that each publication is read and assessed by 2 different reviewers.
The Contractor shall plan and carry out surveys and expert review panel using web-based tools. NCI will invite the trainees, grantees, or expert review panelists to participate in the survey or panel review by email (Attachment 10). The Contractor shall follow up with an email containing a link to the survey or review panel scoring sheet, reminders, instructions, and additional required information, as needed.
The surveys will target trainees (N = 550; Attachments 3 and 4) and grantees (N=300; Attachment 5) associated with the PS-OC program and comparable NCI programs. The surveys will address trainee development and career path post PS-OC program involvement as well as the impact of program involvement on program outputs. The survey will be designed based on previous surveys conducted at NCI at the mid-point of the PSO initiative phase I. After survey respondents meet an 80-85% completion rate, the contractor will prepare summaries of survey results that will include descriptive statistics.
For the expert review panel, a one page introductory letter that will serve as an invitation (Attachment 10, last page) explaining the review process, the panelist’s responsibilities, the role of the contractor, and inviting the panelist to participate. NCI will distribute the introductory letter to the selected panelists via email. The Contractor shall follow up with participants by sending reminders, instructions, materials, a link to the review panel scoring sheet, and additional required information, as needed.
The expert review panel will be asked to review 2 packets consisting of 2-3 representative publications from 12 Physical Sciences - Oncology Centers (PS-OCs), 12 Integrative Cancer Biology Program (ICBP) Centers or comparison R01 groups (4-6 publications total per panelist). The review panel was designed so each publication packet will be reviewed by 2 panelists. The number of publications chosen are based on the previous study of the NIH Director's Pioneer Award1 program in 2012. The representative publication packets will be identified by PSO initiative researchers and comparison group researchers or program officials. The grant mechanism of the publications will be blinded before the review, although the authors and journal titles will not. The panel will conduct reviews individually via a web-based scoring sheet for each packet so as not to impart bias on other panelists. The following data will be collected from the panelists:
The impact of a grant's publications on cancer research, assessed individually, using a five-point scale from extremely to not at all,
The impact of a grant's set of publications on cancer research, assessed as a whole, on a five-point scale from extremely to not at all,
The innovativeness of the approaches of a grant's individual publications on a five-point scale from extremely to not at all, and
The innovativeness of a grant's publications, assessed as a whole, on a five-point scale from extremely to not at all.
Following completion of the expert panel review the contractor will generate a report summarizing the finding of the panel. A descriptive and comparative analysis will be performed on the data collected from the panel.
To maximize response rates, respondents will be informed prior to the evaluation by email and up to 3 follow-up attempts to contact non-responders will be made. Follow-up communications with survey respondents will be performed by the contractor. Response rates will be measured and recorded and once surveys have been completed, a non-response analysis will be conducted, and based on the result the survey data will be weighted to adjust for non-response bias.
A comparison evaluation of the NIH Director’s Pioneer Award (NDPA) program was published in 2012 addressing two primary study questions that followed from the program goals stated in the NDPA solicitations: To what extent does the research supported by the NDPA program produce unusually high impact science, and to what extent are the research approaches used by the NDPA grantees highly innovative? To assess these study questions a combination of bibliometrics and expert peer-review were performed.2 The methodology for conducting a expert peer-review panel and identifying comparison groups for evaluating innovation and impact performed as part of this evaluation were used and shown to be successful in the NDPA study.
A web-based survey of PSO initiative trainees and investigators was performed in 2012 at the mid-point of phase I3. This survey met its objectives of assessing program participation and satisfaction after three years and led change in program infrastructure and center participation. The type of survey identified for the proposed evaluation was selected based on the success of the previous survey.
The current survey and expert review panel have been modified from previous studies as guided by program officials, the advisory committee, and the objectives of the program evaluation. A contractor consisting of experienced survey operations staff formatted the survey and review panel scoring sheet for online ease of completion, as well as to facilitate analysis. Additionally, web-based surveys and the expert review panel process will be pilot tested with up to 9 individuals prior to being fielded.
Individuals who have consulted on statistical aspects and/or in analyzing the information are listed in Attachment 7.
1 The “Outcome Evaluation of the NIH Director’s Pioneer Award Program for NIHs Office of the Director (OD” was approved by OMB on 11/9/2009 (OMB No. 0925-0606)
2Lal, B. et al. " An Outcomes Evaluation of the National Institutes of Health (NIH) Director's Pioneer Award (NDPA) Program, FY 2004-2006." August 2012. http://commonfund.nih.gov/sites/default/files/P-4899_Final_Redacted.pdf
3 The survey received Office of Management and Budget approval on March 7, 2012 under the title of, “PS-OC Survey” (OMB No. 0925-0642-07).
2/11/2015
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | TABLE OF CONTENTS |
| Author | Vivian Horovitch-Kelley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |