Supporting Statement A For:
Outcomes Evaluation of the NCI Cancer Prevention Fellowship Program (NCI)
December 11, 2013
Jessica Faupel-Badger, PhD, MPH
Associate Director
Cancer Prevention Fellowship Program
National Cancer Institute
9609 Medical Center Drive
Room 2W136 MSC 9712
Bethesda, MD 20892-9712
Telephone: 240-276-5650
Fax: 240-276-7883
Email: [email protected]
Table of Contents
A. Justification 1
A.1 Circumstances Making the Collection of Information Necessary 1
A.2. Purpose and Use of the Information Collection 4
A.3 Use of Improved Information Technology and Burden Reduction 8
A.4 Efforts to Identify Duplication and Use of Similar Information 9
A.5 Impact on Small Businesses or Other Small Entities 9
A.6 Consequences of Collecting the Information Less Frequently 9
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 9
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 9
A.9 Explanation of Any Payment or Gift to Respondents 10
A.10 Assurance of Confidentiality Provided to Respondents 10
A.11 Justification for Sensitive Questions 12
A.12 Estimates of Annualized Burden Hours And Costs 13
A.13 Estimates of Other Total Annual Cost Burden to Respondents and
Record Keepers 14
A.14 Annualized Cost to the Federal Government 14
A.15 Explanation for Program Changes or Adjustments 15
A.16 Plans for Tabulation and Publication and Project Time Schedule 15
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 18
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 18
Attachments
1. A. Survey of Alumni (Screenshots)
B. survey of Alumni (Word Document with all questions)
2. A. Survey of Applicants (Screenshots)
B. Survey of Applicants (Word Document with all questions)
3. A. Survey of F32 Awardees (Screenshots)
B. Survey of F32 Awardees (Word Document with all questions)
4. Privacy Impact Assessment
5. Privacy Act Memo
6. IRB Approval Letters
7. Email Survey Notice to CPFP Alumni, CPFP Applicants and F32 Awardees
8. Invitation Letter for Survey to CPFP Alumni, CPFP Applicants and F32 Awardees
9. Email Survey Reminder to CPFP Alumni, CPFP Applicants and F32 Awardees
10. Telephone Follow-up Script for CPFP Alumni, CPFP Applicants and F32 Awardees
This is a request for OMB to approve the new submission titled, “Outcomes Evaluation of the National Cancer Institute Cancer Prevention Fellowship Program” for 1 year. The National Cancer Institute’s (NCI) Cancer Prevention Fellowship Program (CPFP) mission is to train early career scientists from diverse disciplines to become outstanding independent researchers and leaders. This postdoctoral program conducted on-site at NCI has been in existence for over 25 years and has approximately 200 alumni. Despite its long existence, there has not been a comprehensive systematic, formal evaluation of career outcomes of CPFP trainees in the history of the program at NCI. The current study will focus on the implementation of a new survey instrument to capture career outcomes from CPFP alumni and two comparison groups, CPFP applicants and NCI F32 awardees. With the diversity of disciplines represented by CPFP alumni, the results of this evaluation will be of broad interest to the biomedical research training community.
A. Justification
A.1 Circumstances Making the Collection of Information Necessary
The data collection is part of an outcomes evaluation of the National Cancer Institute’s (NCI) Cancer Prevention Fellowship Program (CPFP). The overarching mission of the CPFP is training promising early career scientists from diverse disciplines to become outstanding independent researchers and leaders in their respective fields to address the Nation's cancer prevention research needs. Key goals of the program are:
Section 410 of the Public Health Service Act (42 USC § 285) authorizes the collection of the information. The Public Health Service Act indicates that the purpose of the NCI is to “conduct and support…training…with respect to the cause, diagnosis, prevention, and treatment of cancer…”
The CPFP is a 4-year intramural1 postdoctoral program for early career scientists and has been in existence for 25 years. Each year, the CPFP selects 10 to15 individuals to become Fellows; approximately 200 individuals have completed the program and there are 40 Fellows currently at varying stages in the program. Fellows without a background in epidemiology or statistics are provided funding during their first year to obtain a Master in Public Health degree from an accredited university. Cancer Prevention Fellows have substantial freedom in selecting their preceptors and research projects during their time in the program. Much of a Fellow’s time consists of one-on-one mentoring relationships with experienced NCI scientists to work on research studies, which lead to presentations at national meetings and publication in peer-reviewed journals. The Fellowship provides additional training, including summer courses in cancer prevention, weekly presentations and meetings with CPFP staff and other Fellows, grant writing, oral presentations, leadership, and transition to independence. For more information, see http://www3.cancer.gov/prevention/pob/.
Despite its long existence, there has not been a comprehensive systematic, formal evaluation of the career outcomes of CPFP trainees in the history of the program at NCI. The only outcomes evaluation of the CPFP was conducted in 2006 and compared number of scientific publications of CPFP alumni in the three years since completing the program to the number of publications produced while in the fellowship program. Given the diversity of disciplines recruited to the program and potential career outcomes for postdoctoral fellows trained in cancer prevention and control, a sufficient amount of time was required to have a cadre of CPFP alumni of varying career pathways and stages in order to conduct a comprehensive evaluation of career outcomes.
This evaluation of an intramural training program with training and mentoring experiences provides a unique opportunity to examine many facets of a biomedical research training program. The CPFP contains structured and formal training offerings and is truly transdisciplinary, with Fellows from different scientific backgrounds routinely working together over several years. It also is designed to support fellows toward career opportunities in many settings in addition to traditional academic pathways.
In 2011, the CPFP Branch conducted a comprehensive and systematic literature review of prior studies evaluating postgraduate intramural training programs (defined in part as full-time structured programs conducted on-site within a single institution) was conducted as part of this study. We learned that, although there have been many descriptive studies and much data reported on trainees’ satisfaction, very few studies have examined career outcomes or other evaluation outcome measures of trainees. Even among those studies reporting outcomes, study designs were generally weak (e.g., small sample sizes, absence of a comparison population [comparison group]). Another aspect of the review involved attempting to obtain data from existing, archival sources (e.g., Internet searches for curriculum vita, National Institutes of Health (NIH) database searches for receipt of grants) for a 10% random sample of CPFP alumni; unfortunately, extremely limited outcome information was available from such sources. Based on these findings, we will extend our data collection beyond archival sources by administering a survey to alumni and to two comparison groups.
An evaluation of this program will provide insight to the National Cancer Institute, National Institutes of Health, and the broader postgraduate training community about the roles that structured training programs in general, and transdisciplinary programs specifically, may have on career outcomes. Ultimately, we want to determine if the CPFP is meeting its overarching goal of training leaders in the field of cancer prevention and control. Based on the career outcomes data, changes may be made to the selection and training of future fellows. The results of this evaluation may also inform the structure and training activities of other intramural and extramural training programs funded by the NCI and NIH. The evaluation itself may serve as a model for evaluations of post-doctoral programs around the country.
A.2 Purpose and Use of the Information
The CPFP program was introduced in 1987 as an intramural training program to provide training for post-doctoral fellows in cancer prevention. Although the program has trained over 200 scientists, a full scale external evaluation has yet to be conducted. As part of a literature review, a full scale external evaluation was recommended to inform decision making, to identify opportunities for improvements, and to demonstrate the importance of the program to key stakeholders.
Existing data sources that may be utilized for the evaluation of career outcomes include: National Institutes of Health (NIH) IMPACII (NIH grants applied for and received); Discovery Logic’s ScienceWire (USDA, NSF, DoD grants received); AAMC Faculty Roster (academic position within institutions of medical education); CPFP alumni database (position, current affiliation); PubMed/Medline (publications, co-authorship and collaborations, field of research); society membership lists (professional society memberships); and Web of Knowledge (publications, co-authorship and collaborations, field of research, citations, journal impact factor).
Because use of existing data sources is expected to yield very incomplete career outcome data and no information on the CPFP, additional data collection will be conducted to collect information from 200 CPFP alumni, 283 CPFP applicants and 367 individuals who were recipients of funding from NCI through the F32 mechanism utilizing three newly developed web-based surveys: Survey of Alumni (Attachment 1), Survey of Applicants (Attachment 2), and Survey of F32 Awardees (Attachment 3). The ultimate intent is to collect quantitative outcome data not included in any of the publically available databases. Aspects of the CPFP such as mentorship, creation of leaders in the cancer research and cancer prevention research fields, participant opinions on program benefits, and participant leveraging of skills and relationships developed during the program may only be addressed through additional data collection.
What are the scientific disciplines of CPFP alumni?
What is the scientific productivity of CPFP alumni?
What are the career choices or pathways (e.g., academic, government, or private sector) of CPFP alumni?
What is the career advancement (e.g., leadership positions, promotion to tenured positions) of CPFP alumni?
What is the peer recognition (e.g., service on editorial boards, leadership positions within professional organizations, or receipt of professional awards) for CPFP alumni?
To what extent do CPFP alumni collaborate with persons from other disciplines in their work?
To what extent do CPFP alumni feel that participating in the CPFP had an impact on their career trajectory, including positions held, research focus, and current employment?
Based on their experiences and beliefs about current and future directions for cancer prevention, what recommendations do CPFP alumni have for maintaining or improving the training aspects (process) of the CPFP?
Work history
Type of work
Professional activities
Career advancement
Professional association awards
Career outcomes
Level of preparation for career
Satisfaction with CPFP program
Perceived benefits of the CPFP program
Recommendations for improvement of the CPFP program
Demographics
There are three direct target audiences for this outcomes evaluation: Program administrators, NIH/NCI leadership, and former program fellows. In addition to these direct audiences, other evaluators in both program and academic settings may derive benefit from this study, both in the explication of methods and in a better understanding of program theory. Directors of training programs run by other agencies or universities may also find the results of this evaluation useful in their program planning and evaluation efforts.
CPFP Program Administrators: For purposes of determining the effectiveness of the CPFP program and making possible program adjustments, administrators will have access to data tables for all comparison groups for which data is available. This should include all CPFP fellows, CPFP applicants and F32 awardees. This will enable administrators to address the evaluation questions and to maintain a record of performance for future evaluations.
NCI/NIH Leadership: For purposes of communicating program performance and impact, the final report with aggregate findings and recommendations may be made available in hard copy or electronic format to NCI/NIH Leadership. Findings and recommendations may be used to support, for example, OMB’s Program Assessment Rating Tool (PART) reporting. Individual-level data will not be provided for any of the study groups.
Current and Former CPFP Fellows and Applicants to the CPFP program: Former fellows and applicants will have access to the final report with aggregate findings and recommendations. Individual-level data will not be provided for any of the study groups. Outcomes and process evaluation measures should be aggregated across individuals among all CPFP fellows, CPFP applicants, and F32 awardees to enable a comparison of the observed impact of the CPFP program on career outcomes. For applicants, this may affect their decision to apply or, if offered a position, their decision to enter the program.
Training Program Directors: Directors or administrators of training programs in other federal agencies or in the extramural community may access the results of a full-scale evaluation of the CPFP program including data aggregated by study group among all CPFP fellows, CPFP applicants, and F32 awardees. The methodologies used for the full-scale evaluation may be of use, or the aggregated results may be used as a comparison to findings regarding their alumni.
Research and Evaluation Communities: Researchers in the academic or scientific community may access the results of the full-scale evaluation of the CPFP program including data aggregated by study group, including CPFP fellows, CPFP applicants, and F32 awardees. Some individuals in the target audience may be interested in the methodology used for the CPFP program evaluation. Due to privacy concerns individual-level data will not be provided for any of the study groups. Dissemination to this audience may take the form of the study report, conference presentations, and publication of study methods and findings in scholarly journals.
In the web-based survey approach, respondents provide data using a computerized system to collect and enter data. The respondent provides data directly without an interviewer. The advantages of this format lie in that (1) data may be collected for all respondents using this method allowing for uniform data collection; (2) respondents can complete the survey at their convenience; (3) the system navigates through the survey for the respondent; (4) removes respondent burden of having to return survey by mail; (5) low cost of dissemination; (6) low cost for the dissemination of reminders; and (7) the elimination of manual data entry of completed surveys. In addition, information may be obtained regarding the number of survey respondents who initiated the survey but who did not complete and submit the survey. Challenges include obtaining an email address for each respondent, as well as potential lack of familiarity with online surveys for some alumni (expected to be very low in this very highly educated population). Email addresses are available for CPFP alumni, but are not readily available for other comparison groups. The investigators will trace the addresses of applicants and F32 fellows using simple internet searches. Response rates for an online survey are expected to be about 60 to 80%.
Westat will use an internally maintained Commercial Off-the-Shelf (COTS) web survey platform named Vovici. This platform, which is integrated into Westat systems, allows the collection of web responses and creating reports on completed and outstanding responses. In addition, Westat will create activity reports to further assist in helping project staff monitor activity and perform tracing to increase response rates.
A Privacy Impact Assessment (PIA) has been drafted and approved by NCI and NIH and is currently under review at HHS. The IT System name is "Cancer Prevention Fellowship Program (CPFP) Application System." (Attachment 4).
The NCI CPFP was introduced in 1987 as an intramural training program to provide training for postdoctoral fellows in cancer prevention and, although the program has trained over 200 scientists, a full-scale external evaluation has never been conducted. Doctoral program evaluations have been conducted previously but none in cancer prevention. An extensive literature review, conducted in 2010 by Discovery Logic, indicated the need for this program evaluation and that it would not duplicate already existing information or previous evaluations.
A full-scale evaluation was recommended to inform decision-making, to identify opportunities for improvement, and to demonstrate the importance of the program to key stakeholders.
A.5 Impact on Small Businesses or Other Small Entities
No small entities will be involved in this survey. All respondents will be individuals who participate voluntarily.
This information collection is a one-time collection.
The study is consistent with the information collection guidelines in 5 CFR 1320.5.
The 60-Day Federal Register notice soliciting comments on this study prior to initial submission to OMB was published on August 12, 2013, Vol. 78, P. 48879. One general public comment was received on August 18, 2013, and responded to the following day.
The web-based survey instrument was developed through NCI’s collaboration with Westat Incorporated. NCI worked previously with Discovery Logic on the literature review that provided the foundation that helped inform the full scale study. Between November 2010 and June 2011, Discovery Logic performed a literature search of training program evaluations and searched for surveys, existing instruments and questions that might be useful as NCI developed the survey. They looked at publications, grants, and Google searches to see how much information they could find on a subset of the alumni population. This effort demonstrated that it would be difficult to do the program evaluation without directly contacting individuals, as the information in existing data sources was variable. NCI and Westat distilled down key themes and questions that NCI was interested in asking for the full scale study. Westat developed the instrument using input from the Pilot study2 and then conducted a Pre-test of the instrument. Westat will be responsible for recruiting participants, data collection, data analysis and final reporting.
The NCI project team has consulted with Larry Solomon, Ph.D., Scientific Program Director, NCI, Office of Science Planning and Assessment; James Corrigan, Ph.D., Branch Chief, NCI, Office of Science Planning and Assessment; Mario Cerritelli, Ph.D., Chief, Office of Knowledge and Educational Resources, National Institute of Allergy and Infectious Diseases; Christie Drew, Ph.D., Branch Chief, Program Analysis Branch, National Institute of Environmental Health Sciences; and Julie Mason, Ph.D., Associate Director, NCI Center for Cancer Training. In the future, the NCI project team anticipates consulting with the Program Assessment Branch of NCI’s Office of Science Planning and Assessment and the Evaluation Advisory Committee for guidance on the project. The Committee will potentially include members such as the NCI Center for Cancer Training Leadership, NCI evaluation officers, NIH Intramural Training Program Directors and Extramural Training Program Directors.
This information collection does not involve payment or gifts to respondents.
Potential participants (CPFP Alumni, CPFP Applicants and F32 Awardees) will receive email notifications, invitations and reminders announcing the evaluation, explaining its purpose, detailing the topics the survey will cover, and describing both the voluntary nature of participation and information that the data will be kept private to the extent provided by law. The following steps outline the procedures for contacting potential participants to ensure compliance and maximize response rates:
All proposed emails include language stating that all collected information will be kept private and not disclosed in any identifiable form to anyone but the researcher conducting the study, except as otherwise required by law. Individuals who choose to participate will be providing implicit consent by their participation. This information collection is covered by the NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD” published in the Federal Register on 9/26/2002, Vol. 67, P. 60743. All study personnel will adhere to the provisions stipulated within that announcement (Attachment 5).
As the target population is not vulnerable and the questions are not personal or intrusive, risks to participants are expected to be minimal. The NIH Office of Human Subjects Research Protection (OHSRP) designated the CPFP Evaluation Study as exempt from IRB review on March 26, 2013 and Westat IRB approval was received on November 29, 2012 (Attachment 6).
Study personnel have obtained proper security clearances and are required to adhere to strict professional survey standards and have signed a non-disclosure agreement as a condition of their employment. Web-based, computer-based and any hard copy data collection forms will be maintained in a secure area for receipt and processing. All data files on multi-user systems will be under the control of a database manager and will be subject to controlled access only by authorized personnel. Personal identifying information (PII) will be maintained separately from completed data collection forms, and from computerized data files used for analysis. Final Reports will be based on aggregate data in which individuals are not identified.
After the data collection is completed, all hard copy collected information and study materials will be stored in a locked, secure facility for two years, and then will be shredded. Electronic data will be password protected and stored by the data management contractor, and will also be destroyed after two years.
A.11 Justification for Sensitive Questions
Personally identifiable information (PII) is collected in the form of the participant’s name, email address and phone number which is needed to contact potential participants. This information will be obtained from archival sources, when possible. Additionally, sensitive information will be collected in the form of salary, gender, and race/ethnicity information as part of the survey data collection. The CPFP program evaluation survey aims to assess the career outcomes of program alumni as a proxy to overall post-doctoral program success; if program alumni career outcomes are found to surpass the career outcomes of comparable alumni of other programs or the career outcomes of applicants to CPFP who were not admitted to the program, the latter will be a good indication that the program is succeeding in one of its primary goals, which is to train leaders in the field of cancer prevention and control. Salary is considered a key measure of career outcomes in any profession and is thus commonly used in this type of evaluation research. It is thus critical that salary be included as one of the career outcome measures collected through the CPFP evaluation instrument. If salary where not included, the investigators would miss a unique opportunity to learn whether the program’s alumni are more successful (in terms of salary gains) than program applicants or other NIH postdoctoral fellows.
Participation in the study is voluntary and participants have the right not to answer any questions without consequences. Section A.10 discusses the steps taken to safeguard this information.
Data collection activities for all participants involve completion of a web-based survey. CPFP alumni will complete the Survey of Alumni (Attachment 1) while the two comparison groups, the CPFP Applicants and the F32 Awardees, will complete the Survey of Applicants (Attachment 2) and the Survey of F32 Awardees (Attachment 3) respectively. All data collection will be completed in one year. While federal government employees will be invited to complete the survey, they are not exempt form burden. Their participation is not due to their professional expertise or responsibilities. Respondents were selected based on the criteria stated previously.
The estimated time for completing the CPFP Alumni Survey is 25 minutes while the Survey of Applicants and the Survey of F32 Awardees is estimated at 20 minutes each. Though the invitation is being sent to a total of 850 respondents, it is anticipated that there will be a response rate of between 60% to 80% (see Supporting Statement B, B.1.2 Sampling Methods for further discussion). The number of respondents below has taken into account this estimated response rate. The estimate of the annualized burden hours, an estimated annualized total burden of 197 hours, is summarized in Table A.12-1.
Table A.12 – 1. Estimates of Annualized Burden Hours
Type of Respondents |
Number of Respondents |
Number of Responses Per Respondent |
Average Burden per Response (in hours) |
Total Annual Burden Hours |
CPFP Alumni |
160 |
1 |
25/60 |
67 |
CPFP Applicants |
170 |
1 |
20/60 |
57 |
F32 Awardees |
220 |
1 |
20/60 |
73 |
Totals |
550 |
|
197 |
|
The cost burden to respondents is essentially the time required to read the instructions and complete the survey. The total annualized cost to the respondents is estimated to be $10,967 calculated at $55.67 per hour; an average of the hourly wage rate for epidemiologists at $34, medical scientists at $42 and physicians at $91 (U.S., Department of Labor, Bureau of Labor Statistics, 2012). The costs are summarized in Table A.12-2.
Table A.12 – 2. Estimated Annualized Cost to Respondents
Type of Respondents |
Number of Respondents |
Total Annual Burden Hours |
Hourly Wage Rate |
Total Respondent Cost |
CPFP Alumni |
160 |
67 |
$55.67 |
$3,729.89 |
CPFP Applicants |
170 |
57 |
$55.67 |
$3,173.19 |
F32 Awardees |
220 |
73 |
$55.67 |
$4,063.91 |
Total |
$10,966.99 |
|||
There are no direct costs to respondents other than their time to participate in the study.
The largest cost to the federal government is to pay a contractor $319,762 to conduct the study and deliver data files spread over two years. NCI costs are based entirely on labor. It is estimated that the study will require about 0.25 FTE total per year spread over 2 scientists (Senior Biomedical Scientist and CPFP Program Director) at the GS14 level or above, totaling $33,000 per year. These expenses are related to directing contractors, overseeing and solving problems as they arise, developing materials, supervising data collection, data coding, data cleaning, data analyses, and preparation of manuscripts and presentations. The estimated annualized cost to the Federal Government is $192,881, summarized in Table 14-1.
Table A.14-1 Annual Cost to the Federal Government
|
TOTAL |
ANNUAL AVERAGE |
Contractor Costs |
$319,762 |
$159,881 |
NCI Personnel Subtotal |
$66,000 |
$33,000 |
Grand Total |
$385,762 |
$192,881 |
This is new information collection.
The web-based survey information collection will begin within one month of obtaining OMB approval. The contract period will include fielding, analyzing, and disseminating findings from these studies. Westat, Inc. will be responsible for preparing the analytic databases resulting from the study. The timetable for the data collection is shown below, in Table A.16-1.
Table A.16-1. Project Time Table
Activity |
Timeline |
Email survey notice sent to potential participants |
Within 1 month of OMB approval |
Email invitation letter and web-based survey sent to potential participants |
Within 1 month – 2 months of OMB approval |
Reminder email notice sent to non-respondents |
1-3 months after OMB approval |
Telephone follow-up call to non-respondents |
2-4 months after OMB approval |
Completed field work |
5 months after OMB approval |
Validation |
5-7 months after OMB approval |
Analyses |
7-9 months after OMB approval |
Reporting |
8-9 months after OMB approval |
Publication |
9 months after OMB approval |
A.16.1 Analysis of the Study Data
Many of the Web survey analyses will consist of descriptive statistics (e.g., percentages, means, medians, and standard deviations, as appropriate), cross-tabulations, and graphical summaries. Data may be disaggregated by year to present a longitudinal view of characteristics of CPFP class cohorts. For the subgroup analysis, a major distinction will be made to compare the treatment and two comparison groups. Tests of significance will be conducted using statistics such as chi-squared, t-tests, or analysis of variance (ANOVA) to examine differences. The analysis will be performed in SAS with PROC SURVEY with the non-response adjusted weight. See Table A.16.2 for a list of research questions and associated measures.
Research Questions |
Measures |
What are the scientific disciplines of CPFP alumni? |
Highest degrees/credentials Postdoctoral training |
What is the scientific productivity of CPFP alumni? |
Publications Presentations Collaborations with others Contributions to the fields |
What are the career choices or pathways (e.g., academic, government, or private sector) of CPFP alumni? |
Employment status Current employment Primary employment type/sector Field/disciplines of work Research activity |
What is the career advancement (e.g., leadership positions, promotion to tenured positions) of CPFP alumni? |
Leadership positions Management/supervisory roles, Tenure status |
What is the peer recognition (e.g., service on editorial boards, leadership positions within professional organizations, or receipt of professional awards) for CPFP alumni? |
Professional associations Professional service Leadership positions in professional service, Professional awards |
To what extent do CPFP alumni collaborate with persons from other disciplines in their work? |
Interdisciplinary collaborations |
To what extent do CPFP alumni feel that participating in the CPFP had an impact on their career trajectory, including positions held, research focus, and current employment? |
Impact on career trajectory Impact on research skills Impact on management skills Impact on other professional skills |
Based on their experiences and beliefs about current and future directions for cancer prevention, what recommendations do CPFP alumni have for maintaining or improving the training aspects (process) of the CPFP? |
Open-ended recommendations question |
Table A.16-2. Research Questions and Measures for the CPFP Evaluation Study
The analysis of individual survey items described above will provide an overview of how survey participants responded to individual aspects of a broader outcome (e.g., professional accomplishments). While the analyses of differences between alumni and other comparison groups at the individual item level are informative, it will be difficult to examine them simultaneously to make reliable assessments about the overall impact of the CPFP intervention.
Therefore, we will also employ psychometric techniques (i.e., Item Response Theory or Confirmatory Factor Analysis) to create reliable composite scores. Many of the outcome measures are complex concepts that require multiple questions to be fully captured. The advantage of using psychometric techniques is that the analysis will allow us to combine many relevant items into fewer constructs and statistically verify whether these items belong to the constructs we had in mind.
We anticipate using NVivo software to help perform content analyses of responses to open-ended questions on the Web survey.
A.16.2 Products of the Study
Products of the evaluation include a full final report detailing the methodology, key findings, conclusions, and recommendations. Data in the report will be presented in user-friendly graphs and tables with textual analysis. An executive summary will be developed to concisely convey the salient findings to a general audience. Additionally, a Power Point slide deck will be prepared for presentation purposes to highlight the key findings and will include graphs, tables, and other relevant information.
A.16.3 Dissemination of Results
Intended audiences for the results of the evaluation include CPFP staff as well as Center for Cancer Training staff. Key findings will be presented to NIH intramural training directors and evaluation officers in other NIH ICs and with the extramural training and career development community. CPFP Staff will prepare a manuscript describing the evaluation design and findings for submission to a peer-reviewed scientific journal and will present findings at education-oriented scientific conferences. The results will also be shared with the broader planning and evaluation community. A copy of the final report will be provided to the NIH Evaluation Office.
A.16.4 Use of Results
The results will be used to determine if the CPFP is meeting its main goal of training future leaders in the fields of cancer and cancer prevention research. Based on the career outcomes data, changes may be made to the selection process and training of future fellows. The results of this evaluation may also inform the structure and training activities of other intramural and extramurally funded training programs.
The web-based survey will not require exemption from displaying the expiration date of OMB approval. Any reproduction of the data collection instrument will prominently display the OMB approval number and expiration date.
The CPFP Evaluation Study does not require any exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
1 The intramural program at NIH is research that is conducted by NIH employees within NIH. It is juxtaposed by the extramural program in which research is conducted by people not employed by NIH.
2 Reference the Supporting Statement B, Section B.4 for more information.
| File Type | application/msword |
| File Title | PLCO SCREENING TRIAL |
| Author | Linda Suit |
| Last Modified By | Vivian Horovitch-Kelley |
| File Modified | 2013-12-11 |
| File Created | 2013-12-11 |